[Buildbot-devel] SVN checkouts: preserving directory name, multiple concurrent directories

[Natalie Tasman]

Hi all,

The project that I'm working on has a SVN repository structure like
this:

.../trunk/projectA
.../trunk/projectB
.../trunk/projectC
...
.../trunk/projectF

I only need projectB and projectF, so I really don't want to have to
checkout the entire trunk.

In order to build, I need the following local directory strucure:

<some path>/projectB
<some path>/projectF

ProjectF has references like "../../../projectB", so it's necessary to
preserve this structure.

With commandline svn, it's no problem to preserve the directory name
instead of renaming it to "source", as BuildBot likes to.

It seems that I might be able to get something going with a series of
ShellCommands to manually run SVN (and properly execute the rm -rf /
cp commands to simulate BuildBot's SVN "copy" mode), and so on-- but
to have to redo this for all operating systems? That was the whole
point of using something like BuildBot in the first place:)

I've tried searching the mailing list but wasn't able to find the
answer. Thanks in advance for any assistance!

Natalie

[Zellyn Hunter]

Natalie,

Have you tried using the "workdir" arg to buildbot.steps.source.SVN to
get it to place files in a specific location? That worked for me:
before I added it, the two checkouts were clearing and repopulating
the same folder. I use relative paths like workdir="build/appA" and
workdir="build/appB".

Zellyn



On Fri, Jul 17, 2009 at 4:47 PM, Natalie

Post by Natalie Tasman
Hi all,
The project that I'm working on has a SVN repository structure like
.../trunk/projectA
.../trunk/projectB
.../trunk/projectC
...
.../trunk/projectF
I only need projectB and projectF, so I really don't want to have to
checkout the entire trunk.
<some path>/projectB
<some path>/projectF
ProjectF has references like "../../../projectB", so it's necessary to
preserve this structure.
With commandline svn, it's no problem to preserve the directory name
instead of renaming it to "source", as BuildBot likes to.
It seems that I might be able to get something going with a series of
ShellCommands to manually run SVN (and properly execute the rm -rf /
cp commands to simulate BuildBot's SVN "copy" mode), and so on-- but
to have to redo this for all operating systems?  That was the whole
point of using something like BuildBot in the first place:)
I've tried searching the mailing list but wasn't able to find the
answer.  Thanks in advance for any assistance!
Natalie
------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Buildbot-devel mailing list
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

[Natalie Tasman]

Hello Zellyn,

Thank you for the suggestion. It looks like a good step in the right
direction, but it doesn't quite do the trick for me.

With 0.7.11p1, buildbot.steps.source.SVN set to mode "copy":

BuildFactory.addStep(SVN(
mode="copy",
baseURL=svn_base_url,
defaultBranch="appA",
workdir="appA-src",
))

BuildFactory.addStep(SVN(
mode="copy",
baseURL=svn_base_url,
defaultBranch="appB",
workdir="appB-src",
))

BuildFactory.addStep(ShellCommand(command="pwd"))


Here's what automatically happens:

rm -rf appA-src
rm -rf source
svn checkout [http...]/appA source
svnversion "."
cp [...]/source [...]/appA-src

rm -rf appB-src
rm -rf source
svn checkout [http...]/appB source
svnversion "."
cp [...]/source [...]/appB-src


pwd: "[...]/build" (empty directory)


The problems here:

1) "copy" is intended to keep a clean working checkout, then copy to a
new build directory. That doesn't happen here. Buildbot zaps the
default "source" directory and checks out everything, for both
projects. My checkouts are over 100MB, so doing a clean checkout
every time is prohibitively expensive.

Instead, I would like to see buildbot maintain two source directories
so that the next "svn update" step would just pick up deltas, not
require the entire checkout again.


2) "copy" doesn't copy the sources to the build directory.


Thanks again, and any other suggestions are appreciated!

Natalie

Post by Zellyn Hunter
Natalie,
Have you tried using the "workdir" arg to buildbot.steps.source.SVN to
before I added it, the two checkouts were clearing and repopulating
the same folder. I use relative paths like workdir="build/appA" and
workdir="build/appB".
Zellyn
On Fri, Jul 17, 2009 at 4:47 PM, Natalie

Post by Natalie Tasman
Hi all,
The project that I'm working on has a SVN repository structure like
.../trunk/projectA
.../trunk/projectB
.../trunk/projectC
...
.../trunk/projectF
I only need projectB and projectF, so I really don't want to have to
checkout the entire trunk.
<some path>/projectB
<some path>/projectF
ProjectF has references like "../../../projectB", so it's necessary to
preserve this structure.
With commandline svn, it's no problem to preserve the directory name
instead of renaming it to "source", as BuildBot likes to.
It seems that I might be able to get something going with a series of
ShellCommands to manually run SVN (and properly execute the rm -rf /
cp commands to simulate BuildBot's SVN "copy" mode), and so on-- but
to have to redo this for all operating systems?  That was the whole
point of using something like BuildBot in the first place:)
I've tried searching the mailing list but wasn't able to find the
answer.  Thanks in advance for any assistance!
Natalie
------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Buildbot-devel mailing list
https://lists.sourceforge.net/lists/listinfo/buildbot-devel

[Natalie Tasman]

With Buildbot.steps.source.SVN set to mode "update", it works better,
in that the two checkout directories are correctly re-used for updates
(that is, nothing is zapped and completely downloaded again.)

But it would still be really nice to get the "copy" functionality
working. I've tried looking at the sources to see where the default
svn checkout directory "source" is set, in case there's another
undocumented parameter that can override this; I don't see anything in
the obvious place, source.py.

Natalie

Post by Natalie Tasman
Hello Zellyn,
Thank you for the suggestion. It looks like a good step in the right
direction, but it doesn't quite do the trick for me.
BuildFactory.addStep(SVN(
mode="copy",
baseURL=svn_base_url,
defaultBranch="appA",
workdir="appA-src",
))
BuildFactory.addStep(SVN(
mode="copy",
baseURL=svn_base_url,
defaultBranch="appB",
workdir="appB-src",
))
BuildFactory.addStep(ShellCommand(command="pwd"))
rm -rf appA-src
rm -rf source
svn checkout [http...]/appA source
svnversion "."
cp [...]/source [...]/appA-src
rm -rf appB-src
rm -rf source
svn checkout [http...]/appB source
svnversion "."
cp [...]/source [...]/appB-src
pwd: "[...]/build" (empty directory)
1) "copy" is intended to keep a clean working checkout, then copy to a
new build directory. That doesn't happen here. Buildbot zaps the
default "source" directory and checks out everything, for both
projects. My checkouts are over 100MB, so doing a clean checkout
every time is prohibitively expensive.
Instead, I would like to see buildbot maintain two source directories
so that the next "svn update" step would just pick up deltas, not
require the entire checkout again.
2) "copy" doesn't copy the sources to the build directory.
Thanks again, and any other suggestions are appreciated!
Natalie

Post by Zellyn Hunter
Natalie,
Have you tried using the "workdir" arg to buildbot.steps.source.SVN to
before I added it, the two checkouts were clearing and repopulating
the same folder. I use relative paths like workdir="build/appA" and
workdir="build/appB".
Zellyn
On Fri, Jul 17, 2009 at 4:47 PM, Natalie

Post by Natalie Tasman
Hi all,
The project that I'm working on has a SVN repository structure like
.../trunk/projectA
.../trunk/projectB
.../trunk/projectC
...
.../trunk/projectF
I only need projectB and projectF, so I really don't want to have to
checkout the entire trunk.
<some path>/projectB
<some path>/projectF
ProjectF has references like "../../../projectB", so it's
necessary to
preserve this structure.
With commandline svn, it's no problem to preserve the directory name
instead of renaming it to "source", as BuildBot likes to.
It seems that I might be able to get something going with a series of
ShellCommands to manually run SVN (and properly execute the rm -rf /
cp commands to simulate BuildBot's SVN "copy" mode), and so on-- but
to have to redo this for all operating systems? That was the whole
point of using something like BuildBot in the first place:)
I've tried searching the mailing list but wasn't able to find the
answer. Thanks in advance for any assistance!
Natalie
------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Buildbot-devel mailing list
https://lists.sourceforge.net/lists/listinfo/buildbot-devel