The CruiseControl.NET operation flow, documented in the Operation flow of CCNet FAQ, where the label is generated first makes some sense for a centralized VCS but not so much for a DVCS. With a DVCS, it would be better to generate the label after getting the source because the source itself might be needed for the label (e.g., the .hgtags file in Mercurial or other local source files used to compute version and build numbers).
Has anyone figured out a way to run the CCNet labeller after getting the source?
We don't use the Labeller in CC.NET, but nevertheless, our approach might work for you as well:
create a separate project MercurialCheckout that performs
Mercurial checkout
Executable task that updates a text file triggers/mercurial.txt on the build server
change your "real" project from sourcecontrol type mercurial to sourcecontrol type filesystem, watching the triggers/directory
Your real project then includes the Labeller task, and since the Mercurial checkout was done before, the Labeller should generate the "right" label.
Kind regards
Frank
There's no way to make the labeller task execute after the source control task. But you only need that if your label source data is exclusively stored in the local working directory. You said your label source is in version control, so you should use a labeller than can retrieve it from version control.
Most version control systems have a command-line that will allow you to read the file contents from stdout. CCNet labellers are easy to write and deploy. You can take in the command-line and arguments required to fetch the latest version info directly from the source.
Example ccnet labeller config
<labeller type="svnLabeller">
<svn>C:\Program Files (x86)\svn\bin\svn.exe</svn>
<repository>https://path.to/project</repository>
<versionInfoPath>/AssemblyInfo.cs</versionInfoPath>
</labeller>
Related
I'm working on a collaborative scientific project that is made up by a handful of Python scripts (1M max) and a relatively large dataset (1.5 GB). The datasets are tightly linked to the python scripts since the datasets themselves are the science and the scripts are a simple interface to them.
I'm using Mercurial as my source control tool, but I am not clear on a good mechanism to define the repository. Logistically it makes sense to bundle these together so that by cloning the repository you'd get the entire package. On the other hand, I'm concerned about the source control tool dealing with large amounts of data.
Is there a clean mechanism to handle this?
If the data files change rarely and you normally need all of them anyway, then just add them to Mercurial and be done with it. All your clones will be 1.5 GB, but that is just the way it has to be with that amount of data.
if the data is binary data and changed often, then you might try to avoid downloading all the old data. One way to do this is to use a Subversion subrepository. You will have a .hgsub file with
data = [svn]http://svn.some.edu/me/ourdata
which tells Mercurial to make a svn checkout from the right-hand side URL and put the Subversion working copy into your Mercurial clone as data. Mercurial will maintain an additional file for you called .hgsubstate, in which it records the SVN revision number to checkout for any given Mercurial changeset. By using Subversion like this, you only end up with the latest version of the data on your machine, but Mercurial will know how to get older versions of the data when needed. Please see this guide to subrepositories if you go down this route.
There is an article on the official wiki about large binary files. But the proposition of #MartinGeisler is a really nice new alternative.
My first inclination is to separate the python scripts out into their own repository, but I really need more domain information to make the "right" call.
On the one hand, if new datasets will be created then you would want a core set of tools to be able to handle all of them, right? But I can also see how new datasets may introduce cases that the scripts may not have previously handled... although it seems like in an ideal world you would want scripts that are written in a general way so they can handle future data and existing datasets??
I want to use Mercurial to capture changes made to the vanilla installation of a piece of software we use. Everytime we upgrade the software, we need to manually edit the various configuration files and add 3rd party libraries that we use in the current version of the software. Creating patches for the configuration files changes are fine, but how do I add 3rd party libraries (binaries) to a Mercurial patch? Is it even possible?
If you were to try to get the patch for the 7th revision...
hg export --git -r 7 -o 7.patch
Yes, the mq extension can handle binary data just as well as textual data. It will use Git's extended patch format to save the binary data. This is transparently handled for you when you refresh a patch with modified binary files.
Whether or not this is a good idea is another question — VonC is correct when he writes that this is not the normal use case for a version control system.
Even if it may be possible, it is not advisable! (for Mercurial or any other VCS)
A Version Control System is not made to record binaries (mainly because it quickly grows out of proportion, take a all lot of disk space, and has no efficient way to be stored in delta)
You should record the configuration need for each version you tag.
That can be a text file, or a maven pom for instance. Anything that allow an external mechanism (like maven) to download and locally store for you the right dependencies.
That means your patch will include changes to that text file (pom for instance), as well as the rest of the code modifications.
This is not quite a specifc question, and more me like for a criticism of my current approach.
I would like to include the program version number in the program I am developing. This is not a commercial product, but a research application so it is important to know which version generated the results.
My method works as follows:
There is a "pre-commit" hook in my .hg/hgrc file link to version_gen.sh
version_gen.sh consists solely of:
hg parent --template "r{rev}_{date|shortdate}" > version.num
In the makefile, the line version="%__VERSION__% in the main script is replaced with the content of the version.num file.
Are there better ways of doing this? The only real short coming I can see is that if you only commit a specfic file, version.num will be updated, but it won't be commited, and if I tried to add always committing that file, that would result in an infite loop (unless I created some temp file to indicate I was already in a commit, but that seems ugly...).
The problem
As you've identified, you've really created a Catch-22 situation here.
You can't really put meaningful information in the version.num file until the changes are committed and because you are storing version.num in the repository, you can't commit changes to the repository until you have populated the version.num file.
My solution
What I would suggest is:
Get rid of the "pre-commit" hook and hg forget the version.num file.
Add version.num to your .hgignore file.
Adjust version_gen.sh to consist of:
hg parent --template "r{node|short}_{date|shortdate}" > version.num
In the makefile, make sure version_gen.sh is run before version.num is used to set the version parameter.
My reasons
As #Ry4an suggests, getting the build system to insert revision information into the software at build time, using information from the Version Control System is a much better option. The only problem with this is if you try to compile the code from an hg archive of the repository, where the build system cannot extract the relevant information.
I would be inclined to discourage this however - in my own build system, the build failed if revision information couldn't be extracted.
Also, as #Kai Inkinen suggests, using the revision number is not portable. Rev 21 on one machine might be rev 22 on another. While this may not be a problem right now, it could be in the future, if you start colaborating with other people.
Finally, I explain my reasons for not liking the Keyword extension in a question of mine, which touches on similar issues to your own question:
I looked at Mercurials Keyword extension, since it seemed like the obvious solution. However the more I looked at it and read peoples opinions, the more that I came to the conclusion that it wasn't the right thing to do.
I also remember the problems that keyword substitution has caused me in projects at previous companies. ...
Also, I don't particularly want to have to enable Mercurial extensions to get the build to complete. I want the solution to be self contained, so that it isn't easy for the application to be accidentally compiled without the embedded version information just because an extension isn't enabled or the right helper software hasn't been installed.
Then in comments to an answer which suggested using the keyword extension anyway:
... I rejected using the keyword extension as it would be too easy to end up with the string "$Id$" being compiled into the executable. If keyword expansion was built into mercurial rather than an extension, and on by default, I might consider it, but as it stands it just wouldn't be reliable. – Mark Booth
A don't think that there can be a more reliable solution. What if someone accidentally damages .hg or builds not from a clone but from an archive? – Mr.Cat
#Mr.Cat - I don't think there can be a less reliable solution than the keywords extension. Anywhere you haven't explicitly enabled the extension (or someone has disabled it) then you get the literal string "$ID$" compiled into the object file without complaint. If mercurial or the repo is damaged (not sure which you meant) you need to fix that first anyway. As for hg archive, my original solution fails to compile if you try to build it from an archive! That is precisely what I want. I don't want any source to be compiled into our apps without it source being under revision control! – Mark Booth
What you are trying to do is called Keyword Expansion, which is not supported in Mercurial core.
You can integrate that expansion in make file, or (simpler) with the Keyword extension.
This extension allows the expansion of RCS/CVS-like and user defined keys in text files tracked by Mercurial.
Expansion takes place in the working directory or/and when creating a distribution using "hg archive"
That you use a pre-commit hook is what's concerning. You shouldn't be putting the rest of version_gen.sh into the source files thesemves, just into the build/release artifacts which you can do more accurately with an 'update' hook.
You don't want the Makefile to actually change in the repo with each commit, that just makes merges hell. You want to insert the version after checking out the files in advance of a build, which is is what an update hook does.
In distributed systems like Mercurial, the actual "version number" does not necessarily mean the same thing in every environment. Even if this is a single person project, and you are really careful with having only your central repo, you would still probably want to use the sha1-sum instead, since that is truly unique for the given repository state. The sha1 can be fetched through the template {node}
As a suggestion, I think that a better workflow would be to use tags instead, which btw are also local to your repository until you push them upstream. Don't write your number into a file, but instead tag your release code with a meaningful tag like
RELEASE_2
or
RELEASE_2010-04-01
or maybe script this and use the template to create the tag?
You can then add the tag to your non-versioned (in .hgignore) version.num file to be added into the build. This way you can give meaningful names to the releases and you tie the release to the unique identifier.
So I'm pretty new to version control but I'm trying to use Mercurial on my Mac to keep a large Python data analysis program organized. I typically clone my main repository, tweak the clone's code a bit, and run the code on my data. If the changes were successful I commit and eventually push the changes back to my main repository. I guess that's a pretty typical workflow under version control.
My problem is that my code is run on the command-line, with several command-line arguments that refer to data files in the current working directory (and I have many such directories I need to test the code in, and they're outside of version control). So before using Mercurial I just kept my code in one ~/bin directory which was part of my PATH environment variable. Now, with version control, I need to either (1) after each edit, copy my current clone's executables to the ~/bin directory before running the code on the command line, or (2) each time I clone my code, add my current clone's path to the PATH, or (3) specify the entire/path/to/my/programs on the command line each time I run the code. None of these are very convenient, and I'm left feeling like there must be an elegant solution that I just don't know. Maybe something involving Mercurial's hooks? I want my under-revision code to be runnable on the command line between commits, so this seemed to rule out hooks, but I don't know... Many thanks for any suggestions!
Ry4an's answer is good if you want to continue with the multiple-clones workflow. But it's also worth being aware that Mercurial's powerful enough to allow you most of the benefits of that workflow without ever leaving your single "main" repo. I.e. you can create branches (named or anonymous) for experimental features, easily "hg update" to whatever version of the code you want to test, even use the mq extension to prune branches that didn't work out.
What I do in such a case is set up a two deep chain of symlinks to my binary in my current clone. For example I'll have:
/usr/bin/myappname
which is a symlink to
/home/me/repos/CURRENT/bin/myappname
where /home/me/repos/CURRENT is a symlink to whatever my current working clone is, for example:
/home/me/repos/myproject-expirment
After setting up the initial /usr/bin/myappname symlink all I have to do is update the CURRENT symlink when I create a new clone on which I'm working.
What exactly does the word patch mean when referring to 'submitting a patch'?
I've seen this used a lot, especially in the open source world. What what does it mean and what exactly is involved in submitting a patch?
It's a file with a list of differences between the code files that have changed. It's usually in the format generated by doing a diff -u on the two files. Most version control systems allow the easy creation of patches but it's generally in that same format.
This allows the code change to be easily applied to someone else's copy of the source code using the patch command.
For example:
Let's say I have the following code:
<?php
$foo = 0;
?>
and I change it to this:
<?php
$bar = 0;
?>
The patch file might look like this:
Index: test.php
===================================================================
--- test.php (revision 40)
+++ test.php (working copy)
## -3,7 +3,7 ##
<?php
- $foo = 0;
+ $bar= 0;
?>
Richard Jones, a developer at Red Hat, has a nice little primer on submitting code to open source projects which covers making and submitting patches.
A patch is usually a file that contains information how to change something (very often to fix a bug, but could also be an enhancement). There are different kind of patches.
A source code patch contains information how one or multiple source code files need to be modified. You can easily generate them using the diff command and you can apply them using the patch command (on Linux/UNIX systems these commands are standard).
However, there are also binary patches. A binary patch contains information how certain bytes within a binary need to be changed. Binary patches are, of course, rare in the OpenSource world, but in the early days of computers I saw them a lot to modify shipped binaries (usually to work around a bug).
Submitting a patch means you have locally fixed something and now you send the file to someone, so he can apply this patch to his local copy or to a public copy on the web, thus other users can benefit of the fix.
Patches are also often used if you have some source code that almost compiles on a certain platform, but some tiny changes are necessary to really have it compile there. Of course you could take the source, modify it and offer the modified code for download. But what if the original source changes again (e.g. bugs get fixed or small enhancements were added)? Then you had to re-download the source, apply the changes again and offer the new modified version. It's a lot of work to keep your modified source up-to-date. Instead of modifying, you create a diff between the original and your modified copy and store it on your server. If now a user wants to download and compile the app from source, he can first download the latest & greatest version of the original source, then apply your patch (so it will compile) and always has the latest version, without you having to change the patch. A problem will only arise if the original source has been changed exactly in one of the places your patch modifies. In this case the system will refuse to apply the patch and a new patch needs to be made.
A patch is a file containing all of the necessary information to turn the maintainer's source tree in to your own. It's usually created by tools like diff or svn diff or git format-patch.
Traditionally, open-source projects accept submissions from normal schlubs in the form of patches so they don't have to give others commit access to their repositories.
A patch, ususally in the form of a .patch file, is a common flat file format for transmitting the differences between two sets of code files. So if you are working on an open source project, and make code changes to files, and want to submit that to the project owner to be checked in (usually because you don't have checkin rights), you would do so via a patch.
WinMerge has this functionality built in, as do many other tools like TortoiseSVN.
A patch file represents the difference between existing source and source you've modified. It is the primary means of adding features or fixing bugs in many projects.
You create a patch using the diff command (for example).
You can then submit this patch to the development mailing list and if it received well, then a committer will apply the patch (thus automatically applying your changes) and commit the code.
Patches are applied using the patch command.
Generally it implies submitting a unified diff file with the aggregate changeset for a feature. You can read more about patches on Wikipedia. Several version control systems (svn, git, etc.) will create a patch file for you based on a changeset.
1. n. A temporary addition to a piece of code, usually as a quick-and-dirty
remedy to an existing bug or misfeature. A patch may or may not work, and may or may not
eventually be incorporated permanently into the program. Distinguished from a diff
or mod by the fact that a patch is generated by more primitive means than the rest
of the program; the classical examples are instructions modified by using the front
panel switches, and changes made directly to the binary executable of a program
originally written in an HLL. Compare one-line fix.
See the entire definition in the jargon file here
Patch is also used in the act of updating system binaries. Microsoft sends out patches all the time but they aren't source code. They are .msp files that install improved binaries. As with all computer science terms, patch is overloaded.
I've always believed the term meant a bug fix, like a knee patch Mom used to put on your holey jeans.