I have a Jenkins build job with a Mercurial trigger on the default branch, which works fine for building "release candidates". This job then starts a smoke test job.
We use a branch-per-feature branching scheme, so that at any given time there could be up to a dozen different active branches in Mercurial (but the active branches change regularly).
I would like a Jenkins job triggered by changes to any branch, that will then build and run the smoke tests for all branches that need updating. Each time we do a build, we should create artifacts named to match the branch.
I saw a suggestion in another answer of using "tip" instead of the branch name in the Mercurial trigger - this is a possibility, but I think it would fall into the "mostly works" category. The trigger is polling, so if changes to more than one branch occur within the polling interval, then a branch update could be missed.
I could create a new job each time a branch is created, but due to the dynamic nature of our branches, that would be a lot of ongoing work.
If you decide to go with the job for every branch approach, the following tools can make the task a bit more manageable:
jenkins-build-per-branch (supports git)
jenkins-autojobs (supports git, mercurial and svn)
I think you'll need to customize: the top-level polling job (tuned to tip) runs a custom script that determines branches that have changed or have been added. It then will use Jenkins API to start a job parameterized by the branch name. That parameter can be used in your job to customize everything you need by the branch name (including the artifacts).
Related
I've read similar posts of SO, the official Hg guide, many articles and guides, and it's still unclear to me what the best Hg workflow is for developing by feature. Maybe some of the articles on the web are years old and don't include the latest features from Hg. Obviously there's also a lot of options in how to approach it.
I'm a solo developer working on a project where a request for a fix or feature will be submitted to me as a task, like "Task #546 - Change whatever". Some of these tasks take a few days, and some tasks are open for months and there's often up to a dozen going at one time. A task is shipped to the final site after it's approved by the requestor.
The Hg guide seems to recommend having a clone per feature. But having a dozen full copies of the site on my drive seems... wasteful? I'm up for trying it, but I've seen other suggestions that make more sense. Do people really have a dozen copies of each site on their dev machine at a time?
Name branches at first sound like what I'd want, where's I'd name a branch "task 546" work on it, then merge it back in when it ships. I see a lot of discussion about the permanence of the names and having so many branches (though they can be closed). Some people seem to care about that and some don't. I don't know Hg enough to know if I care or not, and what the downsides really mean.
Finally, bookmarks seem to be popular with the more recent articles and it would seem that the best way to use them would be to set a bookmark like "task 546" then when you merge it back into the main branch using a commit message that has the task number in it to keep a reference to what was being done in the work. I know you can delete bookmarks, but it's unclear if I'd need to do this after the final merge.
So my thought for a combined approach is to have:
one repo
three named branches:
"default" which holds the released version of the site
"dev" on which I do feature development
"test" which would hold all of the tasks being reviewed by the client
on the "dev" branch I would use bookmarks for each of the tasks that I'm working on, so I'd have a head for each task
My workflow for a task/feature would be to:
Update to the main line of the "dev" named branch
Start a new branch using a bookmark for the task "task #123"
Commit changes until I'm ready for the client to review
Merge "task #123" into the "test" branch
Deploy "test" to the test server
Repeat the commit, merge, deploy until ready for production
When approved, merge with the main line of the "dev" branch with a commit message that includes the task name
Merge "dev" into the "default" branch.
Deploy the "default" branch to the live server
Merge "default" into the open feature branches
Thoughts? Would I be better off just having a clone for each feature, and a "live" and "test" repo that I push to?
Edit: I see from some links that I should be doing the development off of "default" so my first change to my listed process would be to use a name "production" branch instead of a named "dev" branch.
Bookmarks-style of branching (Git-like "branches") works poorly in, at least, two common cases
Cross-tasks merges in the process of development
Time-back machine, when you'll want to see "the whole history of changes for task#123" (you can do it visually and, with some grimaces and jumping, using revsets)
While using named branches haven't such problems and, btw, workflow with named branches (and only default branch as aggregation point) will be less complex and more logical way
Default contain only mergesets from task-branches, head of default is always "stable version"
Heads of named branches are WIP; branches, merged to default - finished (and accepted by customer - see below) work
Default, merged to task-branch (after development of task, before merging task-branch to default) is equivalent of your "test": without affecting mainline you can test final state of feature, integrated into your stable app, show results to customer
Accepted work added to stable mainline by merging named branch to default
History (full history) of changes for every task in the past can be easy restored by using single, easy, short, memorable revset for log: -r "branch(TASK-ID)"
I like it. +1. This is the way I'd do it.
How do I configure Hudson/Jenkins (we're still using Hudson) to build all branches with the mercurial plugin? I have seen that we can create separate jobs/etc for specific branches, but we branch often enough that it would be good to just build all branches.
The only way I can think of (and I haven't tried it) is to have our build script [executed by Hudson on the slave] somehow figure out which branch is the latest commit and then hg update -r itself, rather than the mercurial plugin doing it.
I would love to find out a more elegant/correct way to do this!
I've also been looking for this support in the Jenkins Mercurial plugin (partly because it's supported by its Git plugin) but the following Jenkins tickets seem to indicate that this is yet not implemented:
https://issues.jenkins-ci.org/browse/JENKINS-11102
https://issues.jenkins-ci.org/browse/JENKINS-10558
One option if you have branches that are used regularly (ie, named branches for milestones or developers) is to create copies of your master build and simply point it to that branch.
For example on my team's build server we have production ('default' branch) and development ('dev' branch). Our builds are run by feeding the branch name as a parameter to the build script (so it's a simple, one-field change from a copy of the main job).
Obviously this won't scale beyond a handful of branches, but it might be enough for your needs.
I work on a project where we have Live, RC, and QA. Individual issues go to QA when they're ready where the QA team tests the ticket's functionality. Once the ticket passes, it's ready to move to RC. Before a release we take all of the issues that are ready and put them all on RC together where we make sure that tickets don't cause problems with each other. Once that's done all of RC moves to live together.
What is a good way to handle this with Mercurial? Our current process has some problems.
Our current process:
Live runs default, RC runs a branch created off of default (live) and QA runs it's own branch (trunk) that branched from default in the past.
Issues are branched from default, worked, and then merged to trunk. Once a ticket is ready to go to RC it gets merged into the upcoming release branch. When that gets tested the release branch gets merged into default and default gets merged back into trunk. The problem we're running into is that after a while something happens and everything gets conflicts merging to trunk. If we resolve conflicts to trunk in that merge it tends to break trunk much faster, if we have conflicts we merge default into our branch and resolve them in that commit. This usually works but after a few weeks or months trunk seems to break and we can no longer resolve the conflicts.
Update:
Our process works such that ticket A might move into the QA environment and stay there for a while. Ticket B might be developed, moved to QA, QAed and on RC and released before ticket A ever leaves the QA environment. That is not something that can be changed right now. I'm looking for a solution that fits our needs. I have the ability to influence our implementation of the process at the repository level, but not the ability to modify the over all process in a significant way right now.
I think your best bet is reviewing Mercurial's suggested workflows. I can tell you from my own experience with a structured release cycle that had code (features, enhancements, and bug fixes) going to a QA environment for verification and prepping releases, then branching from those release tags for emergency patches, Mercurial fits nicely.
For my steps below, "trunk" is just what others might call mainline. There isn't any term "trunk" in Mercurial. You just have the repository you are working from, which is either just your local repository or a clone of another repository (still local).
trunk is where everyone generally works.
When a particular change is completed, it is pushed to a central server (still trunk and a central server is optional, just makes the workflow easier with continuous integration).
From the central server repository, a build is done and pushed to the QA server.
QA tests the functionality and should also generally test limited interaction outside the known change.
If change passes QA testing, a release manager may choose to tag that changeset as the version number to be released (hg tag "name of tag").
Once code is tagged and ready for a release, another build can be done to push to a user acceptance test environment (UAT), which mimics production configuration.
If UAT passes, a final build can be done to push into production. Note that you can save some build steps by using separate "build" repositories that hold only pre-built copies of your software. That way, only a single build needs to be done if QA and UAT testing passes before pushing to production.
If a defect is found in the production release, a branch can be created from the tagged revision that was released (say 1.0.0) using hg update "tag name" then hg branch "name of branch" (which should probably match the tag name).
While "on" the branch revision, fix the bug, commit, merge it to "trunk".
Do a build from the latest revision "on" the branch and push that to QA.
If testing is successful, that build can be moved to UAT and production as necessary and you can feel confident that you are only releasing that single bug fix.
All the while, new functionality, features, and non-critical bug fixes can be done in "trunk", continuing with the "next" version of the code to be released on a schedule.
We use Jenkins for doing incremental builds of our project on each commit to the SCM. We would like to get separate builds for every single commit. However, the naive approach (setup SCM and use post-commit hooks to trigger a build) exhibits problem in the following scenario:
Build is triggered.
While build takes place (it can take up to several minutes) two separate commits to the SCM are made by two developers.
One new build is triggered. It receives changes from both of the commits, made during previous build.
This "race condition" complicates finding which one of the commits has broken the build/introduced warnings.
The currently employed solution is checking for changes in one job ("scheduler job") and triggering another job to do the actual checkout and build.
Are there any proper solutions to this problem?
Not yet, there's a Feature Request covering this kind of build, but it's still open: Issue 673
Maybe it misses the point, but we have a pretty nice build process running here.
We use git as our source control system
We use gerrit as our review tool
We use the gerrit trigger to run builds on our jenkins server
We check for changes on the develop branch to run jenkins when a changeset is merged
In short the ideal developer day is like this
developer 1 stars a new branch to do his changes, based on our main develop branch
The developer 1 commits as often as he likes
developer 1 thinks he finished his job, he combines his changes into one change and pushes it to gerrit
A new gerrit change is created and jenkins tries to build exactly this change
When there are no errors during the build, a review is made on this change
When the review is submited, the changeset is merged into the develop branch of the main repository (No change is merged into the develop branch, without review)
Jenkins builds the merged version to be sure, that there are no merge errors
no developer 2 joins the party and tries to do some work
the process is exactly the same both start working, in there branches. Developer 1 is faster and his changes are merged into the develop branch. Now, before developer 2 can publish his changes he has to rebase his changes on top of the changes made by developer 1.
So we are sure, that the build process is triggered for every change made to our codebase.
We use this for our C# development - on windows not on linux
I don't believe what you'd like to do is possible. The "quiet period" mentioned by Daniel Kutik is actually used to tell Hudson/Jenkins how much time to wait, in order to allow other commits to the same project to be picked up. Meaning -- if you set this value to 60 seconds and you've made a commit, it will wait for a minute before starting a new build, allowing time for other commits to be picked up as well (during that one minute).
If you use the rule "NO COMMIT on a broken build‏" and take it to it's logical conclusion, you actually end up with "No commit on a broken build or a build in progress", in which case the problem you describe goes away.
Let me explain. If you have two developers working on the same project and both of them try to commit (or push if you're using DVCS). One of them is going to succeed and and they other will fail and need to update before the commit.
The developer who had to do the update knows from the commit history, that the other commit was recent and thus a build in progress (even if it hasn't checked out yet). They don't know if that build is broken yet of not, so the only safe option is to wait and see.
The only thing that would stop you from using the above approach is if the build takes so long, in which case you might find that your developers never get a chance to commit (it's always building). This is then a driver to split up your build into a pipeline of multiple steps, so that the Post Commit job takes no more than 5 minutes, but is ideally 1 minute.
I think what might help, is to set the Quiet Period (Jenkins > Manage Jenkins > Configure System) to 0 and the SCM Polling to a very short time. But even during that short interval there could be two commits. As of now Jenkins does not have the feature to split build into single builds on multiple SVN commit.
Here is a tutorial about that topic: Quiet Period Feature.
As pointed out by someone in Issue 673 you could try starting a parametrized build with the parameter being the actual git commit you want to build. This in combination with a VCS commit hook.
We got all psyched about from from svn to hg and as the development workflow is more or less flushed out, here remains the most difficult part - staging and integration system.
Hopefully this question goes a bit further then your common 'how do I move from xxx to Mercurial'. Please forgive long and probably poorly written question :)
We are web shop that does a lot of projects(mainly PHP and Zend), so we have one huge svn repo, with like 100+ folders, each representing a project with it's own tags,branches and trunk of course. On our integration and testing server(where QA and clients look at work results and test stuff) everything is pretty much automated - Apache is set to pick up new projects automatically creating vhost for each project/trunk; mysql migration scripts right there in trunk too and developers can apply them through simple web-interface. Long story short our workflow is this now:
Checkout code, do work, commit
Run update on the server via web interface(this basically does svn up on server on a particular project and also run db-migration script if needed)
QA changes on the server
This approach is certainly suboptimal for large projects when we have 2+ developers working on the same code. Branching in svn was only causing more headaches, well, hence moving to Mercurial. And here is where the question lies - how does one organize efficient staging/integration/testing server for this type of work(where you have many projects, say single developer could be working on 3 different projects in 1 day).
We decided to have 'default' branch tracking production essentially and then make all changes in individual branches. In this case though how can we automate staging updates for each branch? If earlier for one project we almost always were working on trunk, so we needed one DB, one vhost, etc. now we potentially talking about N-databases per project, N-vhost configs and etc. Then what about CI stuff(such as running phpDocumentor and/or unit tests)? Should it only be done on the 'default'? On branches?
I wonder how other teams solve this issue, perhaps some best practices that we're not using or overlooking?
Additional notes:
Probably worth mentioning that we've picked Kiln as a repo hosting service(mostly since we're using FogBugz anyway)
This is by no means the complete answer you'll eventually pick, but here are some tools that will likely factor into it:
repositories without working directories -- if you clone -U or hg update null you get a repository with no working directory (only the .hg). They're better on the server because they take up less room and no one is tempted to edit there
changegroup hooks
For that last one the changegroup hook runs whenever one or more changesets arrive via push or pull and you can have it do some interesting things such as:
push the changesets on to another repo depending on what has arrived
update the receiving repo's working directory
For example one could automate something like this using only the tools described above:
developer pushes five changesets to central-repo/project1/main
last changeset is on branch 'my-experiment' so csets are automatually re-pushed to optionally created repo central-repo/project1/my-experiment
central-repo/project1/my-experiment automatically does hg update tip which is certain to be on the my-expiriment branch
central-repo/project1/my-experiment automatically runs tests in its working dir and if they pass does a 'make dist' that deploys, which might set up database and vhost too
The biggie, and chapter 10 in the mercurial book covers this, is to not have the user waiting on that process. You want the user to push to a repo that contains possibly-okay-code and the automated processed do the CI and deploy work, which if it passes ends up being a likely-okay repo.
In the largest mercurial setup in which I've worked (20 or so developers) we got to the point where our CI system (Hudson) was pulling from the maybe-ok repos for each periodically then building and testing, and handling each branch separately.
Bottom line: all the tools you need to setup whatever you'd like probably already exist, but gluing them together will be one-off sort of work.
What you need to remember is that DVCS (vs. CVCS) introduces another dimension to versioning:
You don't have to rely anymore only on branching (and get a staging workspace from the right branch)
You now have with DVCS the publication workflow (push/pull between repo)
Meaning your staging environment is now a repo (with the full history of the project), checked out at a certain branch:
Many developers can push many different branches to that staging repo: the reconciliation process can be done in isolation within that repo, in a "main" branch of your choice.
Or they can pull that staging branch in their repo and test things out before pushing back.
From Joel's tutorial on Mercurial HgInit
A developer don't necessary have to commit for other to see: the publication process in a DVCS allows for him/her to pull the staging branch first, reconcile any conflict locally, and then push to the staging repo.