We have an Azure Pipeline that merges into a repository that converts .json files representing customer orders into C# objects. Naturally, if the design or naming of these C# objects ever changes, the old orders will become unusable, so we run a script 'Migrating' all these outdated .jsons to conform to the new model.
Our current pipeline that merges dev into production Migrates our .jsons, and we run a PowerShell unit test script after the pipeline's completion to ensure that the .jsons have successfully Migrated. We'd like to place this test into the pipeline itself, but there are two conditions we'd prefer to meet.
If the Test fails, not only abort the merge, but revert the .jsons to their un-Migrated versions.
Give us the option to continue the merge anyway, in the event that the website encounters an error so critical and urgent we are willing to bear the loss of a few quotes.
Are these conditions feasible?
According to your description, you may consider using Build validation as the Branch policies and settings.
Basically, let's assume your production code is in the Production branch; then you can create a Dev branch and push your new commits into the Dev branch. When setting the Build validation policy on the Production branch, the PR request will not be completed if the build fails, which contains the unit test. Therefore the new code from Dev branch will not be merged into the Production branch.
In the meanwhile, other branch policies may also help you with the code version control. Hope the following documents can help as well.
Require a minimum number of reviewers
Check for linked work items
Check for comment resolution
Limit merge types
Related
Is there any way to prevent to move back to an older version?
I mean, I want to always do hg up to forward and never to back.
Make use of the hooks which mercurial offers and implement a hook to the update command which compares the desired version with the currently checked-out version. Fail the hook, when the desired version is older than the currently checked-out one. See the docs for the available hooks and some examples.
That said, it might be an undesirable constraint on usage of a VCS. If it is about deployment on a production syste, that's more sane; yet then you do not need to copy the whole repository with its history, but just the current version to the deployment target.
For my purpose of deployment, I only check whether the build is set to be a release build (tag, or manually selected in jenkins) or if it is HEAD of a branch. In those cases I trigger the deployment to production after the build passed regression tests: https://github.com/OpenTTD/nml/blob/master/.devzone/build/jenkins_postbuild.sh
I have a Jenkins build job with a Mercurial trigger on the default branch, which works fine for building "release candidates". This job then starts a smoke test job.
We use a branch-per-feature branching scheme, so that at any given time there could be up to a dozen different active branches in Mercurial (but the active branches change regularly).
I would like a Jenkins job triggered by changes to any branch, that will then build and run the smoke tests for all branches that need updating. Each time we do a build, we should create artifacts named to match the branch.
I saw a suggestion in another answer of using "tip" instead of the branch name in the Mercurial trigger - this is a possibility, but I think it would fall into the "mostly works" category. The trigger is polling, so if changes to more than one branch occur within the polling interval, then a branch update could be missed.
I could create a new job each time a branch is created, but due to the dynamic nature of our branches, that would be a lot of ongoing work.
If you decide to go with the job for every branch approach, the following tools can make the task a bit more manageable:
jenkins-build-per-branch (supports git)
jenkins-autojobs (supports git, mercurial and svn)
I think you'll need to customize: the top-level polling job (tuned to tip) runs a custom script that determines branches that have changed or have been added. It then will use Jenkins API to start a job parameterized by the branch name. That parameter can be used in your job to customize everything you need by the branch name (including the artifacts).
I work on a project where we have Live, RC, and QA. Individual issues go to QA when they're ready where the QA team tests the ticket's functionality. Once the ticket passes, it's ready to move to RC. Before a release we take all of the issues that are ready and put them all on RC together where we make sure that tickets don't cause problems with each other. Once that's done all of RC moves to live together.
What is a good way to handle this with Mercurial? Our current process has some problems.
Our current process:
Live runs default, RC runs a branch created off of default (live) and QA runs it's own branch (trunk) that branched from default in the past.
Issues are branched from default, worked, and then merged to trunk. Once a ticket is ready to go to RC it gets merged into the upcoming release branch. When that gets tested the release branch gets merged into default and default gets merged back into trunk. The problem we're running into is that after a while something happens and everything gets conflicts merging to trunk. If we resolve conflicts to trunk in that merge it tends to break trunk much faster, if we have conflicts we merge default into our branch and resolve them in that commit. This usually works but after a few weeks or months trunk seems to break and we can no longer resolve the conflicts.
Update:
Our process works such that ticket A might move into the QA environment and stay there for a while. Ticket B might be developed, moved to QA, QAed and on RC and released before ticket A ever leaves the QA environment. That is not something that can be changed right now. I'm looking for a solution that fits our needs. I have the ability to influence our implementation of the process at the repository level, but not the ability to modify the over all process in a significant way right now.
I think your best bet is reviewing Mercurial's suggested workflows. I can tell you from my own experience with a structured release cycle that had code (features, enhancements, and bug fixes) going to a QA environment for verification and prepping releases, then branching from those release tags for emergency patches, Mercurial fits nicely.
For my steps below, "trunk" is just what others might call mainline. There isn't any term "trunk" in Mercurial. You just have the repository you are working from, which is either just your local repository or a clone of another repository (still local).
trunk is where everyone generally works.
When a particular change is completed, it is pushed to a central server (still trunk and a central server is optional, just makes the workflow easier with continuous integration).
From the central server repository, a build is done and pushed to the QA server.
QA tests the functionality and should also generally test limited interaction outside the known change.
If change passes QA testing, a release manager may choose to tag that changeset as the version number to be released (hg tag "name of tag").
Once code is tagged and ready for a release, another build can be done to push to a user acceptance test environment (UAT), which mimics production configuration.
If UAT passes, a final build can be done to push into production. Note that you can save some build steps by using separate "build" repositories that hold only pre-built copies of your software. That way, only a single build needs to be done if QA and UAT testing passes before pushing to production.
If a defect is found in the production release, a branch can be created from the tagged revision that was released (say 1.0.0) using hg update "tag name" then hg branch "name of branch" (which should probably match the tag name).
While "on" the branch revision, fix the bug, commit, merge it to "trunk".
Do a build from the latest revision "on" the branch and push that to QA.
If testing is successful, that build can be moved to UAT and production as necessary and you can feel confident that you are only releasing that single bug fix.
All the while, new functionality, features, and non-critical bug fixes can be done in "trunk", continuing with the "next" version of the code to be released on a schedule.
We use Jenkins for doing incremental builds of our project on each commit to the SCM. We would like to get separate builds for every single commit. However, the naive approach (setup SCM and use post-commit hooks to trigger a build) exhibits problem in the following scenario:
Build is triggered.
While build takes place (it can take up to several minutes) two separate commits to the SCM are made by two developers.
One new build is triggered. It receives changes from both of the commits, made during previous build.
This "race condition" complicates finding which one of the commits has broken the build/introduced warnings.
The currently employed solution is checking for changes in one job ("scheduler job") and triggering another job to do the actual checkout and build.
Are there any proper solutions to this problem?
Not yet, there's a Feature Request covering this kind of build, but it's still open: Issue 673
Maybe it misses the point, but we have a pretty nice build process running here.
We use git as our source control system
We use gerrit as our review tool
We use the gerrit trigger to run builds on our jenkins server
We check for changes on the develop branch to run jenkins when a changeset is merged
In short the ideal developer day is like this
developer 1 stars a new branch to do his changes, based on our main develop branch
The developer 1 commits as often as he likes
developer 1 thinks he finished his job, he combines his changes into one change and pushes it to gerrit
A new gerrit change is created and jenkins tries to build exactly this change
When there are no errors during the build, a review is made on this change
When the review is submited, the changeset is merged into the develop branch of the main repository (No change is merged into the develop branch, without review)
Jenkins builds the merged version to be sure, that there are no merge errors
no developer 2 joins the party and tries to do some work
the process is exactly the same both start working, in there branches. Developer 1 is faster and his changes are merged into the develop branch. Now, before developer 2 can publish his changes he has to rebase his changes on top of the changes made by developer 1.
So we are sure, that the build process is triggered for every change made to our codebase.
We use this for our C# development - on windows not on linux
I don't believe what you'd like to do is possible. The "quiet period" mentioned by Daniel Kutik is actually used to tell Hudson/Jenkins how much time to wait, in order to allow other commits to the same project to be picked up. Meaning -- if you set this value to 60 seconds and you've made a commit, it will wait for a minute before starting a new build, allowing time for other commits to be picked up as well (during that one minute).
If you use the rule "NO COMMIT on a broken build‏" and take it to it's logical conclusion, you actually end up with "No commit on a broken build or a build in progress", in which case the problem you describe goes away.
Let me explain. If you have two developers working on the same project and both of them try to commit (or push if you're using DVCS). One of them is going to succeed and and they other will fail and need to update before the commit.
The developer who had to do the update knows from the commit history, that the other commit was recent and thus a build in progress (even if it hasn't checked out yet). They don't know if that build is broken yet of not, so the only safe option is to wait and see.
The only thing that would stop you from using the above approach is if the build takes so long, in which case you might find that your developers never get a chance to commit (it's always building). This is then a driver to split up your build into a pipeline of multiple steps, so that the Post Commit job takes no more than 5 minutes, but is ideally 1 minute.
I think what might help, is to set the Quiet Period (Jenkins > Manage Jenkins > Configure System) to 0 and the SCM Polling to a very short time. But even during that short interval there could be two commits. As of now Jenkins does not have the feature to split build into single builds on multiple SVN commit.
Here is a tutorial about that topic: Quiet Period Feature.
As pointed out by someone in Issue 673 you could try starting a parametrized build with the parameter being the actual git commit you want to build. This in combination with a VCS commit hook.
We got all psyched about from from svn to hg and as the development workflow is more or less flushed out, here remains the most difficult part - staging and integration system.
Hopefully this question goes a bit further then your common 'how do I move from xxx to Mercurial'. Please forgive long and probably poorly written question :)
We are web shop that does a lot of projects(mainly PHP and Zend), so we have one huge svn repo, with like 100+ folders, each representing a project with it's own tags,branches and trunk of course. On our integration and testing server(where QA and clients look at work results and test stuff) everything is pretty much automated - Apache is set to pick up new projects automatically creating vhost for each project/trunk; mysql migration scripts right there in trunk too and developers can apply them through simple web-interface. Long story short our workflow is this now:
Checkout code, do work, commit
Run update on the server via web interface(this basically does svn up on server on a particular project and also run db-migration script if needed)
QA changes on the server
This approach is certainly suboptimal for large projects when we have 2+ developers working on the same code. Branching in svn was only causing more headaches, well, hence moving to Mercurial. And here is where the question lies - how does one organize efficient staging/integration/testing server for this type of work(where you have many projects, say single developer could be working on 3 different projects in 1 day).
We decided to have 'default' branch tracking production essentially and then make all changes in individual branches. In this case though how can we automate staging updates for each branch? If earlier for one project we almost always were working on trunk, so we needed one DB, one vhost, etc. now we potentially talking about N-databases per project, N-vhost configs and etc. Then what about CI stuff(such as running phpDocumentor and/or unit tests)? Should it only be done on the 'default'? On branches?
I wonder how other teams solve this issue, perhaps some best practices that we're not using or overlooking?
Additional notes:
Probably worth mentioning that we've picked Kiln as a repo hosting service(mostly since we're using FogBugz anyway)
This is by no means the complete answer you'll eventually pick, but here are some tools that will likely factor into it:
repositories without working directories -- if you clone -U or hg update null you get a repository with no working directory (only the .hg). They're better on the server because they take up less room and no one is tempted to edit there
changegroup hooks
For that last one the changegroup hook runs whenever one or more changesets arrive via push or pull and you can have it do some interesting things such as:
push the changesets on to another repo depending on what has arrived
update the receiving repo's working directory
For example one could automate something like this using only the tools described above:
developer pushes five changesets to central-repo/project1/main
last changeset is on branch 'my-experiment' so csets are automatually re-pushed to optionally created repo central-repo/project1/my-experiment
central-repo/project1/my-experiment automatically does hg update tip which is certain to be on the my-expiriment branch
central-repo/project1/my-experiment automatically runs tests in its working dir and if they pass does a 'make dist' that deploys, which might set up database and vhost too
The biggie, and chapter 10 in the mercurial book covers this, is to not have the user waiting on that process. You want the user to push to a repo that contains possibly-okay-code and the automated processed do the CI and deploy work, which if it passes ends up being a likely-okay repo.
In the largest mercurial setup in which I've worked (20 or so developers) we got to the point where our CI system (Hudson) was pulling from the maybe-ok repos for each periodically then building and testing, and handling each branch separately.
Bottom line: all the tools you need to setup whatever you'd like probably already exist, but gluing them together will be one-off sort of work.
What you need to remember is that DVCS (vs. CVCS) introduces another dimension to versioning:
You don't have to rely anymore only on branching (and get a staging workspace from the right branch)
You now have with DVCS the publication workflow (push/pull between repo)
Meaning your staging environment is now a repo (with the full history of the project), checked out at a certain branch:
Many developers can push many different branches to that staging repo: the reconciliation process can be done in isolation within that repo, in a "main" branch of your choice.
Or they can pull that staging branch in their repo and test things out before pushing back.
From Joel's tutorial on Mercurial HgInit
A developer don't necessary have to commit for other to see: the publication process in a DVCS allows for him/her to pull the staging branch first, reconcile any conflict locally, and then push to the staging repo.