We got all psyched about from from svn to hg and as the development workflow is more or less flushed out, here remains the most difficult part - staging and integration system.
Hopefully this question goes a bit further then your common 'how do I move from xxx to Mercurial'. Please forgive long and probably poorly written question :)
We are web shop that does a lot of projects(mainly PHP and Zend), so we have one huge svn repo, with like 100+ folders, each representing a project with it's own tags,branches and trunk of course. On our integration and testing server(where QA and clients look at work results and test stuff) everything is pretty much automated - Apache is set to pick up new projects automatically creating vhost for each project/trunk; mysql migration scripts right there in trunk too and developers can apply them through simple web-interface. Long story short our workflow is this now:
Checkout code, do work, commit
Run update on the server via web interface(this basically does svn up on server on a particular project and also run db-migration script if needed)
QA changes on the server
This approach is certainly suboptimal for large projects when we have 2+ developers working on the same code. Branching in svn was only causing more headaches, well, hence moving to Mercurial. And here is where the question lies - how does one organize efficient staging/integration/testing server for this type of work(where you have many projects, say single developer could be working on 3 different projects in 1 day).
We decided to have 'default' branch tracking production essentially and then make all changes in individual branches. In this case though how can we automate staging updates for each branch? If earlier for one project we almost always were working on trunk, so we needed one DB, one vhost, etc. now we potentially talking about N-databases per project, N-vhost configs and etc. Then what about CI stuff(such as running phpDocumentor and/or unit tests)? Should it only be done on the 'default'? On branches?
I wonder how other teams solve this issue, perhaps some best practices that we're not using or overlooking?
Additional notes:
Probably worth mentioning that we've picked Kiln as a repo hosting service(mostly since we're using FogBugz anyway)
This is by no means the complete answer you'll eventually pick, but here are some tools that will likely factor into it:
repositories without working directories -- if you clone -U or hg update null you get a repository with no working directory (only the .hg). They're better on the server because they take up less room and no one is tempted to edit there
changegroup hooks
For that last one the changegroup hook runs whenever one or more changesets arrive via push or pull and you can have it do some interesting things such as:
push the changesets on to another repo depending on what has arrived
update the receiving repo's working directory
For example one could automate something like this using only the tools described above:
developer pushes five changesets to central-repo/project1/main
last changeset is on branch 'my-experiment' so csets are automatually re-pushed to optionally created repo central-repo/project1/my-experiment
central-repo/project1/my-experiment automatically does hg update tip which is certain to be on the my-expiriment branch
central-repo/project1/my-experiment automatically runs tests in its working dir and if they pass does a 'make dist' that deploys, which might set up database and vhost too
The biggie, and chapter 10 in the mercurial book covers this, is to not have the user waiting on that process. You want the user to push to a repo that contains possibly-okay-code and the automated processed do the CI and deploy work, which if it passes ends up being a likely-okay repo.
In the largest mercurial setup in which I've worked (20 or so developers) we got to the point where our CI system (Hudson) was pulling from the maybe-ok repos for each periodically then building and testing, and handling each branch separately.
Bottom line: all the tools you need to setup whatever you'd like probably already exist, but gluing them together will be one-off sort of work.
What you need to remember is that DVCS (vs. CVCS) introduces another dimension to versioning:
You don't have to rely anymore only on branching (and get a staging workspace from the right branch)
You now have with DVCS the publication workflow (push/pull between repo)
Meaning your staging environment is now a repo (with the full history of the project), checked out at a certain branch:
Many developers can push many different branches to that staging repo: the reconciliation process can be done in isolation within that repo, in a "main" branch of your choice.
Or they can pull that staging branch in their repo and test things out before pushing back.
From Joel's tutorial on Mercurial HgInit
A developer don't necessary have to commit for other to see: the publication process in a DVCS allows for him/her to pull the staging branch first, reconcile any conflict locally, and then push to the staging repo.
Related
I've been trying to find actual documentation for Sourcetree without much luck, so I figured I'd ask here. I'm a relative newb when it comes to version control, with my current project being my first effort. I'm using Sourcetree on Windows 7 as a frontend for Mercurial, I've got my development code on my local machine in C:\inetpub, and whenever I do a Commit I then switch Sourcetree over to the cloned repository on my backed up network drive and do a Pull of the changes I just Committed so I've got the development history backed up.
What I'm trying to wrap my head around is using Sourcetree to set up a Debug branch so I can fix bugs on the version of the code running on the production server while simultaneously doing development. Obviously a common need, but I can't grok it. I am expecting there to be two code locations, so I can pause in mid-edit on the Development branch, make changes to Debug, and them come back to my changes in Development and finish them up before merging in the changes to Debug. If that's not how it works that's presumably part of my confusion. Any suggestions on where I can find clarity? Pointers to existing tutorials and the like would be fine, I just haven't been having luck searching Google, and I haven't been able to locate any actual Sourcetree documentation.
NOTE: Based on responses I've seen to other questions I've read about Sourcetree and Mercurial, I'll state upfront I have no interest in discussing outside repository hosting unless somebody can explain why it will help with this problem.
Two things here:
You do not need to change repository to pull, you can also push from your local repository;
You do not need 2 code locations for switching from one branch to the other. It might help, for larger projects, but I suggest you get comfortable with Mercurial before doing so.
So, for number 1, there is a default source or remote repository for every local repo. It is either defined by the user or it is the source repo from where it was cloned. Whether you push or pull, it will default to that same source. You can push/pull to multiple sources as well, but this is not your case at the moment. In the normal workflow, just issue a hg push every time you commit, and your changes will be propagated to the other repo.
For number 2, a Mercurial repo, as you already know, can have multiple branches. When you commit a changeset, it is automatically done on the current branch. In SourceTree, click on the Branch button, and enter a new branch name. The next commit you'll do will be the head (and the start) of your new branch. After that, you can update back and forth between the 2 branches, and your code will change accordingly. That means that you can update at any time to the head of any branch, make some changes, commit, and then jump to another branch, and so on. So no, you do not need multiple repositories.
Normally, the proper practice is to have a default branch (default name is rarely changed!) where you have your current development source. For every issue or feature you are fixing/implementing, create a new branch from the default branch to put your new code. Once that branch has been reviewed and tested, merge it back in the default and close the former.
For your current development, if you need an additional stable and safe trunk, you can create a Production branch, which would be the stable code that will run on your server. Once you are satisfied with the default branch tests, you can then merge it in Production to include your changes up to that point.
As a convention, make sure your server is always running the code from the Production branch, for the more stable code. At least, that is what I understood from your initial question.
Coming originally from SVN, I am still new to Mercurial.
I am interested in creating an experimental workflow to see if I can rewrite a troubled feature from scratch. If my attempt fails though, I wish to delete the experimental workflow - abandoning the work — with nobody else ever seeing it.
The problem is though I still need to push changes of this experimental workflow across laptops and PCs and keep working for a couple of weeks. But still keep the option open to delete that branch and fall back to the main branch, without having any trace of the experimental branch.
Is something like this possible in Mercurial and how could I achieve this?
FYI, I am using mercurialeclipse plugin on Aptana Studio 3.0. (so I
use a UI but commands should be fine too)
After changeset is pushed to the central server (assuming you have one) - there is no way to remove it from there.
So the possible (but terribly inconvenient) solution for you now could be to create a personal separated repository and synchronize your devices using it. And if you like the result - you push to the shared central repo then. Otherwise you just delete the temporary repository.
With a Distributed Version Control System like Mercurial you can sync between any clone of a repository, not just a "central" one that all users have agreed to use.
Therefore, you can:
Clone the repository to private a share that the systems "experimenting" can access.
Clone to a USB key and move that between systems.
Use hg serve to start a web server for a local repository on a system and clone and pull that history to other systems.
Use hg bundle/unbundle to package up new history and email it to another system.
To abandon work, just delete all these extra clones and clone from the common "central" repository again.
Having myself found in a role of a build engineer and a systems guy I had to learn end figure out a few things - namely how to set up our infrastructure. Before I came on board they didn't have any. With this in mind please excuse me if I ask anything that should have been obvious.
We currently have 3 level distributed mercurial repositories: level one on each of developer machines, level two on central (trunk) server - only accessible from local network and the third layer on BitBucket. Workflow is as follows:
Local development: developer pulls change-sets from local network server. developer commits to local and pushes to our local server once merge conflicts are resolved. A scheduled script overnight backs everything up to BitBucket.
Working from home: developer pulls change-sets from BitBucket. Developer comits to their local repo and push to BitBucket.
TeamCity picks up repo changes from local network server for each project and runs a build / automated deploy to test environment.
The issue I'm hitting is scenario 2: at the moment if someone pushes something to bitbucket it's their responsibility to merge it back when they're back in office. And it's a bit of a time waster if it could be automated.
In case you're wondering, the reason we have a central repo on local network is because it would be slow to run TeamCity builds of BitBucket repositories. Haven't tested so it's just an educated guess.
Anyhow, the script that is scheduled and pushes all changes from central repository on local network just runs a "hg push" for each of repositories. It would have to do a pull / merge beforehand. How do I do this right?
This is what the pull would have to use switches for:
- update after pull
- in case of merge conflicts, always take newer file
- in case of error, send an email to system administrator(s)
- anything extra?
Please feel free to share your own setup as long as it's not vastly different to what's described.
UPDATE: In light of recent answers I feel an important aspect if the intended approach needs to be clarified. The idea is not to force merges on our local network central repo. Instead it should resolve merge conflicts in same was as per using HgWorkbench on developer machines with post pull: update + merge. All developers have this on by default so it should be OK.
So the script / batch file on server would do the following:
pull from BitBucket
update + auto merge
Any merge auto conflicts?
3.1 Yes -> Send an email to administrators to manually merge -> Break
3.2 No -> Cary on
Get outgoing changesets. Will push create multiple heads? (This might be redundant because of pull / update)
4.1 Yes -> Prompt administrators. Break.
4.2 No -> Push changes
Hope this clears things up a bit. Now, can this be done using hg commands alone - batch - or do I have to script it? Specifically can it send emails?
Thanks.
So all your work is available at BitBucket, right? Why not make BitBucket (as available from anywhere) you primary repo source and dropping your local servers? You can pull changes from BitBucket with TeamCity for your nightly builds and developers whould always work with current repo at BitBucket and resolve all merging problems themselves so there wouldn't be any subsequent merges for you.
I would not try to automatically merge the changes if they are conflicting, this will only lead to broken and inconsistent versions and "lost" changes causing confusion and chaos. Don't merge it automatically if it isn't clear how that merge should look like.
A better alternative would be to just keep the two heads around and push/pull them without merging. This way everybody still can get that version of the data he was working on from work/home. A manual merge will have to be done, but this can also be done at work or from home, enabling developers to resolve the issue from wherever they are. You can also send emails around in this scenario to make sure everybody is aware of the problem.
I guess that you could automize this using a script, I would try PowerShell if I were you. However, sometimes this may require manual change merges when there are conflicts (because when developers commit changes to both BB and local repos, these changes might be conflicting).
Right now, we have a small team of developers using TFS for our version control. I'm evaluating the possibility of us moving to a DVCS, and am wondering if we'd need to give up some of the stuff we like about our current system if we moved to DVCS, or if we can find a way to support it.
Right now we a Stable branch, and 1 branch for each developer (you can think of each dev's branch as a feature branch that is reused from feature to feature). The stuff we like is:
1) Each time a dev checks in changes to his branch, we do a build and test of everything on the servers, followed by automatic deployment of all projects to that developers test environment.
2) Merging from any dev's branch to Stable is done by me, so that I can have 1 last check on what is happening to our stable branch before committing the changes.
3) If I want to help a dev with something, I can just grab latest from their branch and look at it on my machine.
I'm trying to understand how this could work with a DVCS (specifically we are testing with Mercurial).
I'm hoping to be able to pull off something like this:
1) We setup a central repository, and we create Main and Release branches in addition to 1 branch for each developer.
2) All devs clone the repository to their local machine.
3) All work is done in their personal branch against the local repository.
4) When they are done, the would pull from the central repository, and perform a local forward integration merge from Main to to their branch, to integrate any changes that have happened in the past to Main.
5) They would then push their changes to the central repository.
6) Some CI service would pickup this change, causing a build/test/deploy-to-dev of all our projects in that branch.
7) If everything was ok the dev would shoot me an email saying their branch was ready for merging to Main.
8) I could then merge in their changes, either by somehow connecting directly to the remove repository, or by doing a pull->merge->push.
So to deal with our requests:
1) I'm assuming there is some CI tools that can watch a branch in Mercurial, and kick off a build/test/deploy process (like CC.Net).
2) I can still manage the final merge process from DevX to Main either by connecting to the remote repo, or by pulling, merging and pushing through my local repo.
3) I believe I could either pull changes directly from another dev's repo, or I could just pull from the central repo, and then update my working directory to work on their code.
So do I have this mostly right?
Yep, you have all that right. Regarding you final assumptions:
1) There's a Mercurial Source Control Block for CruiseControl.NET. Another CI server I've heard of in use with Mercurial is Jenkins.
2) Correct. For integration with Main, I would prefer pulling (from either) and merging on my own machine before pushing, rather than merging on the server.
3) Exactly so.
It sounds like your developers are fairly disciplined, but just in case you need better control certain aspects of your operations:
You can use hooks to issue warnings when someone tries to merge their branch to Main. In-process hooks have to be written in Python, but they have access to the Mercurial API that way. You could also place hooks on the server that reject pushes containing a merge to Main not done by certain users.
One way some organizations control integration is a pull-only scenario. Only a few developers can push to the official repository and other developers send them pull requests. The Mercurial book's Chapter 6 covers this a bit, too.
A branch per developer is good. A branch per feature is also useful, allowing each developer to work on multiple things in parallel, then merging each to their branch when done. They just have to remember to close that feature branch before doing so, so the branch name doesn't keep popping up. This can be done with with clones as well, but I find myself preferring named branches since I have to keep work/backup/laptop development clones all synced so I can work on whateve, whenever. I still do expendable work as a clone first.
I'm in a small distributed team using Mercurial for a central repository. We each clone it via ssh onto our own linux boxes. Our intent is to review each others' work before pushing changes up to the central repository, to help keep central's tip clean. What is a good way to share code between developers on different linux boxes? I'm new to Mercurial. The options I can think of (through reading, not experience) are:
1: Author commits all local changes and updates working clone with central's tip. Author uses hg bundle, if there's a way to specify which local revs to include in the bundle. (an experiment showed me "bundle" only grabs uncommited changes, even if there are previous local commits that central doesn't know about) Author gets bundle file to reviewer. Reviewer creates a new clean clone from central's tip, and imports the bundle into that clone.
or,
2: After both author and reviewer fetch from central's tip, author uses patch and reviewer imports the patch.
or,
3: Author pushes to reviewer or reviewer pulls from author (but how, exactly? What I read is only about pushing and pulling to/from the original served repository, and/or on the same box instead of between different linux boxes.)
4: Forget reviewing the code prior to pushing to central; go ahead and push, using tags to identify what's been reviewed or not, and use Hudson (already working) to tag the latest safe build so team members can know which one to pull from.
If your team uses Mercurial and does code reviews, how do you do get the reviewer to see your changes?
Most of these are possible, some are more tedious than others.
You can use bundle by specifying the tip of the central repo as the --base:
hg bundle --base 4a3b2c1d review.bundle
Might as well just use bundle. That way, the changeset data is also included.
You can push (and pull) to (from) any repository that has a common ancestor(s). If you want to pull from one of your colleagues, they just need to run hg serve on their machine, and you will be able to pull.
This also works, but you will have to maintain multiple heads and be careful about merging. If you don't, it can become easy to base a stable change on top of an unreviewed changeset, which will make it hard to undo if you need to fix that unreviewed changeset later.
Of the options you presented, #1 and #3 are probably easiest, just depending on whether or not you can reach each other's boxes.
On a related note: This is the question that got my colleague and I started on developing Kiln, our (Fog Creek's) Mercurial hosting and code review tool. Our plan, and the initial prototype, would keep multiple repositories around, one "central" repository, and a bunch of "review" repositories. The review process would be started by cloning the central repo into a review repo on the server, and then running a full repo diff between the two, with a simple web interface for getting and viewing the diffs.
We've evolved that workflow quite a bit, but the general idea, having a branch repo to push unreviewed changes to and an interface to review them before you push them into the central repo, is still the same. I don't want to advertise here, but I do recommend giving it a try.
Half answer to this question is using ReviewBoard with Mercurial extention. It allows to push certain revisions for review by issuing the following command
hg postreview tip
I'll add a 5th option - do all development work on named branches, preferably one per task. Allow anything to be committed to a "development" named branch, whether it's in working state or not.
Push to the central repository, have reviewer pull the branch. Perform the review on the branch.
When the review has passed, merge the development work into the appropriate feature branch.
This workflow, which is (to me) surprisingly unpopular, has many advantages:
All work gets committed - you do not have to wait for a review to be done before committing.
You will not build off the wrong version. You only ever build from the feature branch.
In-progress work does not interfere with other developers.
From the development branch, you can either look at the latest changes (e.g. the changesets addressing review comments), compare with the branch point, or compare with the latest feature branch - all of which can give useful information.