Suppose we have the following (Visual Studio) projects (simplified):
base-lib-1
base-lib-2
product-1: depending on base-lib-1
product-2: depending on base-lib-1 and base-lib-2
product-3: depending on base-lib-1 and base-lib-2 and used as a component in product-2
product-4: like product-3
I wondered what would be a good way to organize this project structure in one or multiple Mercurial repositories. We currently use Subversion and include the dependent libraries as externals.
Now one way would be to put everything but product-1 in a single repository as all those products are always released together as a single package. I would feel most comfortable with this solution as then I would be pretty sure about how to handle the repository. But how would fit product-1 in this scheme without duplicating base-lib-1?
As an alternative I thought about using sub repositories that would be organized something like this:
product-package-A
base-lib-1
product-1
product-package-B
base-lib-1
base-lib-2
product-2
product-3
product-4
The problem with this approach is that I've never used subrepos so I'm not sure about any pitfalls that would arise with this solution.
For example, do the subrepos behave like SVN externals in that you can decide whether to always use the latest or a fixed revision of each subrepo?
How do subrepos behave if you make changes e.g. in base-lib-1 and product-2 at the same time? Are those handled by Mercurial in the same step or do you have to commit/push and pull/update everything manually? And how would the subrepo of base-lib-1 behave in product-package-A in this case?
How does branching work in this scenario if I want to develop a new feature branch that requires changes in multiple subrepos? Do I have to branch and merge every repository manually or is this handled by Mercurial?
Are there any other pitfalls about using subrepos to organize a large project? What is the preferred way of handling larger projects with many dependencies in Mercurial?
Your question is a near-perfect duplicate of a question that's often asked here, but I can't find a good reference in a quick search, so here you go: use the subrepos
Taking your sub-questions one at a time:
For example, do the subrepos behave like SVN externals in that you can
decide whether to always use the latest or a fixed revision of each
subrepo?
Sub-repos are pegged to a specific version, and the parent project specifies which.
How do subrepos behave if you make changes e.g. in base-lib-1 and
product-2 at the same time? Are those handled by Mercurial in the same
step or do you have to commit/push and pull/update everything
manually? And how would the subrepo of base-lib-1 behave in
product-package-A in this case?
The hg commit comma?
nd takes a --subrepos option, so you can recursively commit or not at your option (see all ui.commitsubrepos in man hgrc). Push always pushes subrepo changes.
How does branching work in this scenario if I want to develop a new
feature branch that requires changes in multiple subrepos? Do I have
to branch and merge every repository manually or is this handled by
Mercurial?
Yes, you'll be branching and merging in each repo manually.
Are there any other pitfalls about using subrepos to organize a large
project? What is the preferred way of handling larger projects with
many dependencies in Mercurial?
Just do a test scenario first and practice. Also, make sure people have new-ish clients. This stuff didn't really get polished UI-wise until 2.1.x
Related
I really like the Hg Flow for Mercurial repositories. we are currently using Bitbucket, and in each product multiple developers are working. basically they can work as below:
a team might work on a single feature.
another team might work on a release/hot fix.
So do i keep the "develop" branch in BitBucket or local repositories. and how about feature branches, should i push them to the central repository and remove when required. i assume we should do so right?
Thanks
I personally neither use git flow or hg flow as tools, but I do use some of the methods for my own projects (manually).
Before going into detail, you always need to provide branches in the main/bitbucket repository when multiple people need to merge or branch from them.
This definately includes "develop" and probably also features/fixes multiple people need to work on (unless you have another repository or method to exchange branches/commits between them)
The difference between using git and mercurial/hg is relevant here, since the branching models are quite different.
See A Guide to Branching in Mercurial for details. Using hg bookmarks would be quite similar to what git does with branches, but there is no full support for the bookmark branching model on BitBucket (see this ticket).
hg flow (the tool) uses named branches. In contrast to git branches, these are not at all light-weight, but permanent and global (they can at least be closed now).
This means whenever any commit created on any (named) branch other than "default" is pushed to bitbucket (even after merging) this will create the branch in the bitbucket repository.
So you don't have any other choice than keeping all branches in the main repository.
However, You can decide when to push and when to close these.
I would advise using hg push -r to push only the branches/heads you want to push and only pushing these when they are either needed by somebody else or finished and merged.
Branches should be closed as soon they are not needed anymore. (This is probably done by hg flow automatically)
You should close branches locally whenever possible. This way they might not even appear in the bitbucket interface. Some might reach the bitbucket repository only in closed state (which hides them from the interface).
Obviously you should often push any branches multiple people need to merge from.
In my understanding of the workflow the "develop" branch is always exactly one branch per project that should be pushed frequently (after local testing).
In case you are either not using hg-flow or named branches things are a bit different.
Both, using forks/clones or bookmarks as a branching method doesn't generate permanent or necessarily global branches.
Like mentioned above, you can't use bookmarks (reliably) when you also want to use bitbucket pull requests. You have to push bookmarks separately. A normal push will only update (a head of) the branch so you might miss commits from other team members when marging later. Hg will tell you when a new head is created. In that case you might want to merge the branch with the remote bookmark into your branch before pushing.
When using forks as branches it works a bit like with bookmarks, but bitbucket has full support for that. You need to have a new fork on bitbucket for every branch.
You naturally only want to create extra forks if you need different people to work on it and you don't have other means of commit exchange for them. You will need at least a separate "develop" repository then.
I personally wouldn't use the full "flow" with hg on bitbucket.
For my projects the "develop" branch is the same as master/default, since I don't roll out releases with git (other than development builds, that wouldn't use the release branch anyways). I don't need a separate "production" branch, since tags can mostly be used for production usage.
I also don't create a separate "release-preparation" branch. There is only a point in time when I only apply bugfixes on develop and stop merging features. That obviously won't work when you need to work at the same time on features that are dependendant on features not to be released in the next release.
Always using the full "git flow" is easy because git branching is easy and light-weight.
Depending on the branching model you use and how supportive the other tools are,
using the full "hg flow" might not be "worth it".
The hg guide actually discourages use of named branches for short-lived branches.
See Feature separation through named branches.
The "easy" branching concept promoted in the guide is forking/cloning. Bookmarks would be the natural way to translate git flow if the tool/bitbucket support would be better (and bookmarks longer a core hg feature).
Disclaimer:
I prefer git when I can choose. I do use hg, but not as my personal choice.
You also might have considered most of this, but since you didn't state any of these details and accept an answer (in the comments) that is quite different to what you are asking, I wanted to elaborate a bit.
Edit:
To follow-up on the comments:
I think hg bookmarks are comparable to git branches because both are just movable pointers to commits.
The main difference is, that when you delete a branch in git, the commits are possibly lost (when not part of other branches or pointed to in a another branch before they are garbage collected). When you delete a bookmark in hg, then the commits are still part of the repository (part of the (named or default) branch) unless manually stripped.
Anonymous heads are related, but only as something the bookmarks point to. Without bookmarks pointing to them the anonymous heads are not usable as a branch to work with (for more than just a local merge) and share. When you have anonymous heads in a repository you don't know what they are supposed to be or where they came from, unless you remember or have other clues. In my eyes anonymous heads are only a workaround for late implementation of bookmarks and no good implementation of remotes/remote heads.
Named branches are rather unrelated, as the only thing they have in common with git branches is having a name. They are light-weight in comparision to cloning the whole repository (forking as branch model), but not in terms of "you can't get rid of them". They are permanent.
Most places tell you not to use named branches unless you have a very good reason or it is a long-running branch.
What is a good workflow for using mercurial with two long-running branches that are slightly divergent (i.e. I never intend to entirely merge them back
together)?
In my case, this is CMS software that has been customized differently for two
different web sites. I started with projectA, and once that was working cloned it to projectB and make further tweaks to both A and B to customize them. Now I want to develop some features that show up in both A and B, without merging the site-specific customizations. How?
hg push will push everything, so that won't work
Transplant appears to give me different changeset hashes, which worries me
I feel like maybe the repositories should be set up differently, but I'm not
sure how.
As Thilo comments, the common part would be best developed (and published in A and B) as a third repo declared as a SubRepo.
That way, you respect the first two repos which are independent (one evolution on A doesn't always mean an evolution on B), and you can develop the common part in subrepo C.
A solution for Mercurial might be if you can put the different areas in files that can be in .hgignore, but then they won't be versioned, so that may not be so good.
Another way is to just use 1 repo, and set a global flag, and use template A or B depending on the flag, and / or include different code source file depending on the flag. If the difference is small, then can use if-then-else inside the same file.
You can use hg push to push the changes back together, but you don't necessarily have to merge all the changesets into the trunk. Just take the ones you want.
As stated above, a subrepo is probably the best option. Another alternative would be to have a third branch with the common work, and merge from that branch to projectA and projectB (but never back to the common branch).
This alternative is more likely to have accidents (merging the wrong way) but you might find that it is easier to set up and get working quickly.
Subversion shop considering switching to Mercurial, trying to figure out in advance what all the complaints from developers are going to be. There's one fairly common use case here that I can't see how to handle.
I'm working on some largish feature, and I have a significant part of the code -- or possibly several significant parts of the code -- in pieces all over the garage floor, totally unsuitable for checkin, maybe not even compiling.
An urgent bugfix request comes in. The fix is nice and local and doesn't touch any of the code I've been working on.
I make the fix in my working copy.
Now what?
I've looked at "Mercurial cherry picking changes for commit" and "best practices in mercurial: branch vs. clone, and partial merges?" and all the suggestions seem to be extensions of varying complexity, from Record and Shelve to Queues.
The fact that there apparently isn't any core functionality for this makes me suspect that in some sense this working style is Doing It Wrong. What would a Mercurial-like solution to this use case look like?
Edited to add: git, by contrast, seems designed for this workflow: git add the bugfix files, don't git add anything else (or git reset HEAD anything you might have already added), git commit.
Here's how I would handle the case:
have a dev branch
have feature branches
have a personal branch
have a stable branch.
In your scenario, I would be committing frequently to my branch off the feature branch.
When the request came in, I would hg up -r XYZ where XYZ is the rev number that they are running, then branch a new feature branch off of that(or up branchname, whatever).
Perform work, then merge into the stable branch after the work is tested.
Switch back to my work and merge up from the top feature branch commit node, thus integrating the two streams of effort.
Lots of useful functionality for Mercurial is provided in the form of extensions -- don't be afraid to use them.
As for your question, record provides what you call partial commits (it allows you to select which hunks of changes you want to commit). On the other hand, shelve allows to temporarily make your working copy clean, while keeping the changes locally. Once you commit the bug fix, you can unshelve the changes and continue working.
The canonical way to go around this (i.e. using only core) would probably be to make a clone (note that local clones are cheap as hardlinks are created instead of copies).
You would clone the repository (i.e. create a bug-fix branch in SVN terms) and do the fix from there.
Alternatively if it really is a quick fix you can use the -I option on commit to explicitly check-in individual files.
Like any DVCS, branching is your friend. Branching a repository multiple ways is the bread and butter of these system. Here's a git model you might consider adopting that works quite well with Mercurial, also.
In addition to what Santa said about branching being your friend...
Small-granularity commits are your friend. Rather than making lots of code changes in a single commit, make each logically self-contained code change in its own commit. Then it will be a lot easier to cherry-pick changes to merge between branches.
Don't use Mercurial without using the Mq Extension (it comes pre-packaged in the default installation). In addition to solving your specific problem, it solves a lot of other general problems and really should be the default way that you work (especially if you're using an IDE that doesn't integrate directly with Hg, making switching branches on the fly a difficult way to work).
I'm a Subversion user, and I think I've got my head mostly around it all now. So of course now we're thinking of switching to Mercurial, and I need to start again.
In our single repository, we have the typical branches, tags, trunk layout. When I want to create a feature branch I:
Use the repo browser to copy trunk to branches/Features/[FeatureName].
Checkout a new working copy from branches/Features/[FeatureName].
Start working on it.
Occasionally commit, merge trunk in, resolve conflicts and commit.
When complete, one more merge of trunk, then "Reintegrate" the feature branch into trunk.
(Please note this process is simplified as it doesn't take into account release candidate branches etc).
So I have questions about how I'd fulfil the same requirements (i.e. feature branches rather than working on trunk) in Mercurial:
In Mercurial, is a branch still within the repository, or is it a whole new local repository?
If we each have a copy of the whole repository, does that mean we all have copies of each other's various feature branches (that's a lot of data transfer)?
I know Mercurial is a DCVS, but does that mean we push/pull changes from each other directly, rather than via a peer repository on a server?
I recommend reading this guide
http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial//
In Mercurial, is a branch still within
the repository, or is it a whole new
local repository?
The equivalent of the subversion way of working would be a repository with multiple heads in mercurial. However, this is not the idiomatic way of doing things. Typically you will have only one head in a given repository, so separate repositories for each branch.
If we each have a copy of the whole
repository, does that mean we all have
copies of each other's various feature
branches (that's a lot of data
transfer)?
Yes, if you look at the history of the head of your local repository, then you'll be able to see all the feature branches that were merged in. But mercurial repositories are remarkably space efficient. For example, I have done a hg clone https://www.mercurial-scm.org/repo/hg to get the source for mercurial itself, and it is only 34.3 MB on an NTFS file system (compared to the source code download, which is 1.8 MB). Mercurial will also make use of hardlinks if your file system supports it, so there is little overhead if you clone a repository to another location on the same disk.
I know Mercurial is a DCVS, but does
that mean we push/pull changes from
each other directly, rather than via a
peer repository on a server?
One way of working is indeed to have each developer expose a public repository in which he pushes his own changes. All other developers can then pull what they want.
However, typically you'll have one or more "blessed" repositories where all the changes are integrated. All developers then only need to pull from the blessed repository. Even if you didn't explicitly have such a blessed repository I imagine people would automatically organize themselves like that, e.g. by all pulling from a lead developer.
Steve Losh's article on branching in mercurial linked above is fantastic. I also got into some explaining of branching and how the DAG works in a presentation I gave a couple of months ago on mercurial that's out on slideshare. The pertinent slides start at slide #43.
I think that understanding that all commits to the same repository are stored in a DAG (Directed Acyclic Graph) with some simple rules really helps demystify what's going on.
a node with no child nodes is a "head"
the root node has no parents
regular nodes have a single parent
nodes that are the result of a merge have two parents
if a merge node's parents are from different branches, the child node's branch is inherited from the first parent
Named branches are really just metadata labels on commits, but really aren't any different than the anonymous branches that happen when you merge someone elses work into your repository, or if you go back to an earlier version and then make a commit there to make a new head (which you can later merge).
...so I've gotten used to the simple stuff with Mercurial (add, commit, diff) and found out about the .hgignore file (yay!) and have gotten the hang of creating and switching between branches (branch, update -C).
I have two major questions though:
If I'm in branch "Branch1" and I want to pull in some but not all of the changes from branch "Branch2", how would I do that? Particularly if all the changes are in one subdirectory. (I guess I could just clone the whole repository, then use a directory-merge tool like Beyond Compare to pick&choose my edits. Seems like there ought to be a way to just isolate the changes in one file or one directory, though.)
Switching between branches with update -C seems so easy, I'm wondering why I would bother using clone. I can only think of a few reasons (see below) -- are there some other reasons I'm missing?
a. if I need to act on two versions/branches at once (e.g. do a performance-metric diff)
b. for a backup (clone the repository to a network drive in a physically different location)
c. to do the pick&choose merge like I've mentioned above.
I use clone for:
Short-lived local branches
Cloning to different development machines and servers
The former use is pretty rare for me - mainly when I'm trying an idea I might want to totally abandon. If I want to merge, I'll want to merge ALL the changes. This sort of branching is mainly for tracking different developers' branches so they don't disturb each other. Just to clarify this last point:
I keep working on my changes and pull my fellow devs changes and they pull mine.
When it's convenient for me I'll merge ALL of the changes from one (or all) of these branches into mine.
For feature branches, or longer lived branches, I use named branches which are more comfortably shared between repositories without merging. It also "feels" better when you want to selectively merge.
Basically I look at it this way:
Named branches are for developing different branches or versions of the app
Clones are for managing different contributions to the same version of the app.
That's my take, though really it's a matter of policy.
For question 1, you need to be a little clearer about what you mean by "changes". Which of these do you mean:
"I want to pull some, but not all, of the changesets in a different branch into this one."
"I want to pull the latest version of some, but not all, of the files in a different branch into this one."
If you mean item 1, you should look into the Transplant extension, specifically the idea of cherrypicking a couple of changesets.
If you mean item 2, you would do the following:
Update to the branch you want to pull the changes into.
Use hg revert -r <branch you want to merge> --include <files to update> to change the contents of those files to the way they are on the other branch.
Use hg commit to commit those changes to the branch as a new changeset.
As for question 2, I never use repository clones for branching myself, so I don't know. I use named branches or anonymous branches (sometimes with bookmarks).
I have another option for you to look into: mercurial queues.
The idea is, to have a stack of patches (no commits, "real" patches) ontop of your current working directory. Then, you can add or remove the applied patches, add one, remove it, add another other one, etc. One single patch or a subset of them ends up to be a new "feature" as you probably want to do with branches. After that, you can apply the patch as usual (since it is a change). Branches are probably more useful if you work with somebody else... ?