Talk of Shallow Clone Extension
There had been some talk about work on an unofficial Shallow Clone extension for Mercurial. This extension would function similarly to the git clone --depth X extension, but would provide better push support and merge safety.
Basically, it would let users clone a smaller sub-set of history to save time and space, but still benefit from all the other benefits of hg. After all, not all developers need the whole history back to changeset 0 to be productive.
Links to Discussions
The Shallow Clone extension is discussed on hg's site. They also put up a status page and a plan for implementation. Furthermore, they repeatedly mention that they got a Google Summer of Code (gsoc) intern to work on the extension in the summer of 2010.
Even still, I can't seem to find any up to date status on this feature. I found two bitbucket projects that may be related to it, but neither has seen any recent commits: hg-shallow-clone and hg-shallow.
Is Shallow Clone Still Under Development?
Does anyone know that status of this extension? Is it still being worked on somewhere by someone or is it truly abandoned? What ever came of that GSoC work?
It is not under development. Various people from outside the Mercurial project have made attempts, but none have come to fruition, and the Mercurial project itself considers things like the largefiles extension a better solution for keeping repository clone sizes down.
Related
In this F8 conference video(starting 8:40) from 2015 they speak about the advantages of using Mercurial and a single repository across facebook.
How does this work in practice? Using Mercurial, can i checkout a subdirectory (live in SVN)? If so, how? Do i need a facebook-mercurial-extension for this
P.S.: I only found answers like this or this from 2010 on SO where i am not sure if the answers still apply with all the efforts FB put into it.
From your question it is not clear if you are looking for a workflow (the monorepo vs multiple repos debate) or for performance and scaling for a huge code base.
For the workflow, I suggest googling for monorepo. It has its pros and cons, you need to understand your situation and current workflow to decide. For the performance and scaling, keep reading.
The idea of remotefilelog is not to checkout a subdirectory (as you mention), the idea is to checkout everything. In order to do that in an efficient way, you need two extensions actively developed by Facebook:
remotefilelog. This gives you something conceptually similar to a shallow clone. This reduces hg clone and hg pull time.
fsmonitor (previously called hgwatchman, it is now part of mercurial core). This dramatically reduces time of local operations such as hg status. Note that fsmonitor is independent from remotefilelog. You can start experimenting with this, since it doesn't require any setup on the server side.
With a recent mercurial (which I strongly suggest) you can shave off the additional startup time of the Python interpreter using CommandServer + CHg.
Some additional notes:
I tested extensively fsmonitor. It works very well, on huge repos the time of hg status is reduced from 10 secs to less than 1 sec (and the majority of this 1 sec is Python startup time, see above for CHg). If your repository is really huge, you might need to fine tune some inotify kernel parameters (or the equivalent on MacOSX). The fsmonitor documentation has all the information you need.
I didn't test remotefilelog, although I read everything I found about it and I am sure it works. Depending on how development is done (everybody has always Internet connectivity or not, the organization has its own master repo or not) there can be a caveat: it partially transforms the decentralized hg into a centralized VCS like svn: some operations that normally can be done offline (for example: hg log and the first hg update to a changeset in the past) will now require connectivity to the master repository.
Before considering remotefilelog, I used extensively the largefiles extension on a huge repo. It has the same drawbacks than remotefilelog and some confusing corner cases for users that want to use hg just to get things done without taking the time to understand how it works. If I were to manage another huge repo, I would use remotefilelog instead than largefiles, although their use case is not really the same.
Mercurial has also support for subrepositories (doc1, doc2). The problem is that it changes the behavior of hg depending on where you are in the source tree. Again, if the developers don't care about really understanding how hg works, it will be just too confusing.
Additional information:
Facebook Engineering blog post
scaling mercurial wiki, although not completely up to date
just by googling mercurial facebook.
i am not sure if the answers still apply with all the efforts FB put
into it
(Early 2017) The answers in the questions linked still apply (because they occasionally get updated) but note that you will have to read all the comments and answers.
remotefilelog essentially allows on demand shallow clones (so you don't fetch the history for everything for all time) but you still fetch the essential metadata for, and checkout across, all the directories of the repo at the desired revision.
Using Mercurial, can i checkout a subdirectory (li[k]e in SVN)? If so, how?
https://stackoverflow.com/a/40355673/7836056 discusses how you might use third party extensions to allow narrow/sparse checkouts (Facebook's sparse.py) or narrow clones (Google's NarrowHG) with Mercurial thus only "creating" a single directory from within the main repository (albeit with radically different tradeoffs).
(Note phrasing matters: "sparse checkout" means a very specific action when referring to distributed version control in a manner that doesn't exist when using it to refer to centralised version control)
We are looking at moving away from Perforce (mostly due to the pain of having to always be connected to its sever and the lack of support for moving bug fixes between branches, and also wish to introduce a tool for code reviews)
However we still need all the history that is in Perforce, so is there a way to move to Kiln while keeping the history.
(Someone else in the company has tried importing from Perforce into Mercurial but could not find a good way of keeping the history. As I understand it Kiln is a hosted service that provides Mercurial plus a code review tool.)
First look here:
Mercurial Wiki entry for converting from Perforce
Since the conversion of VC systems with different paradigms is non trivial, you might want to tweak one of the methods considered there.
If this fails totally, you can also try to convert to a intermediate VC system like, SVN, CVS, git and then to mercurial.
Once you got it in mercurial locally you can always move it up to Kiln or one of the other hosted systems.
Does anybody have any advice about using Mercurial as a front end for Perforce? What I would like to do is to use Mercurial to handle really granular changes and then, once I'm done something, push it back up to the Perforce server.
I found this article http://www.dehora.net/journal/2008/01/05/using-mercurial-with-perforce/ but it doesn't suggest any tooling to help out with the integrations. Does any exist? I suppose I am looking for it pull any new changes from Perforce, integrate them into my local Mercurial then roll up all the Mercurial commits I've made since last integration and push them up to Perforce. Similar to git-p4.
I got an error when I followed the link you gave. But I suggest you look into the perfarce extension (I love the name!). I have not used it myself, but it's my understanding that this is what people use to bridge the gap between Mercurial and Perforce.
See also the wiki page on Perforce concepts. It seems to have a lot of good info.
As Martin says, Perfarce is what you want. I've used it at a previous job, and in general it works pretty well if you're just working with a single perforce branch into a single mercurial clone. If you start cloning multiple times from your original Mercurial clone, then things start getting complex. Not impossible, just complex.
In general it works by bundling all changes since you last pulled from Perforce. Creating a single perforce changelist from them. Tagging that changelist's comment with the hash of the Mercurial version and committing it to perforce. It then re-imports that change from Perforce and merges it into your Mercurial tree, and because they're both the same there's no merge.
Basically it works quite well for pull/edit/commit/update workflows. Unfortunately it's not any help when it comes to integrations (unless I missed something) as perforce branches aren't converted to Mercurial ones. It wouldn't know what to merge.
I trying to like mercurial, but is driving me nuts.
I convert from SVN to it thinking in the promise of better workflows. Also, I use git in contract work, so I get the whole distributed idea. I choose mercurial for the promise of more simplicity and better windows support.
I'm a solo developer right now, working in a project that compromise 5 separate repos. Some commits are global and other locally applied to each one.
Normally, I commit each single repo, then fecth/push from the root.
But I found that:
The 2 head thing. Exist a way to disable that forever?
None of the GUI out there truly work for my workflow (I use MacHG, TortoiseHG, and try other half-attempts guis). I need to fall-back to command line for a workable push/pull behavior. No GUI understand subrepo, no understand that 2-head problem, so I need to fix/merge in command line.
But the worst of all, is that subrepo are tooo unreliable. Almost each 2-3 days I get "abort: unknown revision ****". Sometimes the hex value have a "+" at the end. Others not.
Sometimes is the ONLY change in the repo.
I don't wanna fix this behavior, I wanna a working replacement to subrepo, that let me:
Commit each sub repo
At the end, push+pull, automerge anything that make sense... and if work with the available GUIS better.
You are a bit confused in my humble opinion.
Mercurial and GIT are distributed versoin control, so the repository is always...the same.
You have your own copy of the repository with all the history. There is no "sub"-repo.
All repositories are "peer", and you can use clone them to do "sub" developement.
Every developer must have at least one peer repository.
Take a look at this good tutorial http://hginit.com/
Hope this helps
I'd like to evaluate Mercurial for my working projects. But most of my projects very heavily rely on the presence of svn:externals-like support. I've searched over StackOverflow and googled for corresponding support in Mercurial. All I found is subrepo feature added in Mercurial 1.3, but the page for this feature said:
subrepos are an experimental feature for Mercurial 1.3. So don't do this on mission critical repositories!
I don't want to use something unstable.
Can anybody shed some light on the real status of this feature, and the plans of polishing/finishing it and when it will be called "stable" and ready for mission critical repositories?
The word in the #mercurial IRC channel is that subrepos will continue to work as they do, and support will grow. For example currently the 'hg status' command isn't subrepo aware -- it works, it just doesn't recurse, but that in the future it will be. However, the current behaviors, fileformats (.hgsub and .hgsubstate) will only be changed in backward compatible ways.
So, go ahead and count on it now, and look forward to it getting better.
P.S. As of mercurial 1.4.2 the subrepos can now be subversion repos, so you can use a mercurial parent and a svn kid.
I've had good luck with the feature in my (light) usage of it so far. It's come in handy in two places:
Backing up a tree of unrelated repositories with a single hg pull command.
Tying a project together with specific versions of its dependencies, so that a single hg clone gets buildable source code. This is closer to the typical svn:externals usage.
Here are a couple of the limitations I've seen with it so far:
In case #1 above, you have to commit all subrepos at once. This is only occasionally annoying, as Mercurial (like any DVCS) encourages frequent commits—so most repos aren't left sitting around in an incomplete state to begin with.
Only the most basic Mercurial commands are subrepo-aware: clone, push / pull, update / commit, and perhaps a couple of others.
Extension authors are going to need time to test their extensions against repositories with subrepos.
When the Mercurial team describes the feature as "experimental," they don't mean that it's suddenly going to decide to erase all your data. They just mean that they haven't coded around all the edge cases like name conflicts (e.g., one developer adds a subrepo called README, while another developer adds a text file called README).