What are the differences (primarily from a workflow perspective) between the largefiles and lfs extensions?
The largefiles extension is part of Mercurial (and has been for many years). It is an "extension" in a technical sense, but in practice since it is shipped as part of Mercurial that just means its an optional feature you have to enable via configuration if you want to use it.
LFS, as I understand it, is a proposal or work in progress. The page cited in the question hasn't been updated in > 3 years at this point, so perhaps work is no longer underway. And as stated on that page it is "...primarily intended for Mercurial's developers." which I take to mean that end users won't find anything useful for their day to day work.
Related
In this F8 conference video(starting 8:40) from 2015 they speak about the advantages of using Mercurial and a single repository across facebook.
How does this work in practice? Using Mercurial, can i checkout a subdirectory (live in SVN)? If so, how? Do i need a facebook-mercurial-extension for this
P.S.: I only found answers like this or this from 2010 on SO where i am not sure if the answers still apply with all the efforts FB put into it.
From your question it is not clear if you are looking for a workflow (the monorepo vs multiple repos debate) or for performance and scaling for a huge code base.
For the workflow, I suggest googling for monorepo. It has its pros and cons, you need to understand your situation and current workflow to decide. For the performance and scaling, keep reading.
The idea of remotefilelog is not to checkout a subdirectory (as you mention), the idea is to checkout everything. In order to do that in an efficient way, you need two extensions actively developed by Facebook:
remotefilelog. This gives you something conceptually similar to a shallow clone. This reduces hg clone and hg pull time.
fsmonitor (previously called hgwatchman, it is now part of mercurial core). This dramatically reduces time of local operations such as hg status. Note that fsmonitor is independent from remotefilelog. You can start experimenting with this, since it doesn't require any setup on the server side.
With a recent mercurial (which I strongly suggest) you can shave off the additional startup time of the Python interpreter using CommandServer + CHg.
Some additional notes:
I tested extensively fsmonitor. It works very well, on huge repos the time of hg status is reduced from 10 secs to less than 1 sec (and the majority of this 1 sec is Python startup time, see above for CHg). If your repository is really huge, you might need to fine tune some inotify kernel parameters (or the equivalent on MacOSX). The fsmonitor documentation has all the information you need.
I didn't test remotefilelog, although I read everything I found about it and I am sure it works. Depending on how development is done (everybody has always Internet connectivity or not, the organization has its own master repo or not) there can be a caveat: it partially transforms the decentralized hg into a centralized VCS like svn: some operations that normally can be done offline (for example: hg log and the first hg update to a changeset in the past) will now require connectivity to the master repository.
Before considering remotefilelog, I used extensively the largefiles extension on a huge repo. It has the same drawbacks than remotefilelog and some confusing corner cases for users that want to use hg just to get things done without taking the time to understand how it works. If I were to manage another huge repo, I would use remotefilelog instead than largefiles, although their use case is not really the same.
Mercurial has also support for subrepositories (doc1, doc2). The problem is that it changes the behavior of hg depending on where you are in the source tree. Again, if the developers don't care about really understanding how hg works, it will be just too confusing.
Additional information:
Facebook Engineering blog post
scaling mercurial wiki, although not completely up to date
just by googling mercurial facebook.
i am not sure if the answers still apply with all the efforts FB put
into it
(Early 2017) The answers in the questions linked still apply (because they occasionally get updated) but note that you will have to read all the comments and answers.
remotefilelog essentially allows on demand shallow clones (so you don't fetch the history for everything for all time) but you still fetch the essential metadata for, and checkout across, all the directories of the repo at the desired revision.
Using Mercurial, can i checkout a subdirectory (li[k]e in SVN)? If so, how?
https://stackoverflow.com/a/40355673/7836056 discusses how you might use third party extensions to allow narrow/sparse checkouts (Facebook's sparse.py) or narrow clones (Google's NarrowHG) with Mercurial thus only "creating" a single directory from within the main repository (albeit with radically different tradeoffs).
(Note phrasing matters: "sparse checkout" means a very specific action when referring to distributed version control in a manner that doesn't exist when using it to refer to centralised version control)
Talk of Shallow Clone Extension
There had been some talk about work on an unofficial Shallow Clone extension for Mercurial. This extension would function similarly to the git clone --depth X extension, but would provide better push support and merge safety.
Basically, it would let users clone a smaller sub-set of history to save time and space, but still benefit from all the other benefits of hg. After all, not all developers need the whole history back to changeset 0 to be productive.
Links to Discussions
The Shallow Clone extension is discussed on hg's site. They also put up a status page and a plan for implementation. Furthermore, they repeatedly mention that they got a Google Summer of Code (gsoc) intern to work on the extension in the summer of 2010.
Even still, I can't seem to find any up to date status on this feature. I found two bitbucket projects that may be related to it, but neither has seen any recent commits: hg-shallow-clone and hg-shallow.
Is Shallow Clone Still Under Development?
Does anyone know that status of this extension? Is it still being worked on somewhere by someone or is it truly abandoned? What ever came of that GSoC work?
It is not under development. Various people from outside the Mercurial project have made attempts, but none have come to fruition, and the Mercurial project itself considers things like the largefiles extension a better solution for keeping repository clone sizes down.
I've used Mercurial for years locally, but now we are doing a pilot of switching over from Subversion at my company.
We're embracing the fact that developers will now be making more granular changesets--some of which may not even build. When developers push their changes to the central repository, all of these changesets will show up in the history - this is natural and expected.
My question is: how do we deal with the fact that, because changesets are more granular, this makes it possible for developers to update to a revision that doesn't build? We are coming from a world where you can checkout anywhere in the repository and reasonably expect to be able to make a release from that point. With DVCS's, how do you tell where a "safe" revision is?
This issue is somewhat addressed in this question, but I'm more interested in finding out how to deal with this using branch repos (and not named branches).
I understand there are ways to modify history (e.g. the collapse extension) so that the changes get collapsed in the history of the repository, but we'd like to preserve the history.
Looking at the mercurial and mozilla trees, I don't see a clear way to tell where safe revisions are to sync. Is this not as important as we think it is?
Do you really need to check out ANY old revision and expect it to build?
I agree with you that you should be able to expect that the tip will always build.
This can be easily achieved, like Lazy Badger already said, by some kind of "don't push unfinished work to the main repo" policy / mutual agreement.
Concerning older revisions:
if you want to build older official releases of your software again (like, you're at version 1.6 now and want to make an v1.2 executable), you can expect the 1.2 tag to build as well if everybody always sticked to the "don't push unfinished work" policy.
if you want to take ANY revision and build it...well, do you really need this? Any bugs in previous versions will probably refer to official releases (see the "1.2 tag" stuff above), and not to something in between.
if you really need a buildable version from in between the official releases (for whatever reason), you'll have to look at the commit messages and find the commit that says feature 'foo' finished (and not the one that says started implementation of feature 'foo').
Yes, this requires a bit of thinking / common sense, but I can't imagine that you will need this really often.
Political solution may be "don't push crap into main repo", isn't it?
Technical solution will be not tag, but one bookmark ("KnownAsGood"), applied to the last working HEAD of default branch and agreement between devs "Update not to tip, but to bookmark"
Who'll test commits and move bookmark is another question from "project management" tag
If you're using feature branches and merging them without fast forward merges, you still have only stable builds on the mainline, but can see the unstable stuff on the feature branches if desired.
You can always tag your commits. These could be major/minor releases, successful builds or however you like. You can see the mercurial and mozilla tags in their repos.
I'm using Mercurial with TortoiseHg. Each developer has their own repositories, and there's one central repository on the server for synchronizing our changes. (This will sound lame, but we're using it to manage the source for a legacy VB6 project. Nothing we can do about that...)
As has been pointed out elsewhere, there is a big problem in VB6 with merging the .frx (form resources) files. So code changes seem to merge fine, but if two developers both make changes at the same time in the form design view, we can't merge.
I'm ok with disallowing concurrent edits, but of course the whole point of Mercurial is that it's distributed so there is no option to force a file to be locked before editing. I don't believe there's a Mercurial solution for this, so I'm wondering: other developers who are using Mercurial for version control, do you have some 3rd party tool that assists with locking files for editing in the cases where it's necessary? Did we make a mistake using Mercurial instead of something like SVN?
Heard of some people using a standalone lock-server (this one in particular).
This is from Bryan O'Sullivan's book on Mercurial:
There is no single revision control
tool that is best in all situations.
As an example, Subversion is a good
choice for working with frequently
edited binary files, due to its
centralised nature and support for
file locking.
I'd like to evaluate Mercurial for my working projects. But most of my projects very heavily rely on the presence of svn:externals-like support. I've searched over StackOverflow and googled for corresponding support in Mercurial. All I found is subrepo feature added in Mercurial 1.3, but the page for this feature said:
subrepos are an experimental feature for Mercurial 1.3. So don't do this on mission critical repositories!
I don't want to use something unstable.
Can anybody shed some light on the real status of this feature, and the plans of polishing/finishing it and when it will be called "stable" and ready for mission critical repositories?
The word in the #mercurial IRC channel is that subrepos will continue to work as they do, and support will grow. For example currently the 'hg status' command isn't subrepo aware -- it works, it just doesn't recurse, but that in the future it will be. However, the current behaviors, fileformats (.hgsub and .hgsubstate) will only be changed in backward compatible ways.
So, go ahead and count on it now, and look forward to it getting better.
P.S. As of mercurial 1.4.2 the subrepos can now be subversion repos, so you can use a mercurial parent and a svn kid.
I've had good luck with the feature in my (light) usage of it so far. It's come in handy in two places:
Backing up a tree of unrelated repositories with a single hg pull command.
Tying a project together with specific versions of its dependencies, so that a single hg clone gets buildable source code. This is closer to the typical svn:externals usage.
Here are a couple of the limitations I've seen with it so far:
In case #1 above, you have to commit all subrepos at once. This is only occasionally annoying, as Mercurial (like any DVCS) encourages frequent commits—so most repos aren't left sitting around in an incomplete state to begin with.
Only the most basic Mercurial commands are subrepo-aware: clone, push / pull, update / commit, and perhaps a couple of others.
Extension authors are going to need time to test their extensions against repositories with subrepos.
When the Mercurial team describes the feature as "experimental," they don't mean that it's suddenly going to decide to erase all your data. They just mean that they haven't coded around all the edge cases like name conflicts (e.g., one developer adds a subrepo called README, while another developer adds a text file called README).