I have been trying to follow the instructions in the answer to this question, using kiln.
i'd like to be able to arrange things as follows:
/somepath/thirdparty maps to a kiln repository "thirdparty" and contains assorted code
/somepath/common maps to a kiln repository "common" and contains shared code i have written
and
/somepath/project1 maps to kiln repository "project1"
/somepath/project1/thirdparty maps to branch of thirdparty above
/somepath/project1/common maps to branch of common above
and
/somepath/project2 maps to kiln repository "project1"
/somepath/project2/thirdparty maps to another branch of thirdparty above
/somepath/project2/common maps to another branch of common above
I found that when I created the .hgsub file as instructed and added/pushed it to Kiln, I could no longer view the Kiln files in the Kiln web file viewer — it displayed an obscure message about the Kiln "overheating" :-) Additionally, whilst it did automatically create the subfolders in the correct place, they were not populated with files, (possibly because the pull failed).
Anybody tried anything like this before, using Kiln?
As I intend to develop a number of apps using the common code (and potentially eventually release the library as open source), I would like to have it managed in discrete repositories. As some of the projects are for end clients however, I need to be able to give them a single repository that includes things as described above.
Kiln does currently not support subrepos that use nested URLs on the server. This means that you cannot have both the following URLs working:
http://server/kiln/somepath/project1
http://server/kiln/somepath/project1/thirdparty
So you should setup Kiln so that you have four repositories on the server:
http://server/kiln/somepath/project1
http://server/kiln/somepath/project2
http://server/kiln/somepath/thirdparty
http://server/kiln/somepath/common
That's easy — just four normal repositories. Then clone "project" and create the .hgsub file with:
thirdparty = http://server/kiln/somepath/thirdparty
common = http://server/kiln/somepath/common
When you push that back to Kiln, it will notice and display links for the subrepositories. However, the subrepositories wont end up being nested on the server. So there wont be any project1/thirdparty path on the server.
It's also far from clear that you would want that. When you have several projects that collaborate and use some common code base, then you want "project1" and "project2" to get each other's changes to this common code base. So it very useful that the common subrepo in both projects push and pull from http://server/kiln/somepath/common.
In Mercurial, we normally recommend that you use paths of the form common = common in the .hgsub file. This means that the server must support nested repositories. When Kiln doesn't support nested repos, you can use full paths instead.
When you initially setup the subrepositories, then remember that you need to update them manually. So with the above URLs, you would setup "project1" by running:
$ hg clone http://server/kiln/somepath/project1
$ echo "common = http://server/kiln/somepath/common" > .hgsub
$ echo "thirdparty = http://server/kiln/somepath/thirdparty" > .hgsub
$ hg commit -m "Created subrepos"
This creates initial empty subrepositories. They are empty because you haven't told Mercurial which changeset you need in them. This is tracked in .hgsubstate where you'll find:
0000000000000000000000000000000000000000 common
0000000000000000000000000000000000000000 thirdparty
To populate the subrepositories you do
$ cd common
$ hg pull --update
$ cd ../thirdparty
$ hg pull --update
$ cd ..
$ hg commit -m "Updated subrepos"
This updates the 000... lines in .hgsubstate with the current tip changeset IDs for the two subrepos. Future clones of "project1" will notice the .hgsubstate file and make sure to update the subrepos to the revision mentioned there.
Related
Summarised Question:
Are github-hosted sub repositories within a mercurial/kiln repository possible, and if so are they automatically updated/cloned when the parent mercurial repository is operated on by a hg clone or hg commit command?
Detailed Question:
Following on from my question that was answered so excellently here , some of my third party code is in folders I downloaded a while ago from opensource efforts on github. Since at that stage I was not using version control, those folders where just standard folders that now been incorporated as sub repositories in mercurial.
This is obviously not ideal, as for one thing, new versions of the libraries may have bug fixes, or new features I wish to use in the future. I also may need to locally customise some of the libraries.
I can see from reading this link that it possible to have mercurial "know" about those git server urls (and revisions), so I can then have mercurial clone the github hosted libraries direct from their parent repos.
Am I right in saying that when I clone the parent (mercurial) repos, those files will be pulled from github, without having to separately manage this using git?
What is also not clear is, if I were to do this, and it transpired that code might need to be customized from within that github-cloned repository, would I need to use git to manage revisions of the local files, or would mercurial do that by proxy? eg id I were to hg commit -S would mercurial invoke git on my behalf to handle that?
Am I right in saying that when I clone the parent (mercurial) repos, those files will be pulled from github, without having to separately manage this using git?
Yes, clone of a Mercurial repository that contain subrepositories will trigger a clone of the subrepos too. It really happens on update. Mercurial notices the .hgsub file and issues the needed hg clone and git clone commands for you. It uses the information in .hgsubstate to know exactly what revision to checkout.
The subrepositories can be hosted anywhere. For a Git subrepository declared like
foo = [git]https://github.com/user/repo.git
Mercurial will simply issue the corresponding clone command:
git clone https://github.com/user/repo.git foo
It's then your reponsibility to later go into the foo repo and use Git to fetch new commits as necessary. After you fetch/pull new commits, you can make a top-level commit to record the new state of the subrepo in the .hgsubstate file. Use hg summary to see if a subrepo is dirty in this sense.
[...] would I need to use git to manage revisions of the local files, or would mercurial do that by proxy? eg id I were to hg commit -S would mercurial invoke git on my behalf to handle that?
When you edit files and make a top-level hg commit, Mercurial will make sure to commit the subrepo first (if you use hg commit -S or if ui.commitsubrepos=True). If you make a top-level push, then Mercurial will always push the subrepos first so that you always have a consistent set of changes on your server.
Is there a way to clone multiple projects at the same time?
I have this project that contains multiple extensions each extension is on a different repo, I would like to be able to clone all extensions with one call. Is there a way to pipe URLs into gh clone command?
How about basic scripting, something like:
for repo in repo1 repo2 repo3
do
hg clone $repo
done
If the extensions all belong to your "main" project anyway, why not make them subrepositories?
Quote from the link:
Subrepositories is a feature that allows you to treat a collection of repositories as a group. This will allow you to clone, commit to, push, and pull projects and their associated libraries as a group.
We are a small team of developers working with a Web Application which is published using a Web Server that is only accessible throught FTP.
Our workflow is the following one:
A developer is working out some requested feature locally
When its done, commits it and Pushes to a 'central' repository
Few times a day, one of developers publishes the files that have been changed to a testing WebSite, to let key users see how does features have been implemented.
Once in a week, we deploy to our production site
As our Webserver doesnt support SSH, we can't push changesets and update on the server, so we created a custom script which Transfers the changed files throught FTP.
Each time we use that script a new tag is created, so we know -using hg diff- the diference between tags (a release for us).
It was all fine until now, that we introduced branches in our workflow, to let a developer work on a radical changes in the code, and keep contributing in daily small changes which are published to production.
The problem is that hg diff doesnt support Branches (or seems that its still in development)
So, which would be the best way to do it ? some options we have been thinking about are:
Mounting FTP as a Volume localy (using MacFuse or similar) and use mercurial push/update But would be so sloooow.
Play around with Bundles and see if they can help us but seems quite complicated
Example
$ hg tag qa-001 /* init to see diferences QA Site */
$ hg tag prod-001 /* init to see diferences Production Site */
$ hg ci -m "working on a stable feature"
$ hg tag qa-002
$ hg ci -m "change on the stable feature"
$ hg tag qa-003
$ hg tag prod-002
$ hg ci -m "another change on stable"
$ hg pull ../CentralRepo /*Where there is another Branch with unstable files*/
With last operation, a new head is created , so now there are two heads (stable, and unstable branch)
$hg diff -r qa-003 -r tip
The Result of hg diff is showing up the Unstable Files without doing the merge
Many Thanks for your comments
In your example, you are creating tags, not (named) branches. Tags won't help you to create separate lines of development: they are just stand-alone identifiers assigned with particular revisions.
Creating branches
To start using branches, you probably want to review some tutorials, such as:
Chapter 8. Managing releases and branchy development (from Bryan O'Sullivan's book)
A Guide to Branching in Mercurial (Steve Losh)
Based on your description, you probably want to create prod and qa branches based on your current default branch, as well as any feature / topic branches you might want for radical changes.
Once you have these branches in place, it's very easy to compare them, merge between them, see what changes are pending from one to the other, and so on as your workflow demands.
Bundles
Play around with Bundles and see if they can help us but seems quite complicated
If you only have FTP access, then bundles probably won't help you. You could upload a bundle to the server via FTP, but you would need to be able to run hg on the server to unpack the bundle into a repository.
Is it possible to clone a part(a single folder or even a single file) of a repository?
Basically it's not possible, there is nothing like Subversion's svn checkout http://example.com/project/dir1.
But you can get a partial clone by rewriting the changeset history with hg convert. On the upside, it will be a partial clone. On the downside, the resulting repository will be not related any more to the initial one. The changeset IDs will be different and it will be very hard to continue interacting with the source repo.
An example of creating a partial clone. Suppose you want to clone only the doc directory from the repo:
$ hg clone http://example.com/project local-project-repo
$ cat > filemap.txt << END
include doc
exclude .
END
$ hg convert --filemap filemap.txt local-project-repo docs-only-repo
Nope. That's called partial cloning (some file paths but not all) or shallow cloning (some revisions but not all), and not provided because the point of a DVCS is that everyone has a full copy of the full repository.
Some online repositories will let you download .tar.gz files of all the files in a specific revision or a specific file from a specific revision, but that's not done using the Mercurial tool.
I have a Mercurial repository containing a handful of related projects. I want to branch just one of these projects to work on it elsewhere.
Is cloning just part of a repository possible, and is that the right way to achieve this?
What you want is a narrow or partial clone, but this is unfortunately not yet supported.
If you already have a big repository and you realize that it would make sense to split it into several smaller repositories, then you can use the convert extension to do a Mercurial to Mercurial conversion. Note that this creates a new repository foo and you cannot push/pull between your-big-repo and foo.
The convert extension is not enabled by default so add the following to your repo's hgrc file or your mercurial.ini file:
[extensions]
hgext.convert=
Then create a map.txt file with
include "libs/foo"
rename "libs/foo" .
(note you can use forward slashes even on Windows) and run
$ hg convert --filemap map.txt your-big-repo foo
That will make foo a repository with the full history of the libs/foo folder from your-big-repo.
If you want to delete all evidence of foo from your-big-repo you can make another conversion where you use exclude libs/foo to get rid of the directory.
When you have several repositories like that and you want to use them as a whole, then you should look at subrepositories. This feature lets you include other repositories in a checkout — similarly to how svn:externals work. Please follow the recommendations on that wiki page.
Instead of doing a partial clone, you can use the Convert Extension to split your repo into more than one repo by sub repository.
Specifically, see the section, Converting from Mercurial:
It's also useful to filter Mercurial repositories to get subsets of an existing one. For example to transform a subdirectory subfoo of a repository foo into a repository with its own life (while keeping its full history), do the following:
$ echo include subfoo > /tmp/myfilemap
$ echo rename subfoo . >> /tmp/myfilemap
$ hg convert --filemap /tmp/myfilemap /path/to/repo/foo /tmp/mysubfoo-repo
I've stumbled accross this issue and found one way to do it: Using symlinks (Linux only unfortunately)
For example, if you only need /project in the repository, on your computer clone the repo in another folder, then use ln -s /repo/location/ project. Mercurial will handle it
(Late 2016) Mainline Mercurial still doesn't package support for "narrow clones" but there are third party extensions that tackle the problem in different ways.
If you can cope with just a narrow checkout (aka "sparse checkout" or "partial checkout by file path") then Facebook's sparse.py extension from the hg-experimental repository (look inside the hgext3rd/ directory) may be workable. In this scenario, you still clone the full history (thus the .hg directory is no smaller) but your working directory only shows/acts on a subset of the full repository.
Alternatively Google have created a NarrowHG extension that does narrow cloning (aka "partial cloning by file path"). You will need to be in control of the server, the client and be willing to use experimental features but it really does restrict the clone's copied history in .hg to a subset of what was in the original repository.
(2019) The sparse extension was merged into Mercurial 4.3 as the experimental sparse extension. The NarrowHG extension was merged into Mercurial 4.6 as the hgext.narrow extension.
It is not possible, hg clone will clone the whole repository.
You can take a look a the sub-repository extension that allows you to have repositories inside a repository, which might match your needs.
This is straight forward with the Convert extension.