We have 500 plus Hg repositories and am looking for a quick and efficient want of backup. Is there a script or Tool that we can use to backup these repositories. We tried Hg bundle, hg clone and regular file system backup but they are not helping.
Is there a standard practice, or some documentation for Hg repositories backup policy?
A follow-up question, what will happen when a user is in a middle of pushing the changeset and we start the backup ?
We do use RhodeCode to publish the Hg repositories.
Thanks
There's no standard. A true FS-level snapshot taken during a push will be fine, but a non-instantaneous mirroring operation (recursive copy) could end up with a corrupt repo, though it would be repairable to the pre-push state.
In the past I've done something as simple as:
for repo in $(find /srv/repos -type d -name .hg | sed 's/\.hg$//') ; do
hg --cwd $repo --repository $repo push ssh://backupserver/$(basename $repo)
done
That pushes all repos to a remote ssh server, incrementally, with full updating-while-pushing integrity, creating them if necessary.
I have a somewhat similar solution of cloning/pulling from a backup server but I trigger the backup on an hg hook.
In the main server global hgrc, I have:
changegroup.backup = .../backup.sh
and in backup.sh something like:
REPO=`hg root`
ts sshpass -p '...' ssh hg#backupserver "~/bin/pull.exp $REPO" 2>> $LOG
Ts (task spooler) allows the operation to happen asynchronously. The expect script is handling the fact that the repository might be new (thus performing an hg clone --noupdate) or already existing (thus performing an hg pull) and also can give the ssh key a passphrase when requested.
Pushing on the second server is only allowed for the backup hook, so no issue of multiple heads or needed force can happen.
What I find interesting in this type of real time backup is that in case of crash of the main server, it should be much faster to switch to the backup one.
Related
It occurs quite often that I want to know the full status of my local copy of a project, compared to the remote repository. By full status, I mean the following:
Are there some uncommitted changes locally?
Are there some unpushed commits locally?
Are there some unpulled commits remotely?
Am I on head of default branch?
I know that I can use some graphical tool such as HgView or TortoiseHg, or even my IDE to deal with Mercurial repositories, but I find it more convenient to use CLI when working with several projects/repos at the same time.
The way I am doing currently is by using an alias
alias hg_full='hg incoming; hg outgoing; hg status'
If everything is fine (i.e. local synchronized with remote), I then ensure being on head of default by
hg update default
This approach is perfectly working, but when I work with a slow remote repository, it is quite annoying to wait for both the incoming and outgoing command to return before performing the update.
Is there some way (by the mean of an extension or a more advanced command) to get a full status summary of the local copy compare to remote repository without performing hg in and hg out sequentially?
I think hg summary --remote might be exactly what you're looking for:
$ hg summary --remote
parent: 1:c15d3f90697a tip
commit message here
branch: default
commit: 1 modified
update: (current)
remote: 1 or more incoming, 1 outgoing
You can save yourself some network traffic by doing hg incoming --bundle <filename>, which fetches the incoming changesets and stores them in a bundle file. You can then run hg outgoing (or hg pull) against the bundle file, which doesn't use the network at all.
hg incoming --bundle incoming.bundle # Creates the bundle
hg outgoing incoming.bundle
hg pull incoming.bundle
hg update default
When I run hg log -r remote/project I get the last commit on that bookmark.
How can I get a full list of commits from the head of that bookmark?
This is not (easily) possible in general. You can approximate it with hg incoming from an empty repository, but hg incoming actually does a complete pull of the difference and throws the contents away; it does not scale for large repositories. Any solutions that are both practical and general involve ssh-ing into the remote machine or setting up a separate server process on the remote machine.
An intermediate approach uses hg incoming --bundle FILE -T '' (the -T '' part is to suppress normal output). This will store the difference between your local version in an overlay repository called FILE; you can then use hg log -R FILE to perform normal log commands on the overlay repository (and you can also pull from it, as though it were a snapshot of the original remote). This still relies on you having a significant portion of the repository on your local machine, or it will result in a full download of the remote repository.
I have read only permission to an hg repo and am trying to develop and test changes to it locally. The problem is that I am in the middle of changing dev machines and am caught in a weird/akward state across the two machines.
On my old machine I made lots of changes to the repo, locslly. I just cloned the repo on my new machine, but obviously that doesn't contain the changes from my old machine. I need a way to createe a patch/diff from my local working copy on my old machine, and then apply them to my local working copy on my new machine. The problem is that I already commited (hg commit -m "Blah") the changes on my old machine to the distributed repo on it.
What set of specific commands can I use to create a patch/diff of my old machine and then apply it to the repo on my new one?
Update
I commited all changes on my old machine and then ran hg serve, exposing http://mymachine.example.com:8000.
On my new machine, where I had made some different changes (locally) than the changes from my old machine, I ran hg pull http://mymachine.example.com:8000 and got:
myuser#mymachine:~/sandbox/eclipse/workspace/myapp$ hg pull http://mymachine.example.com:8000
pulling from http://mymachine.example.com:8000/
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 16 changes to 10 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
So I run hg merge:
myuser#mymachine:~/sandbox/eclipse/workspace/myapp$ hg merge
abort: uncommitted changes
(use 'hg status' to list changes)
What do I do now?!?
You can use:
$ hg diff > changes.patch
To create a patch file, then:
$ patch -p1 < changes.patch
To apply that patch file on your new machine.
Well, that's actually fantastic, mercurial is a distributed version control system and you do not need to go via any patch file at all: simply pull the changes from your old machine to your new machine:
hg pull URL
where URL can be any network URL or also ssh-login, e.g.
hg pull ssh://mylogin#old.maschine.box or hg pull path/to/old/repository/on/nfs/mount
`
Alternatively you can also use bundle and unbundle. They create bundles which can be imported in the new mercurial easily and keep all meta-information.
hg bundle -r XXX --base YYY > FILENAME
where YYY is a revision you know you have in your new repository. You import it into your new repo with hg unbundle FILENAME. Of course you can bundle several changesets at once by repeating the -r argument or giving a changeset range like -r X:Y.
The least comfortable method is a via diff or export:
hg export -r XXX > FILENAME or equivalent hg diff -c XXX > FILENAME where you need to import the result with patch -p1 < FILENAME or hg import FILENAME.
The easiest way is to do this is to ensure that all work on your old machine is committed. Then use this command on it from the base of your repo:
hg serve
which creates a simple http server on this repo. The monitor should state the name of the http URL it is serving.
On your new machine, just pull from that URL.
Once you've pulled your old changes you can stop the hg serve process with ^C.
The advantages of this method are that it is very quick, and that it works on just about any system. The ssh method is also quick, but it won't work unless your system is configured to use ssh.
Answer to Update
The OPs update is asking an orthogonal question about how to merge changes pulled from a server with local changes. If you haven't already done so, try to digest the information in this merge doc and this one.
Merging is for merging changesets. The error is happening because you have local changes that haven't been committed which mercurial can't merge. So the first thing to do is to commit your local changes, then you will be able to merge.
But before you merge, I strongly recommend that you are merging what you think you are merging. Either ensure there are only 2 heads, or specify which head you are merging with. When merging, you have to be at one of the heads you wish to merge; it's usually better to be at the head with the most changes since the common ancestor because the diffs are simpler.
After you've merged, don't forget to commit the merge. :-)
I have a mercurial repository my_project, hosted at bitbucket. Today I made a number of changes and commited them to my local repository, but didn't push them out yet.
I then majorly stuffed up and fatfingered rm -rf my_project (!!!!!).
Is there some way I can retrieve the changes that I committed today, given that I hadn't pushed them out yet? I know a day's worth of commits doesn't sound like much, but it was!
All the other clones I have of this project are only up-to-date to the most recent push (which didn't include today's changes).
cheers.
mercurial cannot save you. The data from mercurial is stored in a hidden directory in the base of your project folder. In your case, probably at my_project/.hg. Your recursive delete would have trashed this folder as well.
So maybe a file recovery tool?
No. The changes are only stored in the local repository directory (the .hg directory therein) until you've pushed. They're never put anywhere else (not even /tmp).
There is a possibility that you'll be able to recover the deleted files from the disk, though; search around for instructions and tools for doing that.
I'm afraid the commit is deleted together with the working copy and file recovery tools are your only option to recover the missing .hg folder. I see you could recover the code from the install — great!
If you're afraid of this happening again, then you could install a crude hook like
[hooks]
post-commit = R=~/backup-repos/$(basename "$PWD");
(hg init "$R"; hg push -f "$R") > /dev/null 2>&1 || true
That will forcibly push a copy of all your commits to a suitable repo under ~/backup-repos. The -f flag ensures that you will push a backup even if you play with extensions like rebase or mq that modify history. It will also allow pushing changesets from unrelated repos into the same backup repo — imagine two different repos named foo. So the backup repositories will end up with a gigantic pile of changesets after a while and you might want to delete them once in a while.
I tested this briefly and for everyday work I don't think you'll notice the overhead of the extra copy and you might thank yourself later :-)
After a change in my local repo, I commit with hg commit -m <message>, then I push to my server with hg push, then in my server I update working dir with hg update.
Is there a better way to do this?
The first two steps you have described:
hg commit -m <message>
hg push
are required as per the fact that commits are kept completely separate from the server in Mercurial (and most other DVCS as well). You could write a post-commit hook to perform the push after each commit, but this is not advised because it prevents you from correcting simple mistakes during the commit and before the push.
Because you're trying to perform an update on 'the server' I'm assuming you are executing a version of the code in your repository on the server. I'm assuming this because typically the server would simply act as a master repository for you and your developers to access (and also to be subject to backups, etc..), and would not need the explicit hg update.
Assuming you are executing code on the server, you can try and replace the push and the update with this command:
hg pull <path to development repo> -u
which will perform a pull from your local repo and then an automatic update. Depending on your server configuration, it might be difficult to get the path to your local repo.
For the first part of the question (ie. automatically push when you do a commit), you can use the trick described in this answer : mercurial automatic push on every commit .
If you want to automatically update the working directory, you can do this with a hook. Add this in the hgrc of your repository (.hg/hgrc on your server directory) :
[hooks]
changegroup = hg update >&2
This will automatically update the working directory every time a push is made to this server. This hook is described in the Mercurial FAQ.
If you use these 2 solutions, the next time you do hg commit -m "message", the commit will be automatically pushed to the remote server and the working directory on the server will be updated.
There is an extension called autosync you might find useful:
This extension provides the autosync command which automatically and continuously commits working copy changes, fetches (pull, merge, commit) changes from another repository and pushes local changes back to the other repository. Think of configuration files or to-do lists as examples for things to synchronize. On a higher level the autosync command not only synchronizes repositories but working copies. A central repository (usually without a working copy) must be used as synchronization hub: