I'm trying to replicate a workflow using Mercurial. It seems like this should be common, but I'm not quite sure how to do the magic bits.
User A creates changesets A1 and A2. User A does not have changesets B or C.
A1 and A2 are submitted to the server, which puts them in a queue. Changesets from other users B and C are ahead in the queue.
Each changeset (B then C then A1/A2) is rebased into the main branch automatically, built, and then accepted into the trunk (or rejected).
The codebase is very large so builds take a long time. In the meantime A3 and A4 are generated by User A.
User A does a pull and gets B', C', and the new A1'/A2' without getting duplicates. A3 and A4 move to the tip of the trunk as development continues.
Step 5 is the one that has me the most stymied. A "git pull --rebase" seems to recognize that the changesets are the same and so A1/A2 disappear whereas with Hg it is a conflict. I don't expect Hg to be exactly the same workflow, I just need some way for a developer to be able to pull the trunk and not have to do manual fix-up of their tree to get their changesets in order. I also need some explainable workflow as to how to recover if your changeset is rejected. Does anyone have experience with this type of workflow that can recommend a tactic?
Thanks
Edit: Here's a simulator for the workflow. I'm certainly willing to try any other workflow that will solve the problem of being able to continue to build while changesets are progressing through acceptance and coming back smoothly.
rm -rf master
rm -rf build
rm -rf c1
rm -rf c2
rm -rf c3
rm -rf bundles
# Master repository
mkdir master
hg init master
echo x >> master/m1.txt
hg -R master add master/m1.txt
hg -R master commit master/m1.txt -m"m-1"
echo x >> master/m1.txt
hg -R master commit master/m1.txt -m"m-2"
echo x >> master/m1.txt
hg -R master commit master/m1.txt -m"m-3"
# Build repository
hg clone master build
# Setup first client
hg clone master c1
echo x >> c1/client1.txt
hg -R c1 add c1/client1.txt
hg -R c1 commit c1/client1.txt -m"c1-1"
echo x >> c1/client1.txt
hg -R c1 commit c1/client1.txt -m"c1-2"
# Setup second client
hg clone master c2
echo x >> c2/client2.txt
hg -R c2 add c2/client2.txt
hg -R c2 commit c2/client2.txt -m"c2-1"
echo x >> c2/client2.txt
hg -R c2 commit c2/client2.txt -m"c2-2"
# Setup third client
hg clone master c3
echo x >> c3/client3.txt
hg -R c3 add c3/client3.txt
hg -R c3 commit c3/client3.txt -m"c3-1"
echo x >> c3/client3.txt
hg -R c3 commit c3/client3.txt -m"c3-2"
# Create the 3 bundles simulating the queue; all clients have pushed
# Hopefully this is done with a push hook
# All changesets are still draft phase
mkdir bundles
hg -R c2 bundle bundles/c2.bundle
hg -R c3 bundle bundles/c3.bundle
hg -R c1 bundle bundles/c1.bundle
# Process first bundle
hg -R build pull bundles/c2.bundle --rebase
hg -R build update
hg -R build push master
# Client 1 pulls at this point
hg -R c1 pull master -u --rebase
# Process second and third bundle
hg -R build pull bundles/c3.bundle
hg -R build rebase -b 5 -d 4
hg -R build pull bundles/c1.bundle
hg -R build rebase -b 7 -d 6
hg -R build push master
# Client 1 pulls again, getting the changesets that were pushed
hg -R c1 pull master -u --rebase
There's one difference between git and mercurial usually-used setups: git often allows rebasing, pruning and other rewrite / destructive operations on other people or remote repositories - mercurial by default does not and only allows amend operations.
However, there's a mercurial way: mercurial introduced some time ago the concept of phases, notably the unmutable phase public and the mutable phase draft. You can now declare repositories as non-publishing (it's a but unfortunate term in my eyes - it basically means that commits pushed there do NOT become phase public but remain in draft phase, thus mutable.
Thus setup your "central" repository as one of those non-publishing repositories, tell all your contributors to make use of a reasonably modern mercurial and to push changes as phase draft (or make sure so in a server-side hook and reject pushes of new changesets with public phase - but that might be problematic). Mercurials exchange of obsolescence markers makes sure that users get the information of which changesets become deprecated and by which new changesets they are being replaced.
The server-side setup then will need to take care of changing the phase of the accepted changesets from draft to public.
Mind: once a changeset is converted to phase public, that changeset becomes unmutable. And the phase will propagate to all who pull - and thus even reverting the phase to draft, and changing or pruning that changeset WILL inevitably permanently leave the repos of all who pulled with extra changesets which everyone will have to prune manually themselves!
You (and all your contributors) also probably should have a look at the evolve extension which makes handling non-publishing repositories and working with and exchanging mutable changesets and their modifications much more comfortable.
Related
I have two hg branches (dev and stable) that aren't merging like I'd expect.
On stable: I grafted in a one-line commit from dev.
On dev: Changed that one line that was grafted, committed change.
On stable: merged dev into stable (no conflicts).
However after this merge stable still has the grafted version of the line (step 1). Not the latest changes to that same line from dev (step 2). Why is this?
The file looks like:
This
file
to
be
merged
Changesets:
Changes "to" to "might" on dev
Grafts changeset 1 to stable
Changes "might" back to "to" on dev
Merges dev into stable. Result is "might" (not "to" like I'd expect to see from changeset 3).
Sorry about the delay here: As soon as you shrank the reproducer to the five commits, I knew what was going on, but I wanted to write my own reproducer before answering, and the priority of this dropped a lot. 😀 The script I used, mktest.hg, to create the commits, the graft, and the merge, appears at the end of this answer.
The key issue here is the way merge actually works in Mercurial. It uses the same algorithm as Git does: that is, it completely ignores any of the branch information, and completely ignores any timing information. It looks only at three specific commits, as found by examining the commit graph, as shown in your image. Here's a text variant via my own reproducer:
$ cd test-hg-graft/
$ cat file.txt
This
file
might
be
merged
$ hg lga
# 4:b027441200d2:draft stable tip Chris Torek
|\ merge dev into stable (9 minutes ago)
| |
| o 3:01c6cc386a08:draft stable Chris Torek
| | back to "to" on stable (9 minutes ago)
| |
| o 2:ad954507e465:draft stable Chris Torek
| | s/to/might/ (9 minutes ago)
| |
o | 1:f7521e4f0941:draft dev Chris Torek
|/ s/to/might/ (9 minutes ago)
|
o 0:a163d2c4874b:draft stable Chris Torek
initial (9 minutes ago)
The lga alias is one I stole borrowed copied from someone else:
lga = log -G --style ~/.hgstuff/map-cmdline.lg
where map-cmdline.lg is in the link above. It's just log -G (aka glog) with a more-compact format.
What's going on
When we run:
hg merge dev
Mercurial locates three specific commits:
The current commit on stable, -r3 in this case (the SHA ID will vary), is one of the two endpoint commits.
The target commit on dev is the result of resolving dev to a revision. We can do this ourselves with hg id -r dev for instance:
$ hg id -r dev
f7521e4f0941 (dev)
$ hg id -n -r dev
1
Note that we can do the same thing with # to identify our current revision, although hg summary spills everything out more conveniently.
Last (or in some sense first, though we need the other two to get here), Mercurial locates a merge base commit from these two commits. The merge base is the first commit in the graph that is reachable from both of the other inputs to the merge. In our particular case, that's rev zero, since we split the branches apart right after -r0.
Technically, the merge base is the output of a Lowest Common Ancestor algorithm as run on the Directed Acyclic Graph. See Wikipedia for some examples. There can be more than one LCA; Mercurial picks one at (apparent) random for this case. In our case there is only one LCA though.
Having found the merge base, Mercurial now runs the equivalent of two diff operations:
hg diff -r 0 -r 3
to see what we changed, and:
hg diff -r 0 -r 1
to see what they changed, since the merge base snapshot.1 If we do this ourselves, we see what Mercurial sees:
$ hg diff -r 0 -r 3
$ hg diff -r 0 -r 1
diff --git a/file.txt b/file.txt
--- a/file.txt
+++ b/file.txt
## -1,5 +1,5 ##
This
file
-to
+might
be
merged
(I have my hg diff configured with git = true so that I get diffs that I can feed to Git—long ago I was doing a lot of conversion work here.)
As far as Mercurial is concerned, then, we did nothing on our branch. So it combines do nothing with make this change to file.txt and comes up with this one change to file.txt. That one change is applied to the files from the merge base commit. The resulting files—well, file, singular, in this case—are the ones that are ready to go into the final merge commit, even though they're not the ones you wanted.
Because Mercurial has more information than Git—in particular, which branch something happened on—it would be possible for Mercurial to behave differently from Git here. But in fact, both do the same thing with this kind of operation. They both find a merge base snapshot, compare the snapshot to the two input commit snapshots, and apply the resulting combined changeset to the files from the merge base. Mercurial can do a better job of catching file renames (since it knows them, vs Git, which just has to guess) and could do a different job of merging here, but doesn't.
1Some might object that Mercurial stores changesets, not snapshots. This is true—or rather, sort of true: every once in a while, Mercurial stores a new copy of a file, instead of a change for it. But as long as we have all the commits needed, storing changes vs storing snapshots is pretty much irrelevant. Given two adjacent snapshots, we can find a changeset, and given one snapshot and a changeset to move forward or backward, we can compute a new snapshot. That's how we can extract a snapshot in Mercurial (which stores changesets), or show a changeset in Git (which stores snapshots).
Script: mktest.hg
#! /bin/sh
d=test-hg-graft
test "$1" = replay && rm -rf $d
if test -e $d; then
echo "fatal: $d already exists" 1>&2
exit 1
fi
set -e
mkdir $d
cd $d
hg init
hg branch stable
cat << END > file.txt
This
file
to
be
merged
END
hg add file.txt
hg commit -m initial
hg branch dev
ed file.txt << END
3s/to/might/
w
q
END
hg commit -m 's/to/might/'
hg checkout stable
hg graft -r 1 # pick up s/to/might/; graft makes its own commit
ed file.txt << END
3s/might/to/
w
q
END
hg commit -m 'back to "to" on stable'
hg merge dev
hg commit -m "merge dev into stable"
As a followup for Mercurial: enforce "hg pull -u" before "hg commit"
I have started to use a hook
[hooks]
pretxnchangegroup.forbid_2heads = /usr/local/bin/forbid_2head.sh
where forbid_2head.sh looks like this
#!/bin/bash
BRANCH=`hg branch`
COUNT=`hg heads --template '{branch}|{rev}\n' | grep ^${BRANCH} | wc -l`
if [ "$COUNT" -ne "1" ] ; then
echo "=========================================================="
echo "Trying to push more than one head, which is not allowed"
echo "You seem to try to add changes to an old changeset!"
echo "=========================================================="
exit 1
fi
exit 0
It is derivative of the script found at http://tutorials.davidherron.com/2008/10/forbidding-multiple-heads-in-shared.html
where I do allow multiple named branches.
The problem I have now is that
it stops hg push -f which is what I wanted
it also stops hg pull in case there are incoming changeset and I have commits outgoing. This is indeed bad
Can I in any way reuse the same script but change the hook setup and stop "hg push -f"?
Or can I in forbid_2head.sh know whether this is a push or pull command running?
First, the script isn't completely correct: it just counts the number of heads in the branch currently checked out on the server (the one hg branch) reports. You could improve it by using
hg heads tip
to get the heads of the branch of tip. But someone might push changesets on more than one branch at a time, so what you really want is
hg heads --template '{branch}\n' $HG_NODE:tip
to find branch heads for the branches touched by $HG_NODE:tip (the changesets pushed in the not-yet-committed transaction). You can then compare that with
hg log --template '{branch}\n' -r $HG_NODE:tip | sort -u
which are the branches touched by the changegroup.
If you don't want to allow existing multiple heads, then you can simplify the above to just
$(hg heads --template 'x' | wc -c) -eq $(hg branches | wc -l)
which just tests that the number of branch heads is equal to the number of branches — i.e., that there is exactly one branch head per named branch.
With that out of the way, let me mention $HG_SOURCE. That environment variable is set by Mercurial when it runs the hook: it has the value push if the changegroup is being pushed into the repository using direct filesystem access, and the value serve if the changegroup is coming in over SSH or HTTP. See the Mercurial book.
So, to conclude, I believe this is a good "forbid multiple heads" script:
#!/bin/sh
HEADS=$(hg heads --template 'x' | wc -c)
BRANCHES=$(hg branches | wc -l)
test $HG_SOURCE = 'serve' -a $HEADS -ne $BRANCHES
In How do I do a pristine checkout with mercurial? Martin Geisler discuss how to remove already Mercurial commit'ed files using:
hg strip "outgoing()"
But what if I I want to keep my added files which went into "outgoing()" - example:
Two users a and b — starting on the same changeset
User a:
echo "A" > A.txt; hg ci -M -m ""; hg push
User b (forgets to run hg pull -u):
echo "B" > B.txt; hg ci -M -m "" B.txt;
echo "C" > C.txt; hg ci -M -m "" C.txt;
If user b run hg strip "outgoing()" then B.txt and C.txt are lost. hg rollback is not an option since there are two commits.
Can user b revert his files as "locally added - nontracked", then do hg pull -u, which gets A.txt, then handle the add/commit/push for B.txt and C.txt later?
Martin Geisler answered this earlier in the mentioned thread (a comment which I deleted and moved here:
hg update "p1(min(outgoing()))"
hg revert --all --rev tip
hg strip "outgoing()"
hg pull -u
Now user c can finalize his work in the new files B.txt and C.txt and commit+push those.
Other ways to do this?
You could but, by doing so, you are working against one of the biggest features of a DVCS like mercurial, that is, to easily and reliably handle the merging of multiple lines of development as in your case. If user b's goal is to have a line of development with all three changes applied, then the standard way to do that in hg would be to just go ahead and do an hg pull -u which will create a new head containing the change(s) from user a (and any other changes pushed to repo used for pulling) and then use hg merge to merge the two heads, the head containing user b's two change sets and the other containing user a's change set (as pulled). In a simple case like this one with no overlapping changes, hg should do all the right things by default.
$ hg pull -u
[...]
added 1 changesets with 1 changes to 1 files (+1 heads)
not updating: crosses branches (merge branches or update --check to force update)
$ hg merge
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(branch merge, don't forget to commit)
$ hg ci -m "merge"
If there were conflicts between the two heads (i.e. both users committed changes to the same files), there might need to be conflict resolution editing as part of the merge; hg will tell you if that is the case.
Another option is the rebase extension. With your scenario:
A and B start with the same history.
A commits and pushs a change.
B commits two changes, but can't push because of A's commit.
B pulls A's change.
B runs hg rebase and pushes.
Before rebase:
Common ---------------------------- A (tip)
\
B1 - B2 (working parent)
After:
Common - A - B1 - B2 (tip, working parent)
I have searched here, but haven't found any question related to this. I got a problem like this in mercurial:
I manage open source project in bitbucket, so i have clone of the source code in my local. But I also using that project for my own live site, so I made 2 clone of bitbucket repo
Bitbucket Repo
|
==local_clone1
|
==local_clone2-> commit1 => commit2 => commit3
(personalization) (bug fix) (add feature)
The question is, I want to push commit2 and commit3 back to local_clone1, so later on I can push to Bitbucket repo. But don't want to push commit1, since it has my personal data.
Wondering how we do that in mercurial?
This can be done without too much difficulty in this case. See Removing history in the Mercurial guide for more information.
Here's the basics of what you'll need to do:
Go to local_clone2
Get the revision number (hg tip will show you) from the current number. We'll call it 731.
hg export 730-731 > ../local_clone1/changes.diff (or wherever you like)
Go to local_clone1
hg import changes.diff
You may need to edit things manually; refer to that guide for more info in that case.
Here are a couple of options:
backout
Given a history constructed as:
hg init db
cd db
echo >file1
hg ci -Am clone # rev 0
echo >file2
hg ci -Am personalization # rev 1
echo >file3
hg ci -Am bugfix # rev 2
echo >file4
hg ci -Am feature # rev 3 <tip>
Then if the current working directory is the tip, the following commands will "undo" the personalization revision:
hg backout 1
hg ci -m backout
The advantage is history remains immutable, but shows the addition and backout of the personalization changeset.
Mercurial Queues
With the mq extension, history can be edited to remove a changeset:
hg qimport -r 1:3 # convert changesets 1-3 to patches
hg qpop -a # remove all patches (can't delete an applied patch)
hg qdel 1.diff # delete rev 1's patch
hg qpush -a # reapply remaining patches
hg qfin -a # convert all applied patches back to changesets.
The advantage is the personalization changeset disappears. The disadvantage is the changeset hashes change due to the history edit, so this should never be done to changesets that have already been pushed to others. There is also the risk of a mistake editing history.
I have to run a dozen of different build tests on a code base maintained in a mercurial repository. I don't want to run serially these tests on same repository because they modify a set of common files and I want to run them in parallel on different machines. Also, after all tests are run I want to have access to latest test results from those test work areas. Currently I'm cloning the master repository a dozen of times and run in each clone one different test. Before each test execution I do a pull/update/purge preparation sequence in order to start the test on latest clean state. That's good for me.
I'm also preparing new changes using mq extension that I would test on all clones as above before committing them. For testing some ready candidate mq patches I want somehow to deploy/synchronize them to be available in test clones and apply those ready for testing using some guard before running the test.
Did anybody do this synchronization before? What's the most simple way to do it? Do I need to have versioned mq patches for that?
patches can be maintained in their own repository provided you passed the "-c" switch to qinit like so
hg qinit -c
You may still be able to create a patch repo after the fact via
cd .hg/patches
hg init
hg addremove
hg commit -m "my patches"
But I haven't ever tried that personally.
then .hg/patches can be treated like any other mercurial repository. so I think you could probably roll some shell scripting. to get into the .hg dir of your cloned repos and do a
hg clone http://centralrepo.com/patch_repo ./patches
Here is the solution that I've implemented. Few notes:
all patches guarded with ready_for_testing AND those unguarded are applied for testing.
using versioned repos is better because we can make abstraction of queue repository implementation
I use mercurial 1.5.1
master repo is in master dir
clone repos are in clone-x dirs
Here are the steps (some could be optional):
once: put patches in master repository under versioning using mercurial:
a. hg -R master init --mq #no commit happen here, can be done later
b. hg -R master commit --mq --addremove --message 'initial patch queue' #make them visible to clone repos
for each clone, once after clone creation finishes: assuming no patches created yet on clones, initialize mq sub-repository:
a. hg clone master\.hg\patches clone-x\.hg\patches
b. hg -R clone-x qselect ready_for_testing
for each change ready in master (patch created/imported in mq repo): do this before launching tests for it:
a. review/update guards of mq patches: those included for testing should be unguarded or with +ready_for_testing
b. hg -R master commit --mq -A #make them visible to clone repos
for each clone, for each [test] iteration for each clone: do this preparation sequence before running the actual test:
a. hg -R clone-x qpop --all --force
b. hg -R clone-x pull
c. hg -R clone-x update --clean
d. hg -R clone-x purge --all
e. hg -R clone-x pull --mq
f. hg -R clone-x update --mq
g. hg -R clone-x qpush --all
As of Mercurial version 1.5, you can create a patch queue repository in an already existing Mercurial repo using the following command:
hg init --mq
It is OK to have queued patches when you issue that command.