Can BFG repo cleaner perform arbitrary operations on files? - git-rewrite-history

I want to squash images using tools like imageoptim-cli on MacOS from git history to shrink repo size. Can BFG support running some more arbitrary commands? My git filter-branch looked something like this: https://gist.github.com/zbeekman/7482ccd0e87f495387951bd612dc390e
git filter-branch --prune-empty --tree-filter 'pwd ; ls -a ; imageoptim -a -q --verbose -d <path/to/img/dir/here> || true' -- --all
But this ran the image tools on those files every commit, even after they had been introduced.

BFG doesn't currently support running arbitrary commands on commits, trees or blobs it encounters. There are a couple of PRs and issues that request this, notably #169 and #165, so you may be able to fashion something from those efforts, or collaborate there.

Related

How to fix unity package error from a git merge? [duplicate]

How do I resolve merge conflicts in my Git repository?
Try:
git mergetool
It opens a GUI that steps you through each conflict, and you get to choose how to merge. Sometimes it requires a bit of hand editing afterwards, but usually it's enough by itself. It is much better than doing the whole thing by hand certainly.
As per Josh Glover's comment:
[This command]
doesn't necessarily open a GUI unless you install one. Running git mergetool for me resulted in vimdiff being used. You can install
one of the following tools to use it instead: meld, opendiff,
kdiff3, tkdiff, xxdiff, tortoisemerge, gvimdiff, diffuse,
ecmerge, p4merge, araxis, vimdiff, emerge.
Below is a sample procedure using vimdiff to resolve merge conflicts, based on this link.
Run the following commands in your terminal
git config merge.tool vimdiff
git config merge.conflictstyle diff3
git config mergetool.prompt false
This will set vimdiff as the default merge tool.
Run the following command in your terminal
git mergetool
You will see a vimdiff display in the following format:
╔═══════╦══════╦════════╗
║ ║ ║ ║
║ LOCAL ║ BASE ║ REMOTE ║
║ ║ ║ ║
╠═══════╩══════╩════════╣
║ ║
║ MERGED ║
║ ║
╚═══════════════════════╝
These 4 views are
LOCAL: this is the file from the current branch
BASE: the common ancestor, how this file looked before both changes
REMOTE: the file you are merging into your branch
MERGED: the merge result; this is what gets saved in the merge commit and used in the future
You can navigate among these views using ctrl+w. You can directly reach the MERGED view using ctrl+w followed by j.
More information about vimdiff navigation is here and here.
You can edit the MERGED view like this:
If you want to get changes from REMOTE
:diffg RE
If you want to get changes from BASE
:diffg BA
If you want to get changes from LOCAL
:diffg LO
Save, Exit, Commit, and Clean up
:wqa save and exit from vi
git commit -m "message"
git clean Remove extra files (e.g. *.orig). Warning: It will remove all untracked files, if you won't pass any arguments.
Here's a probable use case, from the top:
You're going to pull some changes, but oops, you're not up to date:
git fetch origin
git pull origin master
From ssh://gitosis#example.com:22/projectname
* branch master -> FETCH_HEAD
Updating a030c3a..ee25213
error: Entry 'filename.c' not uptodate. Cannot merge.
So you get up-to-date and try again, but have a conflict:
git add filename.c
git commit -m "made some wild and crazy changes"
git pull origin master
From ssh://gitosis#example.com:22/projectname
* branch master -> FETCH_HEAD
Auto-merging filename.c
CONFLICT (content): Merge conflict in filename.c
Automatic merge failed; fix conflicts and then commit the result.
So you decide to take a look at the changes:
git mergetool
Oh my, oh my, upstream changed some things, but just to use my changes...no...their changes...
git checkout --ours filename.c
git checkout --theirs filename.c
git add filename.c
git commit -m "using theirs"
And then we try a final time
git pull origin master
From ssh://gitosis#example.com:22/projectname
* branch master -> FETCH_HEAD
Already up-to-date.
Ta-da!
I find merge tools rarely help me understand the conflict or the resolution. I'm usually more successful looking at the conflict markers in a text editor and using git log as a supplement.
Here are a few tips:
Tip One
The best thing I have found is to use the "diff3" merge conflict style:
git config merge.conflictstyle diff3
This produces conflict markers like this:
<<<<<<<
Changes made on the branch that is being merged into. In most cases,
this is the branch that I have currently checked out (i.e. HEAD).
|||||||
The common ancestor version.
=======
Changes made on the branch that is being merged in. This is often a
feature/topic branch.
>>>>>>>
The middle section is what the common ancestor looked like. This is useful because you can compare it to the top and bottom versions to get a better sense of what was changed on each branch, which gives you a better idea for what the purpose of each change was.
If the conflict is only a few lines, this generally makes the conflict very obvious. (Knowing how to fix a conflict is very different; you need to be aware of what other people are working on. If you're confused, it's probably best to just call that person into your room so they can see what you're looking at.)
If the conflict is longer, then I will cut and paste each of the three sections into three separate files, such as "mine", "common" and "theirs".
Then I can run the following commands to see the two diff hunks that caused the conflict:
diff common mine
diff common theirs
This is not the same as using a merge tool, since a merge tool will include all of the non-conflicting diff hunks too. I find that to be distracting.
Tip Two
Somebody already mentioned this, but understanding the intention behind each diff hunk is generally very helpful for understanding where a conflict came from and how to handle it.
git log --merge -p <name of file>
This shows all of the commits that touched that file in between the common ancestor and the two heads you are merging. (So it doesn't include commits that already exist in both branches before merging.) This helps you ignore diff hunks that clearly are not a factor in your current conflict.
Tip Three
Verify your changes with automated tools.
If you have automated tests, run those. If you have a lint, run that. If it's a buildable project, then build it before you commit, etc. In all cases, you need to do a bit of testing to make sure your changes didn't break anything. (Heck, even a merge without conflicts can break working code.)
Tip Four
Plan ahead; communicate with co-workers.
Planning ahead and being aware of what others are working on can help prevent merge conflicts and/or help resolve them earlier -- while the details are still fresh in mind.
For example, if you know that you and another person are both working on different refactoring that will both affect the same set of files, you should talk to each other ahead of time and get a better sense for what types of changes each of you is making. You might save considerable time and effort if you conduct your planned changes serially rather than in parallel.
For major refactorings that cut across a large swath of code, you should strongly consider working serially: everybody stops working on that area of the code while one person performs the complete refactoring.
If you can't work serially (due to time pressure, maybe), then communicating about expected merge conflicts at least helps you solve the problems sooner while the details are still fresh in mind. For example, if a co-worker is making a disruptive series of commits over the course of a one-week period, you may choose to merge/rebase on that co-workers branch once or twice each day during that week. That way, if you do find merge/rebase conflicts, you can solve them more quickly than if you wait a few weeks to merge everything together in one big lump.
Tip Five
If you're unsure of a merge, don't force it.
Merging can feel overwhelming, especially when there are a lot of conflicting files and the conflict markers cover hundreds of lines. Often times when estimating software projects we don't include enough time for overhead items like handling a gnarly merge, so it feels like a real drag to spend several hours dissecting each conflict.
In the long run, planning ahead and being aware of what others are working on are the best tools for anticipating merge conflicts and prepare yourself to resolve them correctly in less time.
Identify which files are in conflict (Git should tell you this).
Open each file and examine the diffs; Git demarcates them. Hopefully it will be obvious which version of each block to keep. You may need to discuss it with fellow developers who committed the code.
Once you've resolved the conflict in a file git add the_file.
Once you've resolved all conflicts, do git rebase --continue or whatever command
Git said to do when you completed.
Merge conflicts happens when changes are made to a file at the same time. Here is how to solve it.
git CLI
Here are simple steps what to do when you get into conflicted state:
Note the list of conflicted files with: git status (under Unmerged paths section).
Solve the conflicts separately for each file by one of the following approaches:
Use GUI to solve the conflicts: git mergetool (the easiest way).
To accept remote/other version, use: git checkout --theirs path/file. This will reject any local changes you did for that file.
To accept local/our version, use: git checkout --ours path/file
However you've to be careful, as remote changes that conflicts were done for some reason.
Related: What is the precise meaning of "ours" and "theirs" in git?
Edit the conflicted files manually and look for the code block between <<<<</>>>>> then choose the version either from above or below =====. See: How conflicts are presented.
Path and filename conflicts can be solved by git add/git rm.
Finally, review the files ready for commit using: git status.
If you still have any files under Unmerged paths, and you did solve the conflict manually, then let Git know that you solved it by: git add path/file.
If all conflicts were solved successfully, commit the changes by: git commit -a and push to remote as usual.
See also: Resolving a merge conflict from the command line at GitHub
For practical tutorial, check: Scenario 5 - Fixing Merge Conflicts by Katacoda.
DiffMerge
I've successfully used DiffMerge which can visually compare and merge files on Windows, macOS and Linux/Unix.
It graphically can show the changes between 3 files and it allows automatic merging (when safe to do so) and full control over editing the resulting file.
Image source: DiffMerge (Linux screenshot)
Simply download it and run in repo as:
git mergetool -t diffmerge .
macOS
On macOS you can install via:
brew install caskroom/cask/brew-cask
brew cask install diffmerge
And probably (if not provided) you need the following extra simple wrapper placed in your PATH (e.g. /usr/bin):
#!/bin/sh
DIFFMERGE_PATH=/Applications/DiffMerge.app
DIFFMERGE_EXE=${DIFFMERGE_PATH}/Contents/MacOS/DiffMerge
exec ${DIFFMERGE_EXE} --nosplash "$#"
Then you can use the following keyboard shortcuts:
⌘-Alt-Up/Down to jump to previous/next changes.
⌘-Alt-Left/Right to accept change from left or right
Alternatively you can use opendiff (part of Xcode Tools) which lets you merge two files or directories together to create a third file or directory.
Check out the answers in Stack Overflow question Aborting a merge in Git, especially Charles Bailey's answer which shows how to view the different versions of the file with problems, for example,
# Common base version of the file.
git show :1:some_file.cpp
# 'Ours' version of the file.
git show :2:some_file.cpp
# 'Theirs' version of the file.
git show :3:some_file.cpp
If you're making frequent small commits, then start by looking at the commit comments with git log --merge. Then git diff will show you the conflicts.
For conflicts that involve more than a few lines, it's easier to see what's going on in an external GUI tool. I like opendiff -- Git also supports vimdiff, gvimdiff, kdiff3, tkdiff, meld, xxdiff, emerge out of the box and you can install others: git config merge.tool "your.tool" will set your chosen tool and then git mergetool after a failed merge will show you the diffs in context.
Each time you edit a file to resolve a conflict, git add filename will update the index and your diff will no longer show it. When all the conflicts are handled and their files have been git add-ed, git commit will complete your merge.
I either want my or their version in full, or want to review individual changes and decide for each of them.
Fully accept my or theirs version:
Accept my version (local, ours):
git checkout --ours -- <filename>
git add <filename> # Marks conflict as resolved
git commit -m "merged bla bla" # An "empty" commit
Accept their version (remote, theirs):
git checkout --theirs -- <filename>
git add <filename>
git commit -m "merged bla bla"
If you want to do for all conflict files run:
git merge --strategy-option ours
or
git merge --strategy-option theirs
Review all changes and accept them individually
git mergetool
Review changes and accept either version for each of them.
git add <filename>
git commit -m "merged bla bla"
Default mergetool works in command line. How to use a command line mergetool should be a separate question.
You can also install visual tool for this, e.g. meld and run
git mergetool -t meld
It will open local version (ours), "base" or "merged" version (the current result of the merge) and remote version (theirs). Save the merged version when you are finished, run git mergetool -t meld again until you get "No files need merging", then go to Steps 3. and 4.
See How Conflicts Are Presented or, in Git, the git merge documentation to understand what merge conflict markers are.
Also, the How to Resolve Conflicts section explains how to resolve the conflicts:
After seeing a conflict, you can do two things:
Decide not to merge. The only clean-ups you need are to reset the index file to the HEAD commit to reverse 2. and to clean up working tree changes made by 2. and 3.; git merge --abort can be used for this.
Resolve the conflicts. Git will mark the conflicts in the working tree. Edit the files into shape and git add them to the index. Use git commit to seal the deal.
You can work through the conflict with a number of tools:
Use a mergetool. git mergetool to launch a graphical mergetool which will work you through the merge.
Look at the diffs. git diff will show a three-way diff, highlighting changes from both the HEAD and MERGE_HEAD versions.
Look at the diffs from each branch. git log --merge -p <path> will show diffs first for the HEAD version and then the MERGE_HEAD version.
Look at the originals. git show :1:filename shows the common ancestor, git show :2:filename shows the HEAD version, and git show :3:filename shows the MERGE_HEAD version.
You can also read about merge conflict markers and how to resolve them in the Pro Git book section Basic Merge Conflicts.
For Emacs users which want to resolve merge conflicts semi-manually:
git diff --name-status --diff-filter=U
shows all files which require conflict resolution.
Open each of those files one by one, or all at once by:
emacs $(git diff --name-only --diff-filter=U)
When visiting a buffer requiring edits in Emacs, type
ALT+x vc-resolve-conflicts
This will open three buffers (mine, theirs, and the output buffer). Navigate by pressing 'n' (next region), 'p' (prevision region). Press 'a' and 'b' to copy mine or theirs region to the output buffer, respectively. And/or edit the output buffer directly.
When finished: Press 'q'. Emacs asks you if you want to save this buffer: yes.
After finishing a buffer mark it as resolved by running from the teriminal:
git add FILENAME
When finished with all buffers type
git commit
to finish the merge.
Bonus:
In speaking of pull/fetch/merge in the previous answers, I would like to share an interesting and productive trick,
git pull --rebase
This above command is the most useful command in my Git life which saved a lot of time.
Before pushing your newly committed change to remote server, try git pull --rebase rather git pull and manual merge and it will automatically sync the latest remote server changes (with a fetch + merge) and will put your local latest commit at the top in the Git log. No need to worry about manual pull/merge.
In case of a conflict, just use
git mergetool
git add conflict_file
git rebase --continue
Find details at: What does “git pull –rebase” do?
Simply, if you know well that changes in one of the repositories is not important, and want to resolve all changes in favor of the other one, use:
git checkout . --ours
to resolve changes in the favor of your repository, or
git checkout . --theirs
to resolve changes in favor of the other or the main repository.
Or else you will have to use a GUI merge tool to step through files one by one, say the merge tool is p4merge, or write any one's name you've already installed
git mergetool -t p4merge
and after finishing a file, you will have to save and close, so the next one will open.
There are three steps:
Find which files cause conflicts by the command
git status
Check the files, in which you would find the conflicts marked like
<<<<<<<<head
blablabla
Change it to the way you want it, and then commit with the commands
git add solved_conflicts_files
git commit -m 'merge msg'
Please follow the following steps to fix merge conflicts in Git:
Check the Git status:
git status
Get the patchset:
git fetch (checkout the right patch from your Git commit)
Checkout a local branch (temp1 in my example here):
git checkout -b temp1
Pull the recent contents from master:
git pull --rebase origin master
Start the mergetool and check the conflicts and fix them...and check the changes in the remote branch with your current branch:
git mergetool
Check the status again:
git status
Delete the unwanted files locally created by mergetool, usually mergetool creates extra file with *.orig extension. Please delete that file as that is just the duplicate and fix changes locally and add the correct version of your files.
git add #your_changed_correct_files
Check the status again:
git status
Commit the changes to the same commit id (this avoids a new separate patch set):
git commit --amend
Push to the master branch:
git push (to your Git repository)
CoolAJ86's answer sums up pretty much everything. In case you have changes in both branches in the same piece of code you will have to do a manual merge. Open the file in conflict in any text editor and you should see following structure.
(Code not in Conflict)
>>>>>>>>>>>
(first alternative for conflict starts here)
Multiple code lines here
===========
(second alternative for conflict starts here)
Multiple code lines here too
<<<<<<<<<<<
(Code not in conflict here)
Choose one of the alternatives or a combination of both in a way that you want new code to be, while removing equal signs and angle brackets.
git commit -a -m "commit message"
git push origin master
You could fix merge conflicts in a number of ways as other have detailed.
I think the real key is knowing how changes flow with local and remote repositories. The key to this is understanding tracking branches. I have found that I think of the tracking branch as the 'missing piece in the middle' between me my local, actual files directory and the remote defined as origin.
I've personally got into the habit of 2 things to help avoid this.
Instead of:
git add .
git commit -m"some msg"
Which has two drawbacks -
a) All new/changed files get added and that might include some unwanted changes.
b) You don't get to review the file list first.
So instead I do:
git add file,file2,file3...
git commit # Then type the files in the editor and save-quit.
This way you are more deliberate about which files get added and you also get to review the list and think a bit more while using the editor for the message. I find it also improves my commit messages when I use a full screen editor rather than the -m option.
[Update - as time has passed I've switched more to:
git status # Make sure I know whats going on
git add .
git commit # Then use the editor
]
Also (and more relevant to your situation), I try to avoid:
git pull
or
git pull origin master.
because pull implies a merge and if you have changes locally that you didn't want merged you can easily end up with merged code and/or merge conflicts for code that shouldn't have been merged.
Instead I try to do
git checkout master
git fetch
git rebase --hard origin/master # or whatever branch I want.
You may also find this helpful:
git branch, fork, fetch, merge, rebase and clone, what are the differences?
If you want to merge from branch test to master, you can follow these steps:
Step 1: Go to the branch
git checkout test
Step 2:
git pull --rebase origin master
Step 3: If there are some conflicts, go to these files to modify it.
Step 4: Add these changes
git add #your_changes_files
Step 5:
git rebase --continue
Step 6: If there is still conflict, go back to step 3 again. If there is no conflict, do following:
git push origin +test
Step 7: And then there is no conflict between test and master. You can use merge directly.
Using patience
For a big merge conflict, using patience provided good results for me. It will try to match blocks rather than individual lines.
If you change the indentation of your program for instance, the default Git merge strategy sometimes matches single braces { which belongs to different functions. This is avoided with patience:
git merge -s recursive -X patience other-branch
From the documentation:
With this option, merge-recursive spends a little extra time to avoid
mismerges that sometimes occur due to unimportant matching lines
(e.g., braces from distinct functions). Use this when the branches to
be merged have diverged wildly.
Comparison with the common ancestor
If you have a merge conflict and want to see what others had in mind when modifying their branch, it's sometimes easier to compare their branch directly with the common ancestor (instead of our branch). For that you can use merge-base:
git diff $(git merge-base <our-branch> <their-branch>) <their-branch>
Usually, you only want to see the changes for a particular file:
git diff $(git merge-base <our-branch> <their-branch>) <their-branch> <file>
git log --merge -p [[--] path]
Does not seem to always work for me and usually ends up displaying every commit that was different between the two branches, this happens even when using -- to separate the path from the command.
What I do to work around this issue is open up two command lines and in one run
git log ..$MERGED_IN_BRANCH --pretty=full -p [path]
and in the other
git log $MERGED_IN_BRANCH.. --pretty=full -p [path]
Replacing $MERGED_IN_BRANCH with the branch I merged in and [path] with the file that is conflicting. This command will log all the commits, in patch form, between (..) two commits. If you leave one side empty like in the commands above git will automatically use HEAD (the branch you are merging into in this case).
This will allow you to see what commits went into the file in the two branches after they diverged. It usually makes it much easier to solve conflicts.
As of December 12th 2016, you can merge branches and resolve conflicts on github.com
Thus, if you don't want to use the command-line or any 3rd party tools that are offered here from older answers, go with GitHub's native tool.
This blog post explains in detail, but the basics are that upon 'merging' two branches via the UI, you will now see a 'resolve conflicts' option that will take you to an editor allowing you to deal with these merge conflicts.
Merge conflicts could occur in different situations:
When running git fetch and then git merge
When running git fetch and then git rebase
When running git pull (which is actually equal to one of the above-mentioned conditions)
When running git stash pop
When you're applying git patches (commits that are exported to files to be transferred, for example, by email)
You need to install a merge tool which is compatible with Git to resolve the conflicts. I personally use KDiff3, and I've found it nice and handy. You can download its Windows version here:
https://sourceforge.net/projects/kdiff3/files/
BTW, if you install Git Extensions there is an option in its setup wizard to install Kdiff3.
Then setup the Git configuration to use KDiff3 as its mergetool:
$ git config --global --add merge.tool kdiff3
$ git config --global --add mergetool.kdiff3.path "C:/Program Files/KDiff3/kdiff3.exe"
$ git config --global --add mergetool.kdiff3.trustExitCode false
$ git config --global --add diff.guitool kdiff3
$ git config --global --add difftool.kdiff3.path "C:/Program Files/KDiff3/kdiff3.exe"
$ git config --global --add difftool.kdiff3.trustExitCode false
(Remember to replace the path with the actual path of the KDiff3 EXE file.)
Then every time you come across a merge conflict, you just need to run this command:
$ git mergetool
Then it opens Kdiff3, and first tries to resolve the merge conflicts automatically. Most of the conflicts would be resolved spontaneously and you need to fix the rest manually.
Here's what Kdiff3 looks like:
Then once you're done, save the file and it goes to the next file with a conflict and you do the same thing again until all the conflicts are resolved.
To check if everything is merged successfully, just run the mergetool command again. You should get this result:
$ git mergetool
No files need merging
I always follow the below steps to avoid conflicts.
git checkout master (Come to the master branch)
git pull (Update your master to get the latest code)
git checkout -b mybranch (Check out a new a branch and start working on that branch so that your master always remains top of trunk.)
git add . and git commit and git push (on your local branch after your changes)
git checkout master (Come back to your master)
Now you can do the same and maintain as many local branches you want and work simultaneous by just doing a git checkout to your branch whenever necessary.
I understood what a merge conflict was, but when I saw the output of git diff, it looked like nonsense to me at first:
git diff
++<<<<<<< HEAD
+ display full last name boolean in star table
++=======
+ users viewer.id/star.id, and conversation uses user.id
+
++>>>>>>> feat/rspec-tests-for-cancancan
But here is what helped me:
Everything between <<<<<<< and ======= is what was in one file, and
Everything between ======= and >>>>>>> is what was in the other file
So literally all you have to do is open the file with the merge conflicts and remove those lines from either branch (or just make them the same), and the merge will immediately succeed. Problem solved!
GitLens for Visual Studio Code
You can try GitLens for Visual Studio Code. The key features are:
3. Easily resolve conflicts
I already like this feature:
2. Current Line Blame.
3. Gutter Blame
4. Status Bar Blame
And there are many features. You can check them here.
This answer is to add an alternative for those Vim users like me that prefers to do everything within the editor.
TL;DR
Tpope came up with this great plugin for Vim called fugitive. Once installed, you can run :Gstatus to check the files that have conflict and :Gdiff to open Git in a three-way merge.
Once in the three-way merge, fugitive will let you get the changes of any of the branches you are merging in the following fashion:
:diffget //2, get changes from original (HEAD) branch:
:diffget //3, get changes from merging branch:
Once you are finished merging the file, type :Gwrite in the merged buffer.
Vimcasts released a great video explaining these steps in detail.
I am using Microsoft's Visual Studio Code for resolving conflicts. It's very simple to use. I keep my project open in the workspace. It detects and highlights conflicts. Moreover, it gives GUI options to select whatever change I want to keep from HEAD or incoming.
git fetch <br>
git checkout **your branch**<br>
git rebase master<br>
In this step you will try to fix the conflict using your preferred IDE.
You can follow this link to check how to fix the conflict in the file.
git add<br>
git rebase --continue<br>
git commit --amend<br>
git push origin HEAD:refs/drafts/master (push like a drafts)<br>
Now everything is fine and you will find your commit in Gerrit.
Try Visual Studio Code for editing if you aren't already.
After you try merging (and land up in merge conflicts), Visual Studio Code automatically detects the merge conflicts.
It can help you very well by showing the changes made to the original one and if you should accept incoming or
current change (meaning original one before merging)'.
It helped me and it can work for you too!
PS: It will work only if you've configured Git with with your code and Visual Studio Code.
A safer way to resolve conflicts is to use git-mediate (the common solutions suggested here are quite error prone imho).
See this post for a quick intro on how to use it.
For those who are using Visual Studio (Visual Studio 2015 in my case)
Close your project in Visual Studio. Especially in big projects, Visual Studio tends to freak out when merging using the UI.
Do the merge in a command prompt.
git checkout target_branch
git merge source_branch
Then open the project in Visual Studio and go to Team Explorer → Branch. Now there is a message that says Merge is pending and conflicting files are listed right below the message.
Click the conflicting file and you will have the option to Merge, Compare, Take Source, and Take Target. The merge tool in Visual Studio is very easy to use.

How do I diff incoming changesets with Beyond Compare 4 and hg?

I have been using the mercurial and Beyond Compare 4 tools together for about 2 weeks now and feel fairly confident in my usage, however I still seem to have a problem when comparing incoming changesets against my current local codebase. The problem is emphasized when I attempting a complicated merge.
Just to clarify, I am avoiding the use of tools such as TortoiseHg,
although I do have it installed. I am searching for feedback via cmd line operations only.
My current templated method to pull down the incoming changesets via the following ( as an [alias] )
hg in --verbose -T "\nchangeset: \t{rev}\nbranch: \t{branch}\nuser: \t\t{author}\ndate: \t\t{date(date,'%m-%d-%Y %I:%M%p')}\ndescription: \n\t{desc|fill76|tabindent}\n\n{files % ' \t{file}\n'}\n----------\n"
As an example, here is a simplified (and cleverly abstracted) block returned ::
changeset: 4685
branch: Feature-WI209825
user: Jack Handy <jhandy#anon.com>
date: 01-19-2015 10:19AM
description:
Display monkey swinging from vines while whistling dixie
Zoo/MonkeyCage/Resources/Localization.Designer.cs
Zoo/MonkeyCage/Resources/Localization.resx
Zoo/MonkeyCage/Utility/Extensions.cs
If I were to be comparing changes locally, I would simply use the following command ::
hg bcomp -r 4685 -r default <optional file name>
and then I would get an instance of Beyond Compare with a folder structure and files and I could just navigate accordingly to view the changes...however, when I attempt to do this with a changeset that has yet to be pulled into my local repository, I can't.
How do I diff incoming changesets with my local repository?
---- UPDATE --------------------------------
I pursued the idea of bundling the incoming changes and then trying to use BC4 to diff the bundle to any given branch/revision on my local repo.
hg in --bundle "C:\Sandboxes\Temp\temp.hg"
This creates a compressed file archive containing all the new changes.
Now I simply need to diff this bundle with my local, however am having difficulty optimizing this. Currently, I am using variations on the following command:
hg -R "C:\Sandboxes\Temp\temp.hg" bcomp -r default
Alas, I am still having difficulty perfecting this...any insight is appreciated.
I don't see how you can, since your local repository doesn't yet have that changeset, so mercurial can't create a local copy of the revision, as it doesn't have visibility of what the change actually is.
The -p flag to hg incoming will show you the patch for each revision, but that isn't what you want.
Why not just pull the remote changes anyway? It wont hurt unless you actually update. You can then do your diff in the normal way.
hg diff is a local operation.
But you can simply call hg incoming -p in order to obtain a diff view of what you're going to pull. See hg help incoming for more options and refinement (e.g. if you need to diff against a specific rev etc)

Using Mercurial with Branches - publishing to FTPonly accessible WebServers

We are a small team of developers working with a Web Application which is published using a Web Server that is only accessible throught FTP.
Our workflow is the following one:
A developer is working out some requested feature locally
When its done, commits it and Pushes to a 'central' repository
Few times a day, one of developers publishes the files that have been changed to a testing WebSite, to let key users see how does features have been implemented.
Once in a week, we deploy to our production site
As our Webserver doesnt support SSH, we can't push changesets and update on the server, so we created a custom script which Transfers the changed files throught FTP.
Each time we use that script a new tag is created, so we know -using hg diff- the diference between tags (a release for us).
It was all fine until now, that we introduced branches in our workflow, to let a developer work on a radical changes in the code, and keep contributing in daily small changes which are published to production.
The problem is that hg diff doesnt support Branches (or seems that its still in development)
So, which would be the best way to do it ? some options we have been thinking about are:
Mounting FTP as a Volume localy (using MacFuse or similar) and use mercurial push/update But would be so sloooow.
Play around with Bundles and see if they can help us but seems quite complicated
Example
$ hg tag qa-001 /* init to see diferences QA Site */
$ hg tag prod-001 /* init to see diferences Production Site */
$ hg ci -m "working on a stable feature"
$ hg tag qa-002
$ hg ci -m "change on the stable feature"
$ hg tag qa-003
$ hg tag prod-002
$ hg ci -m "another change on stable"
$ hg pull ../CentralRepo /*Where there is another Branch with unstable files*/
With last operation, a new head is created , so now there are two heads (stable, and unstable branch)
$hg diff -r qa-003 -r tip
The Result of hg diff is showing up the Unstable Files without doing the merge
Many Thanks for your comments
In your example, you are creating tags, not (named) branches. Tags won't help you to create separate lines of development: they are just stand-alone identifiers assigned with particular revisions.
Creating branches
To start using branches, you probably want to review some tutorials, such as:
Chapter 8. Managing releases and branchy development (from Bryan O'Sullivan's book)
A Guide to Branching in Mercurial (Steve Losh)
Based on your description, you probably want to create prod and qa branches based on your current default branch, as well as any feature / topic branches you might want for radical changes.
Once you have these branches in place, it's very easy to compare them, merge between them, see what changes are pending from one to the other, and so on as your workflow demands.
Bundles
Play around with Bundles and see if they can help us but seems quite complicated
If you only have FTP access, then bundles probably won't help you. You could upload a bundle to the server via FTP, but you would need to be able to run hg on the server to unpack the bundle into a repository.

Is there any way to clone a repository from the web incrementally?

I'm on dialup in lousy place (yes, it still happens in 2011), and trying to clone a huge repository. It starts without problem, but every time the dialup disconnects (which is unavoidable, it seems), the !#%$* hg rolls everything back and I'm left again with an empty directory.
Is there a solution other than doing it on a remote PC and then downloading the whole thing by FTP or something?
In a bash-like shell you could do something like this:
$ hg init myclone
$ cd myclone
$ for REV in `seq 10 10 100` ; do hg pull -r $REV <REMOTEREPO>; done
Starting at 10, each pull downloads the next 10 revisions, up to 100. In case of a lost connection, adjust the first argument to seq to match what you've already pulled.
Depending on how flaky your connection is, there are two options for performing initial clones.
First, you can try so-called “streaming clones”. These minimize Time To First Byte, but do generally require a bit more data to be transferred.
Here’s how to do a streaming clone:
$ hg clone --uncompressed https://~~~~
Your second option will be a hg clone –-rev operation, followed by a number of incremental pulls. This behaves similarly to cloning a repository in some distant past and doing occasional updates.
$ hg clone --rev 5 https://~~~~
Based on the suggestions here,
I created a repo that did this.
https://github.com/nootanghimire/hg-clone-bash
It's optimized for a single repo, but i guess you can fork and work on it! :)

cvs to mercurial conversion gets tags wrong

I've tried all the recommended conversion techniques
Mostly they manage to get the latest version of the files right, but every one of them trashes my history. Many (most?) of the tags from my cvs project have at least one file in error when I run "hg up $tag"
My cvs repo is not all that complicated. Why can't anything convert it?
I'd like to dump cvs and convert to mercurial, but not without history.
To recap my frustration:
I tried hg convert
(tried --branchsort,--timesort, fuzz=0)
I tried cvs2svn and then hg convert.
tailor does not work with recent versions of mercurial
fromcvs disappeared from the face of the earth
hg-cvs-import has been abandoned for 4 years and doesn't work with recent versions of hg
I have tried using the two most recent versions of mercurial ( 1.5 and 1.5.1 ).
Mark, it's a sub-optimal solution, but when a company I was with did a CVS->Mercurial migration we decided that all we cared about were tag snapshots, so we build a little for loop like:
for thetag in $(cat LIST_OF_RELEASE_TAGS) ; do
cvs update -r $thetag
hg commit --addremove -m "snapshot $thetag" -u "import"
hg tag $thetag
done
That assumed a linear chain of tags, but we only pulled in the main/production branch. A more sophisticated loop would call 'hg update' before each commit to get parentage that reflects CVS branching.
It's definitely not "full history" but it was enough to make us feel good about continuing in Mercurial without loosing our ability to say "What the hell was in version 1.1.11?!" and we could always go back to cvs is cvs blame level history was needed.
fromcvs is back. I'm testing it out on a very large repo of ours, and it's extremely fast and handles incremental conversion.
I found a solution of sorts. I'm not thrilled with it, but it will have to do for now.
I was able to detect the tags that were causing trouble and omit those tags from the conversion. Missing tags are much better than wrong tags (assuming the original cvs repo is kept for backup)
WARNING: The following assumes you have made a copy of CVSROOT and are working on that. Do not muck with your original.
This is a bash solution that works for me on my linux box. It will probably burn your house down and invite your grade school bully to move next door to you. You've been warned.
It uses cvsps to identify the problem tags, rcs to delete them and then removes the tags from the CVSROOT/history. After removing the cvsps cache, the hg conversion works as expected.
CVSROOT=/path/to/your/copy
MODULE=cvsmodule
rm -rf ~/.cvsps ~/.hg.cvsps # this cache is EVIL!
BADTAGS="`cvsps -q -x $MODULE |grep Tag: |grep -e FUNKY -e INVALID | awk '{print $2}' `"
while [ ! -z "$BADTAGS" ];do
cd $CVSROOT/$MODULE
for badtag in $BADTAGS;do
echo removing tag $badtag
grep -lr $badtag . | xargs --no-run-if-empty -l1 rcs -q -n$badtag
grep -v "$badtag|$MODULE" < $CVSROOT/CVSROOT/history > $CVSROOT/CVSROOT/history_
mv $CVSROOT/CVSROOT/history_ $CVSROOT/CVSROOT/history
done
BADTAGS="`cvsps -q -x $MODULE |grep Tag: |grep -e FUNKY -e INVALID | awk '{print $2}' `"
done
rm -rf ~/.cvsps ~/.hg.cvsps # this cache is EVIL!
mkdir ~/hgcvt
cd ~/hgcvt
cvs co $MODULE
hg convert $MODULE
I realize now that there are certain fundamental incompatibilities between cvs tags and hg tags.
In cvs, a version of a file have tags associated with its different versions.
In hg, a version is an alias for a changeset . In other words the state of the working files at some snapshot in time
The distinction is subtle, but important.
It is possible to make a tagged release in cvs of a version that does not represent a snapshot in time. This is not possible in hg.
Of course one could apply patches to get replicas. However, this would create a lot of new heads on the repository with arguably little benefit (assuming the cvs repo is still kept around for posterity).
I'm afraid a perfect conversion from cvs to mercurial is not practical. Ry4an's solution would work for those who care only about recreating the versions. I am more interested in the history and evolution of the source files.
I wrote the following script to simply munge all the cvs tags in the $CVSROOT prior to the conversion. e.g tag "v321" becomes "v321_prehg". That way developers will know those tags are not-authoritative and they must go back to the read-only cvs tree.
#!/usr/bin/python
import os
import sys
import stat
def die(msg):
sys.stderr.write(msg)
sys.exit(1)
cvsroot =os.getenv("CVSROOT")
if cvsroot is None:
die("CVSROOT not defined" )
print "CVSROOT=%s" % cvsroot
for rcsfile in os.popen("find %s -name '*,v'" % cvsroot).xreadlines():
rcsfile = rcsfile.replace('\n','')
print "rcsfile:%s" % rcsfile
st=os.stat(rcsfile)
if st.st_mode & stat.S_IWUSR == 0:
os.chmod(rcsfile,st.st_mode | stat.S_IWUSR)
f = open(rcsfile,"r")
inlines=f.readlines()
f.close()
outlines=[]
insymbols=False
symbolsDone=False
for l in inlines:
if insymbols and not symbolsDone:
if l.find('\t') == 0:#tag line
l= l.replace(":","_prehg:",1)
else:
symbolsDone=True
else:
if l == "symbols\n":
insymbols=True
outlines.append(l)
f = open(rcsfile,"w")
f.writelines( outlines )
f.close()