I suspect this might be really obvious but I can't find a straightforward solution in the documentation or forums:
I have written some code that is held in a Mercurial repository on BitBucket.
I use this code to build Linux virtual servers. When I build a server, I clone the repo onto the server, run my build script, and then delete the clone. The result is a configured server with several files from my repo located in various folders on the server.
Now, I'm looking for a mechanism where I can roll out bug fixes and improvements to my users' servers after I have handed them over. At that time, I won't have SSH access to the servers and I cannot expect my end users to do anything more complicated than kick off a cron job or launch a script.
To achieve this, I'm thinking of setting up a BitBucket account for my users with read-only access to my repo.
I have no problem writing a script to clone my repo, via this read-only account, and apply the updates, but I don't want to include all my files. In particular, I want to exclude my build script as it is commercially sensitive. I know I could remove it from my repo, but then my build wouldn't work.
Reading around, it seems I may need to create a branch or a fork of my repo (which?). Or maybe a sub-repo? Then, I could remove the sensitive files from that branch/fork/sub-repo and allow my users to clone it via a script.
That's OK, but I need a way to update the branched/forked/sub repo as I make changes to the main one. Can this be automatic? In other words, can it be set up to always reflect the updates made in the main repo? Excluding the sensitive files of course.
I'm not sure I'd want updates to be automatic though, so I'd also like to know how to transfer updates from the main to the branch/fork/sub manually. A merge? If I do a merge, how do I make sure my sensitive files don't get copied across?
To sum up, I have a main repo which contains some sensitive files and I need a way to roll out updates of all but those sensitive files to my read-only users.
Sorry if this is hugely obvious. I'm sure it's a case of not seeing the wood for the trees and being overwhelmed by the possibilities. Thanks.
I don't think that you need to solve this in Mercurial at all.
What you actually need is Continuous Integration / a build server.
The simplest solution goes like this:
Set up a build server with something like TeamCity or Jenkins, that's always online and monitors changes in your Bitbucket repository.
You can set it up so that when there's a change in your repository, the build server runs your build script and copies the output to some FTP server, or download site, or whatever.
Now you have a single location that always contains the most recent code changes, but without the sensitive files like the build script.
Then, you can set up a script or cron job that your end users can run to get the newest version of the code from that central location.
You are ok with two branches, one for the users clone (main) and other for your main development (dev), the tricky part is merging the new changes from dev to main.
You can solve this by excluding files in the merge process. Excluding a file while merging in Mercurial
By setting the [merge-patterns] section in your .hgrc you can sepcify what files are not affected by the merge.
[merge-patterns]
build.sh = internal:local
For more info read hg help merge-tools.
"internal:local"
Uses the local version of files as the merged version.
Entire Mercurial trees always get moved around together, so you can't clone or pull just part of a repository (along the file tree axis). You could keep a branch that has only part of the files, and then keep another branch that has everything, making it easy to merge the the partial (in terms of files) branch into the other branch (but merging the other way wouldn't be particularly easy).
I'm thinking maybe subrepositories work for your particular use case.
Related
Whenever we change a Teamcity build definition (e.g. to use a new version of the NUnit console runner), it breaks the builds if we need to hotfix an earlier version of our code. To address this, I'd like to store the build definition alongside the code, so that it executes the build as it looked at the time the hotfix is branched from.
I've been looking into exporting the TC build definitions as Kotlin scripts, but I have a couple of issues with them:
Versioned Settings is controlled by a higher-level project I don't have access to, and is stored in a separate repository (and not as Kotlin).
It seems like it'll just look at your master branch and import settings from there, but I need the build to use whatever is on my hotfix branch at the point of execution.
In the past, we've solved this with Fake scripts, but the experience tends not to be very good for identifying problems.
What's the best way to execute builds on TeamCity to meet the following requirements?
Output as close to normal build steps as possible
Build defined within the branch/code under execution
Easy to maintain/manage
Test output should remain the same as in normal TC builds
You can have the TeamCity build configuration automatically stored in your repo. So changes made in the configuration will be committed to master and leaving your configuration on the branch intact. If no usable configuration is found on the branch, the configuration on the master will be used.
https://www.jetbrains.com/help/teamcity/storing-project-settings-in-version-control.html
Here's my scenario: When I initially created my Mercurial repo, I used hg add to add all *.pl *.sh and *.sql scripts to the repo. I later learned how to use the .hgignore file to exclude other files from the repo. One of the files I needed to exclude was a *.sql file that is generated by a script, so it is essentially a data file that constantly changes when the script runs that produces it; thus, I added it explicitly to the .hgignore file a few revisions ago.
Today, I want to update to a prior revision before this *.sql file was added to the .hgignore, so that I can create a branch off of it. However, when I try to update the working directory to this prior version, I get the following error:
a.sql: untracked file differs
abort: untracked files in working directory differ from files in requested revision
I know that one way I could get around this problem is to delete the file before trying to update to the prior revision, either by manually deleting it or using hg update --clean --check.
That may work in this particular case, since the file is auto-generated by a script each time, and so I don't care about the data that is currently in it.
However, I'm trying to find out what is the safe way people would generally handle this situation when they decide to ignore a file set (like a set of data files that aren't auto-generated) and need to return to a previous revision before they were marked to be ignored, especially if they wanted to retain the most current content in those file sets while still being able to view earlier revisions of files that Mercurial is actively tracking.
I've also considered that you could backup the files, but I think that is only a reasonable solution if this is a one-off case. If you want the ability to hg update to previous revisions on a frequent at-whim basis, then it becomes quite tedious to backup the data each time before you update to a earlier revision (it's also not a reliable way to guarantee that others may not delete the data that isn't being tracked in the repo).
Thanks for the help.
However, I'm trying to find out what is the safe way people would generally handle this situation when they decide to ignore a file set (like a set of data files that aren't auto-generated) and need to return to a previous revision before they were marked to be ignored, especially if they wanted to retain the most current content in those file sets while still being able to view earlier revisions of files that Mercurial is actively tracking.
It depends.
If you have exclusive control over this repository, and have the practical ability to require everyone to re-clone from it, then you can use hg convert to exclude the files from the old revisions. This is by far the cleanest option, but it will change the revision identifiers (hashes) for those revisions and all of their topological descendants. This is why everyone has to re-clone; their old clones will not interact properly with the new repository.
If you can't do that, you can copy the files somewhere else (you do have backups already, right?), clobber the originals with the old versions, and then restore them from your copy. This has to be done whenever you check the files out, so it is definitely suboptimal. You may be able to make this slightly easier by keeping the files outside the repository and checking in symlinks to the files, but you'll still have to fix up the symlinks whenever you checkout an old version.
However, what you describe is not the normal use case for Mercurial. Typically, untracked files are autogenerated, or at least able to be regenerated from tracked files. The operating assumption is that untracked files are not important and can be discarded at any time. Mercurial doesn't actually do this, because that would be rude, but neither does it make any special effort to preserve them when (for example) you make a bundle of the repository.
If you need to deal with versioning of object files, it is typical to store them in a separate artifact repository or some other system. This can be more difficult to manage because you have to reunite the binaries with the source code when you do a build. But it is much more robust than keeping the binaries loose in the repository and hoping they won't get accidentally overwritten or deleted.
Another option is to collapse the binary to text and then place the text under version control. This is always possible (e.g. take a hexdump) but may or may not be practical or reasonable, depending on the file format. For a compressed file format (e.g. tarballs, most image files, etc.) the hexdump is not going to be any easier to 3-way merge than the original binary, so there's little point in it. Similarly, if the binary is huge, the hexdump will be huge too. On the other hand, if a binary is compiled from source code, it is entirely normal to store the source and discard the binary. For something structured like an SQLite database, you might try storing an SQL script which will generate the database. For a zip file or tarball, store the contents. And so on. All of these things can be regenerated using make or a similar tool whenever you check things out, and you can automate this with a repo hook.
For several years, we have been using Mercurial internally (version 3.1.2) on a designated server, which is now being retired.
On a new server, the entire repository has been copied over, and Mercurial 4.0.0 installed. Everything seems to work, but
http://192.168.0.3:8000/?sort=lastchange
we make heavy use of the above command to inform 'pullers' of repositories that have been updated by other users.
It doesn't order correctly, the last modified date of a repository committed to doesn't update (isn't reported as being updated).
I'm stuck!
It all works fine, the situation arose because of an ignorance of how Mercurial works.
Originally, the parent folder for all our repositories was copied from the old server to the new. After exhausting all other routes, this copy was redone by zipping, copying and unzipping.
(Lasse, one of Mercurial's roles is to inform and manage the situations you describe. We reduce this possibility by knowing which repos to pull every morning, before starting work. Try it!).
I am (still) trying to completely migrate our company's SVN to HG.
For the most part I've succeeded, but we ran across a problem.
Our codebase has over 30 different projects, each one on its folder.
I've been asked multiple times how to commit and then push specific files to our central repository instead of being forced to commit everything everywhere to then push it, it's certainly annoying. Not being able to pull only specific projects is also an nuisance.
Is there any way to handle this like we used to in SVN? Where we could just commit what we wanted and not everything, and update only what was necessary.
Thank you.
A major difference between SVN and Mercurial is that you should have one repository per project in Mercurial.
You can change your repository to be multiple repositories using the convert extension.
Like Steve Kaye said you should create one repo per project, but as well you may want to create one master repo and include all your projects as subrepos This will allow svn like behavior of getting a copy of everything.
I have some files I'd like to add to have them as a "backup". The thing is, I'd like to commit them only one time, and then, I'd like for Mercurial to don't track them anymore ( don't notify me if they're changed, and don't commit them on other commits ).
Basically, something like this:
hg add my_folder
hg commit -m "added first version of my_folder"
Then, after a while, the contents of that folder might change. And if I commit other files, the new version of that folder will get commited as well. This is something I'd like to avoid. Is it possible, without specifying directly which files I want to commit?
I've never seen any option in Mercurial that might allow that... but why not simply copy them elsewhere ?
I mean, what's the point of using a Version Tracking System if you don't need versioning on these items anyway ?
We ran into a similar case with binary documents ('.doc', images, etc...) and finally decided to commit them on a separate repository, dedicated to those.
I think the traditional way of doing this is to commit files named something like "file.ext.default", and just inform users that they should copy the defaults and modify the copies.
VCSs aren't backup sysytems. consider using a proper backup mechanism.
having said that you should be able to do this using hooks, there are many ways you could do this but ACLs would be an obvious one assuming a remote server