I have some systems that automatically write .md to _posts/.
As you know, below the YAML front matter begins the HTML, in these cases it is arbitrary HTML coming from external sources, which sometimes breaks the build. This is all via the GitHub API and at some scale so manually deleting the post is not an option.
Is there a way to configure or programmatically interact with Jekyll or GitHub Pages to ignore files or commits that break the build?
If it helps: This is best effort, stability is more important than integrity. Each post is added in a separate commit, of course Jekyll build does not necessarily run on each commit, but in any case it would be fine to just drop all posts for example by rolling back all commits since the last successful build.
The _drafts folder might be of use to you if you have posts that are unfinished. See the documentation.
In terms of ignoring posts that break the build, is there no way to locally test the Markdown-to-HTML conversion before pushing to GitHub Pages?
You can use a Continuous Integration (CI) service with github (most have free plans).
The idea is to :
Commit your automatic extracts in a development branch.
Push this development branch on github.
Have Continuous Integration service build the branch and take action depending on the build result.
Depending on CI build success/error, you can choose to :
merge in master for publication is everything went ok
send you a mail if a branch build fails
make your site say hello world if result is 42
Related
I have my documents for read-the-docs in a github repository. Before pushing to the repo, I always run sphinx locally to produce the html. When I am satisfied I push my changes to the github repo and read-the-docs builds the documentation.
In a most recent change I have used a lot of math. My document looks fine produced locally, but the build by read-the-docs looks different especially for formatting the tabulators &.
Is there a way to produce the document in the same way as read-the-docs does it in order to avoid such surprises?
Thanks for any help.
Helmut
I'm using Mercurial and Fabric to deploy my site. I'd never used Fabric before so I copied an example fab file online and then changed the vars to match my own site.
Here are the lines of code:
def prod():
env.hosts = ['kevinburke.webfactional.com']
env.user = 'kevinburke'
def deploy():
require('hosts' , provided_by=[prod])
local ("hg tag --local --force production")
local ("hg push --remotecmd /home/kburke/bin/hg") # this works fine
run ("cd /my/web/directory; hg update -C production")
and this is invoked from the command line as
fab prod deploy
When I was the only person deploying the site, this worked with no problems. But recently I added two committers who are running the same fabfile, and when they try deploying the site, the remote version of the site doesn't update to the latest version - it updates only to the latest version that I tagged as production, not the one that they tagged.
I expect that it would use their "production" tag to update the file. Why does this happen? How can I get the program to behave as I expect in the future?
Thanks, Kevin
You can't publish local tags. This means that either your first step is already performed in the /my/web/directory repo, or that there is already a production called revision there (you can check with hg tags, hg branches and hg bookmarks).
You have several ways to fix your workflow (in order of preference):
use common-prefixed tags to distinguish between different production revisions, like production-23 or production-42, which you can parse on the production box.
Create a production branch, where every revision to deliver is merged into this branch. I recommend this one if you already have experience with branches.
Use the bookmark extension, and create a production bookmark to keep track of your deployed version. This looks like the solution you currently want to establish. When you want to use bookmarks, you need to enable them both on the server and all clients, and use hg push -B production to push the current state of your bookmark to the server. One disadvantage of this process is that you will never see if anyone had pushed an other bookmark to the server, since the transmission of the bookmark silently rewrites the bookmark on the server.
Use regular tags to keep track of the production version. On the one hand it seems to be wrong to use tags for this kind of tracking, since tags are meant to be static. On the other hand you would have a track about which revisions where alive on some point in time. But the first solution does the tracking job much better.
May be basic, but did you see that it did indeed do anything? It might be that nothing happened and the deployed was your "production" tag when you had run it.
Since hg tag --local means the tag is only for your local repo and is not versioned, I cannot think of any other reason. Others won't even be able to know about the tag.
At a point in our development process we send all *.resx files to a translator. The translator usually takes a week to send back the files. During this time no one is allowed to add, remove or update any resx file.
How can I configure mercurial to enforce that policy?
Our setup: Each dev works with a local clone of our central repository.
Nice to have:
I'll turn the "policy" on and off every few weeks. So ideally, I'd like something that is easy to configure at one place and that affect all devs.
I'd rather enforce that policy at the local repository level then at the central repository level because if we prevent the "push" on the central repository, it will be harder for the dev to undo the already locally committed changeset.
Thanks
UPDATE:
More info on the translation process:
Merging is not an issue here. The translator does not change the files that we sent to him. We send him a bunch of language neutral .resx (form1.resx) and returns a bunch of language specific resx (form1.FR.resx).
Why prevent adding new resx? Adding a resx occurs when we add a new UI to our application. If we do that after the translation package has been sent, the translator won't know about the new UI and we'll end up with a new UI with no translation.
Why prevent updating resx? If the dev changes a label value from "open" to "close", he has made a very important semantic change. If he does that after the translation package has been sent, we won't get the right translation back.
You cannot stop by people from committing changes to .resx files unless you have control over their desktop machines (using a pretxncommit hook), and even then it's easily bypassed. It's much more normal to put the check on the central server at time of push using a pretxnchangegroup hook, but you're right that they'll have to fix up any changesets and re-push, which is advanced usage. In either case you'd used the AclExtension to enforce the actual restriction.
Here are two alternate ways to go about this that might work out better for you:
Clone your repository at the start of the translation process, warn developers to leave .resx alone for awhile, apply the work of the translators when they're done, and then merge those changes back into the main development repository with a merge command that always gives the incoming changes priority: X . Then use a simple hg log command to find all the changes in .resx that just got overwritten and tell the developers to re-add them. Chide them at this time.
alternately
Make the .resx files a Subrepository of the larger outer repository. Then turn off write access to that resx repository during the forbidden period. Developers will be able to commit in the outer repository but not the inner one, but clones will still get both exactly as they always did.
For what it's worth, everyone else handles this problem with simple merging, .resx is (XML) text, and it merges just fine.
When working with a DVCS it's not always easy to exactly mirror your svn experience, but there's usually a better option anyway.
You could add *.resx to the hgignore file
We got all psyched about from from svn to hg and as the development workflow is more or less flushed out, here remains the most difficult part - staging and integration system.
Hopefully this question goes a bit further then your common 'how do I move from xxx to Mercurial'. Please forgive long and probably poorly written question :)
We are web shop that does a lot of projects(mainly PHP and Zend), so we have one huge svn repo, with like 100+ folders, each representing a project with it's own tags,branches and trunk of course. On our integration and testing server(where QA and clients look at work results and test stuff) everything is pretty much automated - Apache is set to pick up new projects automatically creating vhost for each project/trunk; mysql migration scripts right there in trunk too and developers can apply them through simple web-interface. Long story short our workflow is this now:
Checkout code, do work, commit
Run update on the server via web interface(this basically does svn up on server on a particular project and also run db-migration script if needed)
QA changes on the server
This approach is certainly suboptimal for large projects when we have 2+ developers working on the same code. Branching in svn was only causing more headaches, well, hence moving to Mercurial. And here is where the question lies - how does one organize efficient staging/integration/testing server for this type of work(where you have many projects, say single developer could be working on 3 different projects in 1 day).
We decided to have 'default' branch tracking production essentially and then make all changes in individual branches. In this case though how can we automate staging updates for each branch? If earlier for one project we almost always were working on trunk, so we needed one DB, one vhost, etc. now we potentially talking about N-databases per project, N-vhost configs and etc. Then what about CI stuff(such as running phpDocumentor and/or unit tests)? Should it only be done on the 'default'? On branches?
I wonder how other teams solve this issue, perhaps some best practices that we're not using or overlooking?
Additional notes:
Probably worth mentioning that we've picked Kiln as a repo hosting service(mostly since we're using FogBugz anyway)
This is by no means the complete answer you'll eventually pick, but here are some tools that will likely factor into it:
repositories without working directories -- if you clone -U or hg update null you get a repository with no working directory (only the .hg). They're better on the server because they take up less room and no one is tempted to edit there
changegroup hooks
For that last one the changegroup hook runs whenever one or more changesets arrive via push or pull and you can have it do some interesting things such as:
push the changesets on to another repo depending on what has arrived
update the receiving repo's working directory
For example one could automate something like this using only the tools described above:
developer pushes five changesets to central-repo/project1/main
last changeset is on branch 'my-experiment' so csets are automatually re-pushed to optionally created repo central-repo/project1/my-experiment
central-repo/project1/my-experiment automatically does hg update tip which is certain to be on the my-expiriment branch
central-repo/project1/my-experiment automatically runs tests in its working dir and if they pass does a 'make dist' that deploys, which might set up database and vhost too
The biggie, and chapter 10 in the mercurial book covers this, is to not have the user waiting on that process. You want the user to push to a repo that contains possibly-okay-code and the automated processed do the CI and deploy work, which if it passes ends up being a likely-okay repo.
In the largest mercurial setup in which I've worked (20 or so developers) we got to the point where our CI system (Hudson) was pulling from the maybe-ok repos for each periodically then building and testing, and handling each branch separately.
Bottom line: all the tools you need to setup whatever you'd like probably already exist, but gluing them together will be one-off sort of work.
What you need to remember is that DVCS (vs. CVCS) introduces another dimension to versioning:
You don't have to rely anymore only on branching (and get a staging workspace from the right branch)
You now have with DVCS the publication workflow (push/pull between repo)
Meaning your staging environment is now a repo (with the full history of the project), checked out at a certain branch:
Many developers can push many different branches to that staging repo: the reconciliation process can be done in isolation within that repo, in a "main" branch of your choice.
Or they can pull that staging branch in their repo and test things out before pushing back.
From Joel's tutorial on Mercurial HgInit
A developer don't necessary have to commit for other to see: the publication process in a DVCS allows for him/her to pull the staging branch first, reconcile any conflict locally, and then push to the staging repo.
I'm trying to sell our group on using Mercurial as a source repository rather than VSS. In the process of updating our build scripts, I'm running into an issue trying to retrieve files from the Hg repository.
Our builds are automated with NAnt and currently work for local builds or builds from VSS (ie, pull the source as needed from VSS). I'm trying to update them to work with Mercurial as well.
Basically, when I'm working with single files, I don't have any issues since I can just use NAnt's 'get' task (after getting the appropriate revision hash) to retrieve the individual file.
The problem that I'm having is when I need to work with a directory (and subdirectories) of files that aren't at the root of the repository. I can't seem to figure out the proper commands to retrieve/copy a subdirectory from the repository to my 'working' directory for the builds. I've spent basically the whole afternoon trying to figure out how to do this with the mercurial executables (so I can use a NAnt 'exec' task), and have basically hit a wall so I figured I'd try posting here.
Can someone confirm whether this is possible, and provide some suggestions as to how I might be able to do this? I realize that Mercurial tracks changes by files and not directories, but it seems odd to me that this isn't available out of the box (from what I can tell).
If it's just not possible, the only workarounds I see are either maintaining NAnt fileset lists of expected files to work with (ugh!), or cloning the entire repository to a temporary directory and then just copying the files from that source as needed (this feels like a cludge to me).
I realize that I could simply create another repository for the directory that I want to work with, but I'd prefer to not go that route since I think that would increase the complexity of what I'm trying to do by a significant amount (I would have to apply this a large number of times for all of the different libraries that we build..).
Mercurial doesn't let you get only part of a repository. You have to get the whole tree. It's much more whole-repo focused than svn is.
You could try and segment your repository into multiple repos and manage them using the subrepos feature. Then you can pull the subdirectories independently.