I've just moved a large project out of a legacy version control system, and into Mercurial. I'm now faced with the task (opportunity!) of overhauling my old build system to work with this new source control system.
Are there any good build systems that:
Can build projects in a variety of languages and compilers
Integrate well with Mercurial (For example, knowing to pull certain revisions or tagged versions of sub repositories for dependency management (as opposed to continuous integration.))
Extendable (we have several auditing steps that we need to perform before and after our build)
Are easy to use
that are already out there? As much fun as building and maintaining yet another build system might be, I'd rather reuse a mature, stable product and get back to developing my own code. :) If no tool exists with Mercurial in mind, what else might meet the requirements? What else might be out there that I'm forgetting?
Usually your build tool/system and your continuous integration system are separate, and the later is the one that knows about your DVCS system. In my setup the build tool is sbt (formerly ant) and the CI system is Jenkins (which I highly recommend).
The CI system speaks Mercurial, but it doesn't have to speak much of it. Mercurial lets you create URLs that track a specific named branch or tag, and so long as your CI system can clone/pull you're set. Once the CI system has updated to the version you want to build then it triggers you build system, which needs only look at the working directory.
Related
Disclaimer: I am relatively unfamiliar with the flash build processes, so some/all of this may be misinformed nonsense. Please feel free to suggest alternative approaches.
We're currently developing a flex web app and our build situation is far from ideal. At present we're (as in individual developers) just building using FlashBuilder and deploying manually. The programmers are currently screaming bloody murder for two reasons, though:
The lack of CI is like going back to the stone age
We don't much care for FlashBuilder
(Note: We're only using FlashBuilder because it was the easiest way to set up a flex project in conjunction with Away3d and get it building / rendering correctly -- it's a stopgap solution).
As a predominately .NET development shop, we're used to doing continuous integration as well as continuous deployment. Ideally, we'd like to get something comparable to this for our flash projects without tying ourselves to a particular IDE.
Requirements:
The build process must be:
.. runnable via the commandline
.. runnable on both developer and CI build machines (and certainly not requiring an IDE!)
.. preferably as IDE-independent as possible (pragmatism will kick in though; if this causes a lot of friction we'll just pick one).
.. able to run on Windows (we develop using Windows)
We don't mind a touch of duplication or a few manual steps (e.g. tarting up the build scripts if we add a new project via an IDE, or generating one configuration from another if tools exist), but the less duplication / maintenance required the better.
I've read quite a few articles / blog posts and watched some short screencasts, but most of them are very thin on the ground on how the build system sits alongside IDEs. Most articles/screencasts have the same formula: How to create a "Hello World" build using a single file & text editors (no IDE).
I've not seen the topic of multiple libraries/projects etc. being broached, either.
After reading around the issue for a while, I'm considering investigating the following options:
Project Sprouts
Flexmojos
Maven Flex Plugin
buildr as3
Does anyone have any experience of the above solutions (or others I'm unaware of) and, if so, what do you make of them? Any help / pointers appreciated.
I recently started building with Gradle and the GradleFx plugin and I immediately fell in love with its power and ease of use.
Gradle is ANT + Maven + Ivy evolved and is primarily used from the command-line. You can:
write scripts in Groovy (a powerful Ruby-like language that runs in the Java Virtual Machine)
access all existing Maven and Ivy repositories as well as your own repos
use existing ANT tasks
integrate with CI (in Jenkins you just tick a checkbox to activate Gradle support)
although it has originally grown from the Java/Groovy community, it is in fact language agnostic. You add language-specific plugins for added functionality. GradleFx is such a plugin that provides you with additional ActionScript/Flex building tasks.
do easy multi-project builds. e.g. you can compile, unit test, package and deploy both your .NET service layer and your Flex client application with just one command.
use convention over configuration: if you stick to the conventions, your build scripts will be extremely terse
generate all kinds of reports: unit testing, checkstyles, codenarc, ...
generate Eclipse, IDEA or other IDE projects
all the things I haven't discovered yet
And best of all: it's very easy to learn. I had no knowledge of Maven before I started with Gradle and could get a multi-project build with some customizations working quite quickly.
Edit (comparison to Buildr AS3 and Maven)
I can compare this only to one of the projects you mentioned: Buildr AS3. It seems to start from a philosophy that is similar to Gradle's. But I've tried to use it about half a year ago and couldn't even get a simple 'Hello World' app to work. I e-mailed the developer for help: no response.
Compared to GradleFx: I had a small forum discussion with the developer (on a rather philosophical topic, since I didn't really need any help because it just worked right away). He answered within minutes.
As for Maven: (for what it's worth) I've only glanced at some configurations and they seem overly complicated when I compare them to a Gradle script.
There is one thing Maven does that you can't do with GradleFx (yet). It can pull the right Flex SDK from a Maven repo and build against that. With GradleFx you have to have your SDK's available locally.
I'm quite familiar with using maven as the main build tool and the flexmojos plugin from Sonatype. My experience has been a bit of a roller coaster with flexmojos. Maven is completely solid, it works all the time without issue, the only issue is the flexmojos plugin which has fluctuated a lot between versions. If you choose to go this route make sure to grab the source for flexmojos so you can see what your configuration options are actually doing to the command line parameters etc. For Flex 3.x flexmojos 3.x up to around 3.9 is good and works fine with regard to the goal for generating the .project eclipse files, believe there's also a mojo (a maven plugin) for generating intelliJ IDEA project files as well as others. If you're using Flex 4 you can compile with the latest flexmojos 4.0RC2 but it appears to me that the goal for generating flex/flashbuilder project properties is now gone (I'm not sure if this is because it's been replaced by another plugin altogether or what the deal is). However building with maven and flexmojos does fulfill all of your goals above (we also use it for building our service layer, so in a single mvn clean install we get a jar packed in a war packed in an ear with everything configured and a swf, that part is really nice). Also you can do continuous integration using bamboo (or simply write your own script that is triggered from a cron job or in windows as a batch file executed with a scheduled task if you don't have a *nix server around). Let me know if you'd like any more details or if I missed something major.
Shaun
I have been using Hudson, now Jenkins, with Ant for Flex automated builds and FlexUnit testing. Jenkins has some really useful plugins for integration with eclipse (and hence, FDT or FlashBuilder), Jira, SVN, Git etc., and it's free. Also, you can integrate the Ant build into Maven scripts, so I've found this to be a good and flexible solution for all purposes I've come across so far.
The Flex SDK comes with Ant tasks, and writing even elaborate Ant build scripts is quite easy - in fact, I'd been using Ant locally before, and I could reuse my existing scripts with only a few added extra compiler options for FlexUnit tasks.
However, it took a while to set up the system correctly for unit testing, because I'm running a headless server on Linux, and that implicates a rather complicated environment for ActionScript tests, because they run only in Flash Player. This, of course, is true for all CI scenarios using FlexUnit, regardless of which server you use.
Here's what I've learned:
FlexUnit needs a standalone debugger version of Flash Player installed, but Adobe only distributes binaries for the standard version on Linux. Therefore, compiling from source was necessary, and since my server system is stripped down to the bare necessities, it took some effort to install all the correct dependencies and get them to work.
The Flash Player needs hardware to run correctly: It uses graphics, therefore it needs a graphics card, and sounds, therefore it needs a sound card. On my headless server, this meant I had to install a VNC host to get it to run at all, and I had to eliminate any tests using sounds (those will now only run on local machines). If anyone ever comes across a working sound card emulation for openSuSE that I could use with the VNC client - you'll be my hero forever!
If you've set asynchronous timeouts in your unit tests, and/or you need to use setTimeout() to send delayed procedure calls, make sure the intervals aren't too short - I've had problems with tests that ran fine on any local machine, but broke the build on the CI server, because the Flash Player is considerably slower on the VNC client than on an actual graphics card.
I've also found this last issue to be a healthy lesson: Criteria for unit tests should not be based on assumptions about the system's performance, or at least be tolerant enough to succeed even on a slow machine.
Time and again I am faced with the issue of having multiple environments that must be configured individually for an application that would run in all of them (e.g. QA, regional production env's, dev, staging, etc.) and I am wondering what would be the best way to organize different configurations?
Would it be in the database? Different configuration files per environment? Or maybe the same file with different sections/xml tags? How would these be then deployed? Embedded within the app? Or put manually in after installation to be modified in-place?
This question is not technology-specific - I've worked with .net and Java, web-apps and desktop apps and this issue comes up time and again. I'm looking to learn different approaches to maybe adapt a hybrid to address this.
EDIT: There's one caveat that I must point out - when configuration is part of the deployed solution, it is generally installed under root user on the host. In large organizations developers usually don't have a root access to production hosts so any changes to the configuration require a new build and deployment. Needless to say this isn't the nicest approach - especially at organizations that have a very strict release process involving multiple teams and approval levels... (sigh I know!)
Borrowed from Jez Humble and David Farley's book "Continuous Delivery (page 41)", you can:
Your build scripts can pull configuration in and incorporate it into your binaries at build time.
Your packaging software can inject configuration at packaging time, such as when creating assemblies, ears, or gems.
Your deployment scripts or installers can fetch the necessary information or ask the user for it and pass it to your application at
deployment time as part of the installation process.
Your application itself can fetch configuration at startup time or run time.
It is considered bad practice by them to inject configuration files in build and compile times, because you should be able to deploy the same binary file to every environments.
My experience was that you could bake all configuration files for every environments (except sensitive information) to your deployment file (war, jar, zip, etc). And you design your application to take in an extra parameter when starts, to pickup the right sets of configuration files (from your extracted deployment file, or from local/remote file system if they are sensitive, or from a database) during application's startup time.
The question is difficult to answer because it's somewhat vague. There is no technology-agnostic approach to configuration as far as I know. Exactly how configuration is set up will depend on the language/technology in question.
I'm not familiar with .net but with java a popular approach is to have a maven build set up with different profiles. Each profile is specific to an environment. You can then define different properties files that have environment-specific values, an example from the above link is:
environment.properties - This is the default configuration and will be packaged in the artifact by default.
environment.test.properties - This is the variant for the test environment.
environment.prod.properties - This is basically the same as the test variant and will be used in the production environment.
You can then build your project as follows:
mvn -Pprod package
I have good news and bad news.
The good news is that Config4* (of which I am the maintainer) neatly addresses this issue with its support for adaptive configuration. Basically, this is the ability for a configuration file to adapt itself to the environment (including hostname, username, environment variables, and command-line options) in which it is running. Read Chapter 2 of the "Getting Started" manual for details. Don't worry: it is a short chapter.
The bad news is that, currently, Config4* implementations exist only for C++ and Java, so your .Net applications are out of luck. And even with C++ and Java applications, it won't make pragmatic sense to retrofit Config4* into an existing application. Because of this, I'd advise trying to use Config4* only in new applications.
Despite the bad news, I think it is worth your while to read the above-mentioned chapter of the Config4* documentation, because doing so may provide you with ideas that you can adapt to fit your needs.
I just started working in an environment that uses multiple programming languages, doesn't have source control and doesn't have automated deploys.
I am interested in recommending VSS for source control and using CruiseControl.net for automated deploys but I have only used CC.NET with ASP.NET applications. Is it possible to use CC.NET to deploy python, php, ASP.NET and ??? apps all from the same instance?
Yes you can.
CC .net is written in .Net but can handle any project. Your project langage does not matter, you can still use Batch, Powershell, Nant or MsBuild scripts. You may also use Cruise Control or Hudson, as you like.
As for the source control provider, I would prefer svn (or even git) but that's more a matter of habbits : from my point of view VSS is too linked to VS and I don't like the lock on check out by default behaviour.
VSS is unsafe for any source and is damn near useless outside visual studio. And cruise control is painful to learn and make work at best. Your heart is in the right place, but you probably want slightly different technical solutions. For SCM, you probably want either subversion (cf VisualSVN Server) or Mercurial (on IIS7). For continuious integration, I'd look at TeamCity first or Hudson second. Either of which is vastly superior to CCNet in terms of ease of use.
Our group uses Visual Sourcesafe as a file repository for all of our "content" (HTML, CSS, Javascript, JSP). None of it requires building or compilation but we would like to automate the copying of it to a Unix dev server upon check-in.
I have used Cruisecontrol.NET in the past for CI at other companies but it was for .NET. What would be the easiest way to achieve our current requirements? Would using CruiseControl.NET be overkill or even a good idea? Thanks in advance.
-Sean
This sounds like overkill for a CI tool.
Visual SourceSafe and other version control systems should have hooks allowing you to automate a simple file copy operation.
From http://msdn.microsoft.com/en-us/library/aa302175.aspx
Use events, such as OnBeforeCheckout
or OnAfterCheckIn to automate your
process.
Whether this makes sense for you depends on a couple of factors. If you are talking about a large, geographically team with only change based deployment then yes, those are valid concerns. If you only have a few local developers and you deploy the world on each copy operation, then no, I don't think you'd need a CI tool.
This is not to say other reasons may influence you to use a CI tool, testing for instance. Your problem might also be solved by running a polling script on the Unix box to sync the source control with the dev server. I guess the main point is, if you are deploying all non-compiled software, why do you have a separate source control and dev server? You're deployment can be done by a source control tool. If it is only for backup, there are plenty of existing solutions for that problem.
Sean,
Our AnthillPro customers do this kind of thing pretty frequently (and we even do it internally when new content is committed for our website). It's a really good idea, totally appropriate for a CI tool, and you can get quality feedback if you wire in some automated functional / regression tests.
Eric
You could try using Hudson http://hudson-ci.org/
It is easy to configure, is completely GUI (unless you want to go into the details), and has a plugin for Visual Sourcecafe http://wiki.hudson-ci.org/display/HUDSON/Visual+SourceSafe+Plugin
While CI would probably be overkill for what you are trying to do, since Hudson is all GUI and easy to use, you would not spend a lot of time just trying to configure it.
Hudson also has plugins for copying stuff over to other systems, and so it would be easy to deploy your content to another system.
If you are worried about the process, get in touch with a hosted CI provider, such as MikeCI, a quick message on their support board will get you the answer. I don't see why triggering a "build" can't be replaced with copy and paste!
We are developing a closed-source project, versionned with Mercurial. We are using two libraries in our project :
One of those libraries is being developed by a third-party. They are using git, and we usually just pull from their repo once in a week to get the latest changes.
The other library is being developed by ourselves, and is under active development. It must live in its own public mercurial repository, as it is licensed under LGPL. (It's a fork of a third-party LGPL component, ported to our platform)
So my question is: How should I organize the source to ensure that:
A developer from our team should be able to get all the source (main project + libraries) with a single "clone" command
We should be able to pull easily the latest changes from the libraries, even though one of them is managed by git
Should we use mercurial sub-repos functionnality, with hg-git to access to the library under git? Is it well supported by TortoiseHg and BitBucket? (pros: easy to pull library changes / cons: does it works well?)
Or should we keep only snapshots of the libraries under our project? (thus, when there are new upstream changes in the libraries, we pull them to a separate place, and then copy the whole source to our project? (pros: will work / cons: pain in the ass, especially for the library that is being developed by ourselves, which is subject to a lot of daily changes)
Yeah, use the subrepo with hg-git. It's easy, well-supported, and effective. Your .hgsubstate file will include pointers to snapshots of the subrepos, and that file is controlled, so at any point in time you'll be able to answer the question: what version of library X was this working against. It's good stuff.
Alternately, you can use a dependency manager like ivy or maven as appropriate for your language, but don't include their libraries in your repo if you can avoid it. Pointers to versions of their code is better and a dependency manager or subrepos are the cleanest ways to do that.