Our group uses Visual Sourcesafe as a file repository for all of our "content" (HTML, CSS, Javascript, JSP). None of it requires building or compilation but we would like to automate the copying of it to a Unix dev server upon check-in.
I have used Cruisecontrol.NET in the past for CI at other companies but it was for .NET. What would be the easiest way to achieve our current requirements? Would using CruiseControl.NET be overkill or even a good idea? Thanks in advance.
-Sean
This sounds like overkill for a CI tool.
Visual SourceSafe and other version control systems should have hooks allowing you to automate a simple file copy operation.
From http://msdn.microsoft.com/en-us/library/aa302175.aspx
Use events, such as OnBeforeCheckout
or OnAfterCheckIn to automate your
process.
Whether this makes sense for you depends on a couple of factors. If you are talking about a large, geographically team with only change based deployment then yes, those are valid concerns. If you only have a few local developers and you deploy the world on each copy operation, then no, I don't think you'd need a CI tool.
This is not to say other reasons may influence you to use a CI tool, testing for instance. Your problem might also be solved by running a polling script on the Unix box to sync the source control with the dev server. I guess the main point is, if you are deploying all non-compiled software, why do you have a separate source control and dev server? You're deployment can be done by a source control tool. If it is only for backup, there are plenty of existing solutions for that problem.
Sean,
Our AnthillPro customers do this kind of thing pretty frequently (and we even do it internally when new content is committed for our website). It's a really good idea, totally appropriate for a CI tool, and you can get quality feedback if you wire in some automated functional / regression tests.
Eric
You could try using Hudson http://hudson-ci.org/
It is easy to configure, is completely GUI (unless you want to go into the details), and has a plugin for Visual Sourcecafe http://wiki.hudson-ci.org/display/HUDSON/Visual+SourceSafe+Plugin
While CI would probably be overkill for what you are trying to do, since Hudson is all GUI and easy to use, you would not spend a lot of time just trying to configure it.
Hudson also has plugins for copying stuff over to other systems, and so it would be easy to deploy your content to another system.
If you are worried about the process, get in touch with a hosted CI provider, such as MikeCI, a quick message on their support board will get you the answer. I don't see why triggering a "build" can't be replaced with copy and paste!
Related
Disclaimer: I am relatively unfamiliar with the flash build processes, so some/all of this may be misinformed nonsense. Please feel free to suggest alternative approaches.
We're currently developing a flex web app and our build situation is far from ideal. At present we're (as in individual developers) just building using FlashBuilder and deploying manually. The programmers are currently screaming bloody murder for two reasons, though:
The lack of CI is like going back to the stone age
We don't much care for FlashBuilder
(Note: We're only using FlashBuilder because it was the easiest way to set up a flex project in conjunction with Away3d and get it building / rendering correctly -- it's a stopgap solution).
As a predominately .NET development shop, we're used to doing continuous integration as well as continuous deployment. Ideally, we'd like to get something comparable to this for our flash projects without tying ourselves to a particular IDE.
Requirements:
The build process must be:
.. runnable via the commandline
.. runnable on both developer and CI build machines (and certainly not requiring an IDE!)
.. preferably as IDE-independent as possible (pragmatism will kick in though; if this causes a lot of friction we'll just pick one).
.. able to run on Windows (we develop using Windows)
We don't mind a touch of duplication or a few manual steps (e.g. tarting up the build scripts if we add a new project via an IDE, or generating one configuration from another if tools exist), but the less duplication / maintenance required the better.
I've read quite a few articles / blog posts and watched some short screencasts, but most of them are very thin on the ground on how the build system sits alongside IDEs. Most articles/screencasts have the same formula: How to create a "Hello World" build using a single file & text editors (no IDE).
I've not seen the topic of multiple libraries/projects etc. being broached, either.
After reading around the issue for a while, I'm considering investigating the following options:
Project Sprouts
Flexmojos
Maven Flex Plugin
buildr as3
Does anyone have any experience of the above solutions (or others I'm unaware of) and, if so, what do you make of them? Any help / pointers appreciated.
I recently started building with Gradle and the GradleFx plugin and I immediately fell in love with its power and ease of use.
Gradle is ANT + Maven + Ivy evolved and is primarily used from the command-line. You can:
write scripts in Groovy (a powerful Ruby-like language that runs in the Java Virtual Machine)
access all existing Maven and Ivy repositories as well as your own repos
use existing ANT tasks
integrate with CI (in Jenkins you just tick a checkbox to activate Gradle support)
although it has originally grown from the Java/Groovy community, it is in fact language agnostic. You add language-specific plugins for added functionality. GradleFx is such a plugin that provides you with additional ActionScript/Flex building tasks.
do easy multi-project builds. e.g. you can compile, unit test, package and deploy both your .NET service layer and your Flex client application with just one command.
use convention over configuration: if you stick to the conventions, your build scripts will be extremely terse
generate all kinds of reports: unit testing, checkstyles, codenarc, ...
generate Eclipse, IDEA or other IDE projects
all the things I haven't discovered yet
And best of all: it's very easy to learn. I had no knowledge of Maven before I started with Gradle and could get a multi-project build with some customizations working quite quickly.
Edit (comparison to Buildr AS3 and Maven)
I can compare this only to one of the projects you mentioned: Buildr AS3. It seems to start from a philosophy that is similar to Gradle's. But I've tried to use it about half a year ago and couldn't even get a simple 'Hello World' app to work. I e-mailed the developer for help: no response.
Compared to GradleFx: I had a small forum discussion with the developer (on a rather philosophical topic, since I didn't really need any help because it just worked right away). He answered within minutes.
As for Maven: (for what it's worth) I've only glanced at some configurations and they seem overly complicated when I compare them to a Gradle script.
There is one thing Maven does that you can't do with GradleFx (yet). It can pull the right Flex SDK from a Maven repo and build against that. With GradleFx you have to have your SDK's available locally.
I'm quite familiar with using maven as the main build tool and the flexmojos plugin from Sonatype. My experience has been a bit of a roller coaster with flexmojos. Maven is completely solid, it works all the time without issue, the only issue is the flexmojos plugin which has fluctuated a lot between versions. If you choose to go this route make sure to grab the source for flexmojos so you can see what your configuration options are actually doing to the command line parameters etc. For Flex 3.x flexmojos 3.x up to around 3.9 is good and works fine with regard to the goal for generating the .project eclipse files, believe there's also a mojo (a maven plugin) for generating intelliJ IDEA project files as well as others. If you're using Flex 4 you can compile with the latest flexmojos 4.0RC2 but it appears to me that the goal for generating flex/flashbuilder project properties is now gone (I'm not sure if this is because it's been replaced by another plugin altogether or what the deal is). However building with maven and flexmojos does fulfill all of your goals above (we also use it for building our service layer, so in a single mvn clean install we get a jar packed in a war packed in an ear with everything configured and a swf, that part is really nice). Also you can do continuous integration using bamboo (or simply write your own script that is triggered from a cron job or in windows as a batch file executed with a scheduled task if you don't have a *nix server around). Let me know if you'd like any more details or if I missed something major.
Shaun
I have been using Hudson, now Jenkins, with Ant for Flex automated builds and FlexUnit testing. Jenkins has some really useful plugins for integration with eclipse (and hence, FDT or FlashBuilder), Jira, SVN, Git etc., and it's free. Also, you can integrate the Ant build into Maven scripts, so I've found this to be a good and flexible solution for all purposes I've come across so far.
The Flex SDK comes with Ant tasks, and writing even elaborate Ant build scripts is quite easy - in fact, I'd been using Ant locally before, and I could reuse my existing scripts with only a few added extra compiler options for FlexUnit tasks.
However, it took a while to set up the system correctly for unit testing, because I'm running a headless server on Linux, and that implicates a rather complicated environment for ActionScript tests, because they run only in Flash Player. This, of course, is true for all CI scenarios using FlexUnit, regardless of which server you use.
Here's what I've learned:
FlexUnit needs a standalone debugger version of Flash Player installed, but Adobe only distributes binaries for the standard version on Linux. Therefore, compiling from source was necessary, and since my server system is stripped down to the bare necessities, it took some effort to install all the correct dependencies and get them to work.
The Flash Player needs hardware to run correctly: It uses graphics, therefore it needs a graphics card, and sounds, therefore it needs a sound card. On my headless server, this meant I had to install a VNC host to get it to run at all, and I had to eliminate any tests using sounds (those will now only run on local machines). If anyone ever comes across a working sound card emulation for openSuSE that I could use with the VNC client - you'll be my hero forever!
If you've set asynchronous timeouts in your unit tests, and/or you need to use setTimeout() to send delayed procedure calls, make sure the intervals aren't too short - I've had problems with tests that ran fine on any local machine, but broke the build on the CI server, because the Flash Player is considerably slower on the VNC client than on an actual graphics card.
I've also found this last issue to be a healthy lesson: Criteria for unit tests should not be based on assumptions about the system's performance, or at least be tolerant enough to succeed even on a slow machine.
I just started working in an environment that uses multiple programming languages, doesn't have source control and doesn't have automated deploys.
I am interested in recommending VSS for source control and using CruiseControl.net for automated deploys but I have only used CC.NET with ASP.NET applications. Is it possible to use CC.NET to deploy python, php, ASP.NET and ??? apps all from the same instance?
Yes you can.
CC .net is written in .Net but can handle any project. Your project langage does not matter, you can still use Batch, Powershell, Nant or MsBuild scripts. You may also use Cruise Control or Hudson, as you like.
As for the source control provider, I would prefer svn (or even git) but that's more a matter of habbits : from my point of view VSS is too linked to VS and I don't like the lock on check out by default behaviour.
VSS is unsafe for any source and is damn near useless outside visual studio. And cruise control is painful to learn and make work at best. Your heart is in the right place, but you probably want slightly different technical solutions. For SCM, you probably want either subversion (cf VisualSVN Server) or Mercurial (on IIS7). For continuious integration, I'd look at TeamCity first or Hudson second. Either of which is vastly superior to CCNet in terms of ease of use.
Is there any API or tool that can automate software updating? It should take care of checking for updates from a URL for a provided list of files and downloading and replacing the ones that need updating. It would also be nice if it contained an authentication module so that only authorized parties could access the updates. It should be language-agnostic - takes a list of files without extra knowledge except their versions and replaces them with newly downloaded copies if on the site there are newer versions.
I'm specifically interested in something for the Windows platform, that would run on Win Xp to Win 7.
This makes me think about apt-get ...
take a look here, as well: Is there an auto-update framework for C++/Win32/MFC (like Sparkle)?
I did see some articles a while back about embedding subversion into your application to manage version control.
Edit:
http://svnbook.red-bean.com/en/1.5/svn.developer.html
Subversion has a modular design: it's implemented as a collection of libraries written in C. Each library has a well-defined purpose and application programming interface (API), and that interface is available not only for Subversion itself to use, but for any software that wishes to embed or otherwise programmatically control Subversion. Additionally, Subversion's API is available not only to other C programs, but also to programs written in higher-level languages such as Python, Perl, Java, and Ruby."
Just saw UpdateNode launching a pretty cool update and messaging system. It seems to be cross platform and free for Open Source.
UPDATE, did some further analysis on that, posted at: https://stackoverflow.com/a/22528011/3257300
For windows, I'd use Google Update, also known as omaha.
Since you didn't tag this question as windows, I'd also mention a UpdateEngine for Mac.
And (best of all) apt, which is available for free on all Debian-based Linux and BSD distributions, like Ubuntu
There is open source project WIPT inspired by APT of Debian Linux.
Head over to Launchpad and use a PPA: it is a Debian/Ubuntu repository management platform. Of course this is not really platform independent but it is language wise :-)
You should take a look at ClickThrough, I don't know much about it but it sounds similar to what you're looking for. As for authorization, I would imagine this to be handled by your webserver based on the URL.
InstallShield has an offering. Never used it but researched it a few years back but we decided on a roll your own solution.
InstallShield Update Manager
InstallShield Update Service
You didn't state what platform you needed this for. The easiest way I can think of doing this is with subversion using rsync.
The concept is to write a post-commit hook for subversion. This script would update a "working folder" on the repository machine and then use rsync to update the differences to another machine.
Data protection and authentication would be set up using rsync over ssh.
If this is for windows, you could try doing the same with cygwin installs on the two machines.
Good luck.
If you use .NET, I'm a happy customer of AppLife Update
CRONw is a scheduled execution service for Windows. (Sorry, I can't link it, I'm apparently limited to 1 as a new user. It's hosted on Sourceforge.)
Powershell is a Windows scripting language (Microsoft-official) that allows you to do most system administration operations you could conceivably want to do. It is very easy to pick up even if you haven't worked with it before.
I would say your best bet is to write a simple update script in Powershell and, optionally, set it up as a crontask so you don't have to manually execute it.
IIRC, Powershell is an optional install on XP, and CRONw requires you be running a 32-bit system. You didn't say, so I'd guess you're doing 32-bit, but the alternative bears mentioning.
And in all this, I'm assuming that the URLs you're describing are designed for this purpose - if they're not and you don't own them, it will rapidly become more suffering than you're willing to bear. (Making a computer navigate a human-readable website usually does.)
We use Incredibuild here to compile our code in a distributed fashion. I was wondering if there are any open source (or free) alternatives to use on a home network?
Failing that, are there any other simple solutions with good integration with Visual Studio out there?
EDIT: I should say that I am quite happy to get my hands dirty and manually configure everything on each machine should that be required.
I can't look past TeamCity as a CI environment - among other features it allows multiple build agents to be linked together in one build grid.
Oh and it also has excellent integration with VS and SubVersion. And it's free to use, up to a maximum of 20 build configurations and 2 build agents.
Many programs include an auto-updater, where the program occasionally looks online for updates, and then downloads and applies any updates that are found. Program bugs are fixed, supporting files are modified, and things are (usually) made better.
Unfortunately no matter how hard I look, I can't find information on this process anywhere. It seems like the auto-updaters that have been implemented have either been proprietary or not considered important.
It seems fairly easy to implement the system that looks for updates on a network and downloads them if they are available. That part of the auto-updater will change significantly from implementation to implementation. The question is what are the different approaches of applying patches. Just downloading files and replacing old ones with new ones, running a migration script that was downloaded, monkey patching parts of the system, etc.? Concepts are preferred, but examples in Java, C, Python, Ruby, Lisp, etc. would be appreciated.
I think that "language agnostic" is going to be a limiting factor here. Applications come in so many shapes and sizes that there is no one-size-fits-all answer. I have implemented several auto-updaters in several languages, and no two were similar.
The most general philosophy is that the application checks with some home location (web address, web query, corporate network location, etc.) to either ask if it's version is current, or ask what the most current version is. If the answer calls for an update, that process will be different for each situation.
A popular alternative is to invite the home location to run a script when the application is initiated. The script can check the version, download updates if necessary, and ask for usage feedback, for example.
We can probably help better if you narrow the parameters.
UPDATE: The approach to "patching" also depends on the nature of the application, and there's a very wide diversity here. If you have a single executable file, for instance, then it's probably most practical to replace the executable. If your application has many files, you should look for ways to minimize the number of files replaced. If your application is highly customized or parameterized, you should strive to minimize the re-tailoring effort. If your application employs interpreted code (such as an Excel VBA application or MS Access MDB application), then you may be able to replace parts of the code. In a Java application you may only need to replace a JAR file, or even a subset of the JAR contents. You'll also need to have a way to recognize the current client version, and update it appropriately. I could go on and on, but I hope you see my point about diversity. This is one of those many times when the best answer usually starts with "Well, it depends ...!" That's why so many answers include "Please narrow the parameters."
Be sure to also consider the security implications of sucking down information about the update, as well as the update binaries themselves.
Do you trust the source of the download? You maybe phoning home to got your update, but what if there is a man in the middle who redirects to a malicious server. An HTTPS or similar secure connection will help, but double checking the bits that you eventually download by using a digital signature check is recommended.
First you need a file on your application home web site with the latest version.
The best way I think to have special SQL table for this task and populate it automatically after publishing new version / nightly build completion.
Your application creates new thread which requests built-in http link with version and compares in with current. In .NET use can use code like this:
Version GetLatestVersion() {
HttpWebRequestrequest = (HttpWebRequest)WebRequest.Create(new Uri(new Uri(http://example.net), "version.txt));
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (request.HaveResponse)
{
StreamReader stream = new StreamReader(response.GetResponseStream(), Encoding.Default);
return new Version(stream.ReadLine());
}
else
{
return null;
}
}
Version latest = GetLatestVersion();
Version current = new Version(Application.ProductVersion);
if (current < latest)
{
// you need an update
}
else
{
// you are up-to-date
}
In this example, version.php in only one plain string like 1.0.1.0.
Another tip I can give - how to download an update.
I like very much next idea: in the resources of your application there is a string of CLR-code which you compile on-the-fly (using CodeDom) to a temporary folder, main application calls it and goes to close. Updater reads arguments, settings or registry and downloads new modules. And calls main application which deletes all temporary files. Done!
(But everything here is about .NET)
The simplest solutions (used by many programs) is running the uninstaller for the previous version and the running the installer for the new one (optionally skipping questions which the user has already answered, like the EULA). The only catch is that the new version must be able to read the configuration options from the old version.
Also, on Windows you can't delete an executable file which is in use, so you probably will want to drop a small executable in Temp folder, which runs the whole process and then delete it at the end from the instance of the new version which got launched (or just register it to be deleted at the next reboot).
Because auto updating is a common scenario, most languages have at least one package available to support this. (Below I list some of the available packages)
One of the really nice idea's is the ClickOnce distribution for .NET, it's an installer which sandboxes your application and installs in the user context, so no administrator rights required. You can configure the ClickOnce in your publish to check for updates each application start.
Java has Java Web Start which offers the same kind of functionality for java applets.
Delphi has numerous articles about auto-updating, Torry has a list of WebUpdate components, for instance GoUpdater seems to have a very wide range of functionality.
They all use a website/network share to check for a new version and than retrieve either a patch or a complete install file and run it. So you should try to find a nice package for your application, to save you the hassle of developing and maintaining your own solution.
The simplest approach would be to have your program query a server (website) to see if there is an update. If there is an update you could display a message to the user that prompts them to download a newer version and provides a link.
An alternative and more complex solution would be to create a small windows service (or unix daemon) that checks periodically to see if there are updates, this service can download the update and launch the installer.
The general architecture is that you have a central server that you control that knows the latest version and where to get it. Then the programs query the server. I am not going to include sample code because it is highly defendant on the server and the format you choose. It is not terrible difficult though.
This is not so much a complete answer, but rather one example of auto-updating mechanism I implemented recently. The situation is a little different from the tradition Firefox-type of user application, since it was an internal tool used at work.
Basically, it's a little script that manages a queue of Subversion branches to be built and packaged in an installer. It reads a little file, where the names of the branches are written, takes the first one, re-writes it at the end of the file, and launches the build process, which involves calling a bunch of scripts. The configuration for each branch to build is written in a .INI file, stored in a Subversion repository along with the tool itself.
Because this tool runs on several computers, I wanted a way to update it automatically on all machines as soon as I made a change either to the tool itself, or to the configuration scripts.
The way I implemented it was simple: when I launch the tool, it becomes an "outer shell". This outer shell does 2 very simple things:
svn update on itself and on the configuration files
launch itself again, this time as the "inner shell", the one that actually handles one configuration (and then exits again).
This very simple update-myself-in-a-loop system has served us very well for a few months now. It's very elegant, because it is self-contained: the auto-updater is the program itself. Because "outer shell" (the auto-updater part) is so simple, it doesn't matter that it does not benefit from the updates as the "inner shell" (which gets executed from the updated source file every time).
One thing that hasn't really been mentioned is that you should seriously consider that the user running your program might not actually have sufficient privileges to upgrade it. This should be pretty common at least for business users, probably less so for home users.
I'm always working with a (self-imposed) limited account for security reasons and it always pisses me off that most auto-updaters simply assume that I'm running as admin and then after downloading just fail and offer no other way of performing the update other than actually closing the program and running it again in an administrative context. Most do not even cache the downloaded update and have to do it all over again.
It'd be much better if the auto-updater would simply prompt for admin credentials when needed and get on with it.
I'm going to assume answer for Windows.
This way seems to work well.
In the installer do:
1. Create a manual-start service that runs as LocalSystem that when started does the update then stops.
2. Change the service permissions so all users can start the service (if all users should be able to update w/o admin rights).
3. Change the main program to check for updates when started using a simple mechanism. If it detects an update, prompt if the user wants to apply it.
4. If user accepts the update, start the service.
If the architecture allows for it, create a way to monitor the update as it is running.
In a Java-Webstart setting you start a JNLP file which then triggers the download of the Jar files needed to run the application. Everytime webstart checks if there are newer versions of the Jars and would download them replacing the locally cached ones. With a tool named jardiff you will create only diffs towards the newer jars and distribute these via the server (e.g. only get an update).
Pros:
always up to date
Cons:
you need an application server (tomcat, JBoss) in order to distribute the files
you need an internet connection in order to get the application
Reading Carl Seleborgs answer gave me some ideas how a generic code-repository could be useful.
svn comes with a tool called svnsync, which sort of behaves like an svn export but keeps track of the actual revision your export is at.
Someone could utilize this system in order to only fetch the changed files from the users actual revision.
In actuality, you will have a repository with the binaries compiled, and running svnsync will only fetch the binaries that has been modified. It might also be able to merge local changes to text-based configuration files with new configuration-options.
The function of installing a patch to a program is basically one of the basic functions of an installer. Installer software is documented in numerous places but usually on a per-installer basis: There the Microsoft Installer (with Install Shield Extensions), Ruby gems, Java .jar files, the various Linux package manager systems (RPM, Apt-get)and others.
These are all complex systems which solve the problem of patching program in general but for slightly different systems. To decide what is best for you, consider which of these system your application most resembles. Rolling your own is fine but looking at these systems is a place to start.
You can write an internal module of your application to do updates. You can write an external mini application to do updates.
Also look at .NET on-the-fly compilation technology, it makes possible to create such mini application on-the-fly on demand. For example, http://fly.sf.net/
You can use my solution (part of the Target Eye project).
http://www.codeproject.com/Articles/310530/Target-Eye-Revealed-part-Target-Eyes-Unique-Auto
If your software is open sourced, and target Linux or developers. It is interesting to install your software as a git repo. And having it pull the stable branch occasionally or everytime when it is launched.
This is particular easy when your application is managed via npm, sbt, mavan, stack, elm-package or alike.
After hours of searching some working solution for this problem I've finally implemented auto update mechanism for python script that works on Linux and Windows.
In short - the script before running actual work checks for update on S3 and if it's available downloads it, unzips, creates or updates the symlink (or junction on Windows) and re-runs the script with already the new version with original arguments.
The full source code and the explanation can be found here.
If you are searching for an cross-platform software update solution, take a look at www.updatenode.com
Some highlights:
free for Open Source projects
cross-platform & Open Source update client tool
localized already for the most important languages
easy to integrate and easy to handle
cloud based management platform to define and manage updates
provides additionally support for displaying messages (inform about new events, products, etc.)
web interface is open (you can create your own client using the service)
many usage statistics, as used operating systems, geo location, version usage, etc.
Android API for mobile App updates
Just try it.
BTW, I am part of the dev team for the open source client. :)