I plan to use the Azkaban https://azkaban.github.io/ for running batch jobs. According to CI ideas we have few environments like dev, test, stage, production and of course job should have different configuration for each environment.
According to Azkaban documentation http://azkaban.github.io/azkaban/docs/latest/#job-configuration Azkaban allows for replacing of parameters whenever a ${parameter} is found. And solution looks like:
system.properties
myFlow/
dev.properties
....
prod.properties
foo.job
#system.properties
env=dev
#dev.properties
dev.database=localhost:2181
#prod.properties
prod.database=aws:port
#foo.job
some command --db ${${env}.database}
And later on each environment we can override the env variable. From my point of view this solution looks strange. Can I just say to Azkaban which property file should be used on environment?
What is the best approach to do it?
An other approach can be found here https://github.com/azkaban/azkaban/issues/1545#issuecomment-339750417
Related
I'm trying to make my tasks run tests in a certain directory. I was looking at sourceSets, however I inferred that they are useful if you are running outside the test/groovy folder. All of my tests are within the test/groovy folder.
I've got a set of Geb tests as well as a set of service tests. I would like to run them both together and independently. Essentially my tree structure would look like this, being able to run all tests.
Test
--gebTest
----firefoxTest
----chromeTest
----ieTest
--servicesTest
----service1Test
----service2Test
----service3Test
----etc.
My file structure is as follows:
project
-src
--test
---groovy
----com
-----acme
------functional <---where my geb tests sit
------services <---umbrella for services
-------service1 <---each unique service
-------service2
-------service3
-------etc
Can anyone lend me a hand. For the life of me I don't know how Gradle picks what tests to execute.
Thank you in advanced.
SourceSets are indeed a solution to your problem, but I notice you only differentiate your tests by their package names. I'm not sure but that may prove problematic with source sets.
Personally I would prefer a directory structure like this anyway
src
-test
--groovy
---functional
----com etc
---services
----com etc
However, if you are attached to your current structure then take a look at Gradle's test filtering support, which will allow you to filter by package name.
http://www.gradle.org/docs/current/userguide/java_plugin.html#sec:java_test
I am porting over an MSBuild script from CCnet to run in Jenkins. The MSBuild project is used to create a deployment package. I would rather have Jenkins drive this process itself but that's a longer term aim.
The problem i am having is (as in the title) when we try and use the HgPull target, from the MSBuild mecurial task (http://msbuildhg.codeplex.com/) we get the error message
HgProcessException: The command hg.exe is not available [Path to project]
I have seen on the project web site that someone solved it by adding the LibraryLocation proeprty to the target but that seems to have made no difference. My target currently looks like this:
Target Name="UpdateSources">
<HgPull
LibraryLocation="C:\Program Files\TortoiseHg\hg.exe"
Force="true"
Update="true"
LocalPath="$(SourcePath)"
/>
<HgUpdate
LibraryLocation="C:\Program Files\TortoiseHg\hg.exe"
Clean="true"
LocalPath="$(SourcePath)"
/>
I'm rather at a loss. Please let me know if you need any more information added to this post to solve this issue. I'm really quite new to MSBuild so really not sure where to start investigating this.
EDIT:
One thing i forgot to mention was that i have tried running the MSbuild command in a console window on the build server and still get the same result. This is really odd given it works fine in CCNet, what magic is CCNet doing to make this command work?
This is now resolved, unfortunately i'm not sure what changes i made to correct these. I believe it may have been down to path separators and whether they where trailing or not in another part of the config file. It does so annoy me with the lack of resilience/consistency between applications where you need to specify paths with/without trailing slashes.
Just a thought, but try adding 'C:\Program Files\TortoiseHg' to your system path. Maybe CCNet has it specified somewhere that Jenkins doesn't have access to.
Also, just for sanity's sake, verify that hg.exe actually exists at that location.
I need to distribute some sort of static configuration through my application. What is the best practice to do that?
I see three options:
Call application:get_env directly whenever a module requires to get configuration value.
Plus: simpler than other options.
Minus: how to test such modules without bringing the whole application thing up?
Minus: how to start certain module with different configuration (if required)?
Pass the configuration (retrieved from application:get_env), to application modules during start-up.
Plus: modules are easier to test, you can start them with different configuration.
Minus: lot of boilerplate code. Changing the configuration format requires fixing several places.
Hold the configuration inside separate configuration process.
Plus: more-or-less type-safe approch. Easier to track where certain parameter is used and change those places.
Minus: need to bring up configuration process before running the modules.
Minus: how to start certain module with different configuration (if required)?
Another approach is to transform your configuration data into an Erlang source module that makes the configuration data available through exports. Then you can change the configuration at any time in a running system by simply loading a new version of the configuration module.
For static configuration in my own projects, I like option (1). I'll show you the steps I take to access a configuration parameter called max_widgets in an application called factory.
First, we'll create a module called factory_env which contains the following:
-define(APPLICATION, factory).
get_env(Key, Default) ->
case application:get_env(?APPLICATION, Key) of
{ok, Value} -> Value;
undefined -> Default
end.
set_env(Key, Value) ->
application:set_env(?APPLICATION, Key, Value).
Next, in a module that needs to read max_widgets we'll define a macro like the following:
-define(MAX_WIDGETS, factory_env:get_env(max_widgets, 1000)).
There are a few nice things about this approach:
Because we used application:set_env/3 and application:get_env/2, we don't actually need to start the factory application in order to have our tests pass.
max_widgets gets a default value, so our code will still work even if the parameter isn't defined.
A second module could use a different default value for max_widgets.
Finally, when we are ready to deploy, we'll put a sys.config file in our priv directory and load it with -config priv/sys.config during startup. This allows us to change configuration parameters on a per-node basis if desired. This cleanly separates configuration from code - e.g. we don't need to make another commit in order to change max_widgets to 500.
You could use a process (a gen_server maybe?) to store your configuration parameters in its state. It should expose a get/set interface. If a value hasn't been explicitly set, it should retrieve a default value.
-export([get/1, set/2]).
...
get(Param) ->
gen_server:call(?MODULE, {get, Param}).
...
handle_call({get, Param}, _From, State) ->
case lookup(Param, State#state.params) of
undefined ->
application:get_env(...);
Value ->
{ok, Value}
end.
...
You could then easily mockup this module in your tests. It will also be easy to update the process with some new configuration at run-time.
You could use pattern matching and tuples to associate different configuration parameters to different modules:
set({ModuleName, ParamName}, Value) ->
...
get({ModuleName, ParamName}) ->
...
Put the process under a supervision tree, so it's started before all the other processes which are going to need the configuration.
Oh, I'm glad nobody suggested parametrized modules so far :)
I'd do option 1 for static configuration. You can always test by setting options via application:set_env/3,4. The reason you want to do this is that your tests of the application will need to run the whole application anyway at some time. And the ability to set test-specific configuration at that point is really neat.
The application controller runs by default, so it is not a problem that you need to go the application-way (you need to do that anyway too!)
Finally, if a process needs specific configuration, say so in the configuration data! You can store any Erlang-term, in particular, you can store a term which makes you able to override configuration parameters for a specific node.
For dynamic configuration, you are probably better off by using a gen_server or using the newest gproc features that lets you store such dynamic configuration.
I've also seen people use a .hrl (erlang header file) where all the configuration is defined and include it at the start of any file that needs configuration.
It makes for very concise configuration lookups, and you get configuration of arbitrary complexity.
I believe you can also reload configuration at runtime by performing hot code reloading of the module. The disadvantage is that if you use configuration in several modules and reload only one of them, only that one module will get its configuration updated.
However, I haven't actually checked if it works like that, and I couldn't find definitive documentation on how .hrl and hot code reloading interact, so make sure to double-check this before you actually use it.
The main problem is: How do i incorporate an appSettings.Config file with a particular build(dev, stage, live)? My appSettings.Config changes the conx strings for data sources based on which server the package is being deployed to. I am able to go through Package configurations and add my appSettings.Config, however, I can only specifically add one file dev, stage, or live. What i need to do is be able to build the solution and based on teh build type incorporate the dev/stage/live appsettings. How could I do this?
You could include all of the configuration files in the install and then just point to the correct one through an environment variable. I know you're wanting to switch the configuration file based on the solution build configuration, but you'll be looking at a complex solution when a simpler alternative exists.
Its quite straight-forward to add registry information during the package install that will set the machine's environment variable under the key:
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment\MyVariable
...to the path of the .dtsConfig for the current environment.
Is there any way to create new Hudson job by one more Hudson job based one previous Jobs?
For example if I need to create new bunch of jobs one by one, Automatically create 4 jobs with similar configuration with different parameter
Basically steps like this
create SVN branch I can call svn cp command and make it parametrized using script
Create some build based on new svnbranch name
Later tag it
Or other word, I need to clone the previous job and give the new branch name where ever $ Branch comes in new job.
Thanks
You can try the Hudson Remote API for this kind of task (setting up an Hudson project).
See this tutorial for instance, and remember you can display the help quite easily:
java -jar hudson-cli.jar -s http://your_Hudson_server/ help
So, to copy a job:
java -jar hudson-cli.jar -s http://your_Hudson_server/ copy-job myjob copy-myjob
You could use groovy system script like this :
def jenkins = hudson.model.Hudson.instance
def template = jenkins.getItem("MyTemplate")
def job = jenkins.copy(template,"MyNewJob")
job.scm = new hudson.scm.SubversionSCM("http://base/branches/mybranche")
job.save()
Kind of already covered in the other answers, but for an easy way to copy the config.xml over:
curl --user USER:PASS -H "Content-Type: text/xml" -s
--data-binary "#config.xml" "http://hudsonserver:8080/createItem?name=newjobname"
There seems to be a plugin for jenkins.
https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
I have not tested the plug-in yet. But if the plugin works, it should alleviate some of human errors from straight copying a job and modifying variables/values.
def jenkins = hudson.model.Hudson.instance
def template = jenkins.getItem("MyTemplate")
def job = jenkins.copy(template,"MyNewJob")
job.save()
I used this now I have to change the parameter values of MyNewJob using Groovy how will I do that?
ex I have a parameter called "Build_BranchName" and the default is //perforce/mybranch
I have to change it to
//perforce/mynewbranch
You have the option that VonC just gave you (which is probably the safest way but you can also go a different rout by just creating a new directory in {Hudson_Home}\jobs (the directory name will be the job name) and copy a modified config.xml in there. The modification will basically just be the SVN URL. You should check out the xml from the job that you are copying. You need to find out how you change the xml file via script, but this is a secondary problem.
Unfortunately, you have to either restart Hudson, or force a reload of the configuration. (visit the page http://:/reload to reload the config).
In case you're willing to use GIT (like I do, mirroring the main SVN repo, onto the Hudson/Jenkins server, and it works great)....
..you could try Stephen Haberman's post-receive-hudson:
This hook creates new jobs for each
branch in the Hudson continuous
integration tool. Besides creating the
job if needed, the user who pushed is
added to the job's email list if they
were not already there.
In any case, that script can give you new hints on how to remote control Jenkins(Hudson).