Hello is it possible to work with two jenkins?
Jenkins slave do dirty job, and save image, then he upload to jenkins master. Or jenkins master download from slave. What would be the easiest way to do sthing like this?
ps. i dont know how tag this topic/help
edit1: I mean about working with two computers in the same network. I tried some with manage jenkins > new nodes but with no success. I will report if i will success.
edit2
Okay i set up slave to work by JWS, and tied to project. Then build project to refresh configuration. Now i have problem with port listening by slave but i think i will have to ask admin to unlock. I gues unlock number which value is above 60.000.
I still don't know how use slave to do something and upload to master workspace /or/ use slave to save data in his temp and force master to grab this data from slave.Ant suggestion will be helpful.
Michal,
You don't need two instances of Jenkins to get information from the slave to the master. There are plugins for that.
===========================================================
There are two ways to add a Windows slave:
by service
by JNLP (this should work for sure)
Once you have a slave connected, install the following plugin:
https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin
===========================================================
Once this is done, set up two jobs. The first job runs on your slave (restrict where this job is run), and make it run your Windows XP program and store the image. Archive this file as an artifact.
On the second job, use the plugin above to copy the artifact from the first job, and use the data in the file as you need it.
For more information, look at the links below:
Restricting jobs to a slave: https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds
Using the copy plugin: https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin
Archiving an artifact: Archive the artifacts in hudson/jenkins (the first picture in the question should be useful)
===========================================================
Related
Background: I used to develop using MAMP and over the months/years I've accumulated a large mysql database (a few gigs) that I use for development for my different projects. I finally got around to setting up a VM using Vagrant and I've gotten everything set up and working nicely except my database. I'm running a CentOS 6.5 guest box on an OSX host.
My problem: I need my database to be completely persistent so I can vagrant up/destroy as many boxes as I need to, but the mysql persists.
My solution #1: I initially mounted a synced folder using vboxsf. This works pretty well and seems to be my best option so far, but performance is pretty bad. Query-intensive pages on my dev sites take 1-3 seconds to load whereas they might normally take under a second to load.
My solution #2: I then tried mounting a synced folder using nfs because the performance should be much better. The issue here is that mysql complains b/c, given the nature of nfs, it can't chown the data directory to the mysql:mysql user. I get the following errors when trying to start up the mysqld service:
chown: changing ownership of '/www/mysql': Operation not permitted
chmod: changing permissions of '/www/mysql': Permission denied
Sooo, my question is: are there any better ways to accomplish what I need? I feel like NFS would be the best solution, but I don't know how to get around the whole ownership/permission issues automatically with Vagrant. Any help would be appreciated.
I had the same issue or requirement for my local dev on Mac. And I found a solution for a MySQL-only Vagrant box with external data linked as folder_sync. But it'll run on Win too I guess.
Here is the Vagrant box config: https://github.com/ronnyhartenstein/vagrant-mysql-shared-folder
And if you understand German, here is my blog article with some background infos and tests (and fails of course): http://blog.rh-flow.de/2014/11/11/es-hat-sich-ausgemampft-vagrant-ist/
First of all, let me start with saying this is not best practice. You may know yourself that this can lead to problems if e.g. your PC goes blank or you want to give one project to another person for development. Of course, especially as a one-person-endevour, there are more important things than having test data importers and stuff :) So let's look for solutions.
NFS Permissions
To get NFS permissions right, your users need to have the same UID and GUID on host and guest. It's pretty tricky to setup and you should not change it from the guest. Maybe you can change it on the host to make it writeable to mysql and make UID and GUID the same. Of course, the moment the host changes this won't work anymore.
rsync shared folder
Rsync might not be the fastest in terms of syncing, but if you create on rsync shared folder where only MySQL is writing and which syncs back to some folder on your host this might be a solution. The "real" projects could still live inside a virtualbox share or nfs and you don't need to bother with correct permissions.
There might be some other solutions as well:
Create a backup/restore strategy
One way to go would be to backup MySQL inside your vagrant box at various points, e.g. every day. You could also run the backup when the box is shut down, thus creating a backup right before you destroy the box. Placing this backup at a shared folder, you'd have up-to-date data in case you destroy a box. Performance should be pretty good as the data MySQL is using wouldn't be on a shared folder.
Run MySQL on host or other vagrant box
It's of course possible to connect from within your vagrant box to your host or another vagrant box which runs MySQL. Your host or this box could be long-lived and could serve as a central "MySQL Server" for all your projects.
Have a MySQL slave running on the same machine which writes to shared folder
I believe with MySQL a master/slave combination is possible. Running both on one machine with the master (which you use in your projects) living inside your vm and not writing anything to a shared folder and a slave which writes to your shared folder and is a mirror of your master. This would mean that you have high performance and a few secs of delay between writing something and having it written to your shared folder. Of course, keeping this setup running and making sure it works all the time can be tricky.
You can use bindfs for changing the user/group of a share. I'm actually using a plugin called vagrant-bindfs which let's you remount a share with different ownerships. It works, but i haven't tried it with mysql to see how it performs.
Relevant lines on my Vagrantfile:
unless Vagrant.has_plugin?("vagrant-bindfs")
raise 'vagrant-bindfs is not installed! Please install with vagrant plugin install vagrant-bindfs'
end
config.vm.synced_folder "../", "/temp-nfs-mounts/sites-unbinded", type: :nfs
config.bindfs.bind_folder "/temp-nfs-mounts/sites-unbinded", "/sites", :force_user => "vagrant", :force_group => "vagrant", :create_as_user => true
I just created a free php gear...
Is the instance automatically configured to roll logs and delete old logs (to make sure we dont go over disk quota?)
Can you pls tell me how often logs are rolled and when old ones get deleted?
thanks
At this moment (April 2014), Apache RotateLogs does not seem to be used anymore. This commit seems to have changed to use logshifter, which reportedly seems to default to rotating every 10MB with a max of 10 log files.
So, to answer your question, it seems like things are automatically configured to roll logs and delete old logs to prevent us from going over disk quota.
BTW, the new logshifter setup combines the access_log and error_log into one log file instead of keeping them separate.
At this moment (Feb 2014), all OpenShift Apache-based cartridges use Apache RotateLogs program to rotate logs every midnight:
/usr/sbin/rotatelogs <gear-dir>/php/logs/access_log-%Y%m%d-%H%M%S-%Z 86400
The log files are not deleted automatically. However, you can delete them manually using rhc app-tidy <app> command. (Read more about rhc tools.)
If concerned about logs eating all your gear capacity, you might consider using monit community cartridge to trigger automatic email notifications when the app hits 80% of gear storage quota, or to tidy your app automatically. If you already created your app, you can add the monit cartridge with the following commands:
rhc env set MONIT_ALERT_EMAIL=my#email.com -a YOUR_APP
rhc cartridge-add http://goo.gl/jiIB2C -a YOUR_APP
And last but not least, feel free to open a new bug report or new feature request for OpenShift.
I have MASTER and SLAVE configured (ssh-slave-plugin).
I would like to display output of the slave executed script under job on master,
as so far I get
Building remotely on SubAgent
Triggering SubAgent
Triggering a new build of XXXX #126
Finished: SUCCESS
and that is all. So the whole execution is hidden.
Is there any way to do that?
I am using the same master-slave configuration (ssh) on Hudson and all the logs are visible on the Hudson Interface.
There might be a couple of things that you can check:
What tool are you using to build (eg, ANT, MAVEN...). Check the execution if the logs are being produced at all
Check the Console Output [raw]
Manage Hudson > Manage Nodes > Select the Slave > Configure
Make Sure that "Remote FS root" is mentioned.
Check Launch Method. I am Connecting to my slaves via jnlp (I believe this could be the key)
Cheers!!
Go to Nodes, choose the node (or hover on the name) and select Build History. The logs of the job that ran on the slave node will be there.
I have a couple of jobs that use a shared resource (database), which sometimes can cause builds to fail in the (rare) event that the jobs happen to get triggered simultaneously.
Given jobs A through E, for example, is there any way to specify that A and C should never be run concurrently?
Other than the aforementioned resource, the builds are independent of each other (not e.g. in a upstream/downstream relation).
A "brute-force" way would be limiting number of executors to one, but that obviously is less than ideal if most jobs could well be executed concurrently and there's no lack of computing resources on the build server.
There are currently 2 ways of doing this:
Use the Throttle Concurrent Builds plugin.
Set up those jobs to run on a slave having only 1 executor.
The Locks and Latches plugin here should help.
This question is probably a dupe of How do I ensure that only one of a certain category of job runs at once in Hudson?
That's an old question, but the topic can still be relevant, especially when running application tests on Jenkins.
The Lockable Resources Plugin allows you to define lockable resources that can be used by builds. If your build requires an resource, it takes the lock. If a second build requires the same resource (which then is already locked), it will be queued for the resource to be free.
Although the docs use computers or printers as examples for lockable resources, the database example from above should work as well.
In opposite to the Locks and Latches Plugin mentioned in answers from 2012, this package seems to be currently maintained (currently ~2016).
Have a look at the External Resource Dispatcher Jenkins plugin, which was first published in November 2012. This (relatively) new plugin seems to exactly cover this use case.
N.B. you don't need physical or virtual hardware for a slave/node, you can set up "slaves" that run on the master server.
Manage Jenkins > Manage Nodes > New node
and make a "dumb slaves" each with its own root directory.
Create a few slaves, execute them when the server boots, and then you have essentially created pools of executors.
You might have, say...
db - only one executor in your case.
compile - limit according to hardware or # of CPUs.
scripts - have many executors for all those little jobs that Jenkins is good at doing.
Old question, and whether this will work for your application I can't be sure as you didn't mention details of your application. However, I wanted to add the way that I handled this in our Rails application test suite.
Our application's database configuration (database.yml) isn't in the source repository. Instead, it lives in /var/lib/configs/uniquing_database.yml on the VM which runs our Jenkins instance.
One of the steps of our build process involves copying this config file to the project workspace:
cp /var/lib/jenkins/configs/myapp_unique_database.yml config/database.yml
and that config takes workspace and build number information exposed to the environment by Jenkins into account in order to create a uniquely named database for that job and it's specific execution:
test:
adapter: postgresql
encoding: unicode
host: 127.0.0.1
port: 5432
database: myapp_test<%= ENV['JOB_NAME'].split('/').last %><%= ENV['BUILD_NUMBER'] %>
The rest of our build proceeds without any knowledge or care that it's running in a distinct database. Finally, at the end of our build, we make sure to drop that database so we don't have a bunch of test databases polluting the file system:
RAILS_ENV=test bundle exec rake db:drop
I have a Jenkins (Hudson) server setup that runs tests on a variety of slave machines. What I want to do is reconfigure the slave (using remote APIs), reboot the slave so that he changes take effect, then continue with the rest of the test. There are two hurdles that I've encountered so far:
Once a Jenkins job begins to run on the slave, the slave cannot go down or break the network connection to the server otherwise Jenkins immediately fails the test. Normally, I would say this is completely desirable behavior. But in this case, I would like for Jenkins to accept the disruption until the slave comes back online and Jenkins can reconnect to it - or the slave reconnects to Jenkins.
In a job that has been attached to the slave, I need to run some build tasks on the Jenkins master - not on the slave.
Is this possible? So far, I haven't found a way to do this using Jenkins or any of its plugins.
EDIT - Further Explanation
I really, really like the Jenkins slave architecture. Combined with the plugins already available, it makes it very easy to get jobs to a slave, run, and the results pulled back. And the ability to pick any matching slave allows for automatic job/test distribution.
In our situation, we use virtualized (VMware) slave machines. It was easy enough to write a script that would cause Jenkins to use VMware PowerCLI to start the VM up when it needed to run on a slave, then ship the job to it and pull the results back. All good.
EXCEPT Part of the setup of each test is to slightly reconfigure the virtual machine in some fashion. Disable UAC, logon as a different user, have a different driver installed, etc - each of these changes requires that the test VM/slave be rebooted before the changes take affect. Although I can write slave on-demand scripts (Launch Method=Launch slave via execution of command on the master) that handle this reconfig and restart, it has to be done BEFORE the job is run. That's where the problem occurs - I cannot configure the slave that early because the type of configuration changes are dependent on the job being run, which occurs only after the slave is started.
Possible Solutions
1) Use multiple slave instances on a single VM. This wouldn't work - several of the configurations are mutually exclusive, but Jenkins doesn't know that. So it would try to start one slave configuration for one job, another slave for a different job - and both slaves would be on the same VM. Locks on the jobs don't prevent this since slave starting isn't part of the job.
2) (Optimal) A build step that allows a job to know that it's slave connection MIGHT be disrupted. The build step may have to include some options so that Jenkins knows how to reconnect the slave (will the slave reconnect automatically, will Jenkins have to run a script, will simple SSH suffice). The build step would handle the disconnect of the slave, ignore the usually job-failing disconnect, then perform the reconnect. Once the slave is back up and running, the next build step can occur. Perhaps a timeout to fail the job if the slave isn't reconnectable in a certain amount of time.
** Current Solution ** - less than optimal
Right now, I can't use the slave function of Jenkins. Instead, I use a series of build steps - run on the master - that use Windows and PowerShell scripts to power on the VM, make the configurations, and restart it. The VM has a SSH server running on it and I use that to upload test files to the test VM, then remote execute them. Then download the results back to Jenkins for handling by the job. This solution is functional - but a lot more work than the typical Jenkins slave approach. Also, the scripts are targeted towards a single VM; I can't easily use a pool of slaves.
Not sure if this will work for you, but you might try making the Jenkins agent node programmatically tell the master node that it's offline.
I had a situation where I needed to make a Jenkins job that performs these steps (all while running on the master node):
revert the Jenkins agent node VM to a powered-off snapshot
tell the master that the agent node is disconnected (since the master does not seem to automatically notice the agent is down, whenever I revert or hard power off my VMs)
power the agent node VM back on
as a "Post-build action", launch a separate job restricted to run on the agent node VM
I perform the agent disconnect step with a curl POST request, but there might be a cleaner way to do it:
curl -d "offlineMessage=&json=%7B%22offlineMessage%22%3A+%22%22%7D&Submit=Yes" http://JENKINS_HOST/computer/THE_NODE_TO_DISCONNECT/doDisconnect
Then when I boot the agent node, the agent launches and automatically connects, and the master notices the agent is back online (and will then send it jobs).
I was also able to toggle a node's availability on and off with this command (using 'toggleOffline' instead of 'doDisconnect'):
curl -d "offlineMessage=back_in_a_moment&json=%7B%22offlineMessage%22%3A+%22back_in_a_moment%22%7D&Submit=Mark+this+node+temporarily+offline" http://JENKINS_HOST/computer/NODE_TO_DISCONNECT/toggleOffline
(Running the same command again puts the node status back to normal.)
The above may not apply to you since it sounds like you want to do everything from one jenkins job running on the agent node. And I'm not sure what happens if an agent node disconnects or marks itself offline in the middle of running a job. :)
Still, you might poke around in this Remote Access API doc a bit to see what else is possible with this kind of approach.
Very easy. You create a Master job that runs on the Master, from the master job you call the client job as a build step (it's a new kind of build step and I love it). You need to check that the master job should wait for the client job to finish. Then you can run your script to reconfigure your client and run the second test on the client.
An even better strategy is to have two nodes running on your slave machines. You need to configure two nodes in Jenkins. I used that strategy successfully with a unix slave. The reason was that I needed different environment variables to be set up and I didn't wanted to push that into the jobs. I used ssh clients, so I don't know if it is possible with different client types. Than you might be able to run both tests at the same time or you chain the jobs or use the master strategy mentioned above.