I'm using slurm with multiple partitions such as dev, dev1, dev2..
Therefore when I need to submit huge job array,
I'm always using "-p dev,dev1,dev2" option with python submitit library.
https://github.com/facebookincubator/submitit/blob/main/docs/examples.md
However, the slurm does not distribute the job array to all partitions. job array is pending in dev partition only. other partitions are idle.
In addition, even though there are no jobs in other partitions, new jobs submitted are not running but pending in other partitions.
srun -p dev1 echo "hello world"
srun: job 5037027 queued and waiting for resources.
# slurm does not distribute the job to empty partition.
Is it related to fair share policy? or queue configuration?
Related
New to openshift so I am bit confused if we can run user pods on master or infra nodes. We have 2 worker nodes and one master and infra nodes each making 4 nodes. The reason for change is to share the loads between all 4 nodes rather than 2 compute nodes.
By reading some documents it seems possible to assign 2 roles to one node but is there any security risk or is it not best practice?
Running on openshift version v3.11.0+d699176-406
if we can run user pods on master or infra nodes
Yes, you absolutely can, the easiest way is to configure that at installation time, refer to https://docs.openshift.com/container-platform/3.11/install/example_inventories.html#multi-masters-using-native-ha-ai for example
is there any security risk or is it not best practice
Running single master node or a single infra node is already a risk to high availability of your cluster. If master fails - your cluster is basically headless, if infra node fails - you lose your internal registry and routers - thus losing an external access or an ability to create new images for your imagestreams. This also applies to host OS upgrades, you will have to reboot master and infra nodes some day, are you okay with having a guaranteed downtime during patching? What if something goes wrong during update?
Regarding running user workload on master and infra nodes - if you are not running privileged SCCs (which can allow to run privileged pods or using any uids on system etc) - you are somewhat safe from container breach, assuming there are no known bugs in container engine you are using. However you should pay close attention to your resource consumption and avoid running any workloads without CPU and memory limits, because overloading master node may result in cluster degradation. You should also monitor disk usage, since running user pods results in more images loaded to your docker storage.
So basically it boils down to this:
Better have multiple master (ideally 3) and couple of infra nodes than a single point of failure for both of these roles. Having separate master and worker nodes is of course better than hosting them together, but multimaster setup even with user workloads should be more resilient if you are watching resource usage carefully.
I have batch jobs that I want to run on Kubernetes. The way I understand Jobs:
If I choose restartPolicy: Never it means that if the Job fails, it will destroy the Pod and reschedule onto (potentially) another node. If restartPolicy: OnFailure, it will restart the container in the existing Pod. I'd consider a certain number of failures unrecoverable. Is there a way I can prevent it from rescheduling or restarting after a certain period of time and cleanup the unrecoverable Jobs?
My current thought for a workaround to this is to have some watchdog process that looks at retryTimes and cleans up Jobs after a specified number of retries.
Summary of slack discussion:
No, there is no retry limit. However, you can set a deadline on the job as of v1.2 with activeDeadlineSeconds. The system should back off restarts and then terminate the job when it hits the deadline.
FYI, this has now been added as .spec.backoffLimit.
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
I know Mysql slaves read Master's bin-log sequentially with single thread and dump the data as relay log locally. My question is that will slave use multiple threads to read the relay logs for sync up the operations(Insert,Update,Delete)?
If it's not multiple threads, I guess the following scenario Mysql slave may not be able to catch up the Master forever.
Let's say Master has 100 tables and there is no any relationship between them. So It could be 100 writing requests to the 100 table simultaneous. If we suppose in 1 second the 100 tables finish writing in Master, then Slave may need 100 seconds to finish the sync up if it's single thread operation. Right please? Thanks!
Unless you are using MySQL Server 5.6 or later, then yes, the replication events are executed sequentially by a single thread (the "SQL" thread) after having been received from the master by the "IO" thread and written to the relay log.
However, MySQL 5.6 introduced slave_parallel_workers, which, if configured (along with other prerequisites for multithreaded replication) allows multiple (theoretically up to 1024) parallel replication worker threads, controlled and coordinated by the slave SQL thread, so that queries among independent databases can be replicated in parallel.
It uses one reading thread and one updating thread.
Your scenario doesn't make sense. The 100 updates will take as long at the master as at the slave, maybe longer, unless there is a remarkable discrepancy between the CPUs and disks of the master and slave hosts.
I have a Jenkins (Hudson) server setup that runs tests on a variety of slave machines. What I want to do is reconfigure the slave (using remote APIs), reboot the slave so that he changes take effect, then continue with the rest of the test. There are two hurdles that I've encountered so far:
Once a Jenkins job begins to run on the slave, the slave cannot go down or break the network connection to the server otherwise Jenkins immediately fails the test. Normally, I would say this is completely desirable behavior. But in this case, I would like for Jenkins to accept the disruption until the slave comes back online and Jenkins can reconnect to it - or the slave reconnects to Jenkins.
In a job that has been attached to the slave, I need to run some build tasks on the Jenkins master - not on the slave.
Is this possible? So far, I haven't found a way to do this using Jenkins or any of its plugins.
EDIT - Further Explanation
I really, really like the Jenkins slave architecture. Combined with the plugins already available, it makes it very easy to get jobs to a slave, run, and the results pulled back. And the ability to pick any matching slave allows for automatic job/test distribution.
In our situation, we use virtualized (VMware) slave machines. It was easy enough to write a script that would cause Jenkins to use VMware PowerCLI to start the VM up when it needed to run on a slave, then ship the job to it and pull the results back. All good.
EXCEPT Part of the setup of each test is to slightly reconfigure the virtual machine in some fashion. Disable UAC, logon as a different user, have a different driver installed, etc - each of these changes requires that the test VM/slave be rebooted before the changes take affect. Although I can write slave on-demand scripts (Launch Method=Launch slave via execution of command on the master) that handle this reconfig and restart, it has to be done BEFORE the job is run. That's where the problem occurs - I cannot configure the slave that early because the type of configuration changes are dependent on the job being run, which occurs only after the slave is started.
Possible Solutions
1) Use multiple slave instances on a single VM. This wouldn't work - several of the configurations are mutually exclusive, but Jenkins doesn't know that. So it would try to start one slave configuration for one job, another slave for a different job - and both slaves would be on the same VM. Locks on the jobs don't prevent this since slave starting isn't part of the job.
2) (Optimal) A build step that allows a job to know that it's slave connection MIGHT be disrupted. The build step may have to include some options so that Jenkins knows how to reconnect the slave (will the slave reconnect automatically, will Jenkins have to run a script, will simple SSH suffice). The build step would handle the disconnect of the slave, ignore the usually job-failing disconnect, then perform the reconnect. Once the slave is back up and running, the next build step can occur. Perhaps a timeout to fail the job if the slave isn't reconnectable in a certain amount of time.
** Current Solution ** - less than optimal
Right now, I can't use the slave function of Jenkins. Instead, I use a series of build steps - run on the master - that use Windows and PowerShell scripts to power on the VM, make the configurations, and restart it. The VM has a SSH server running on it and I use that to upload test files to the test VM, then remote execute them. Then download the results back to Jenkins for handling by the job. This solution is functional - but a lot more work than the typical Jenkins slave approach. Also, the scripts are targeted towards a single VM; I can't easily use a pool of slaves.
Not sure if this will work for you, but you might try making the Jenkins agent node programmatically tell the master node that it's offline.
I had a situation where I needed to make a Jenkins job that performs these steps (all while running on the master node):
revert the Jenkins agent node VM to a powered-off snapshot
tell the master that the agent node is disconnected (since the master does not seem to automatically notice the agent is down, whenever I revert or hard power off my VMs)
power the agent node VM back on
as a "Post-build action", launch a separate job restricted to run on the agent node VM
I perform the agent disconnect step with a curl POST request, but there might be a cleaner way to do it:
curl -d "offlineMessage=&json=%7B%22offlineMessage%22%3A+%22%22%7D&Submit=Yes" http://JENKINS_HOST/computer/THE_NODE_TO_DISCONNECT/doDisconnect
Then when I boot the agent node, the agent launches and automatically connects, and the master notices the agent is back online (and will then send it jobs).
I was also able to toggle a node's availability on and off with this command (using 'toggleOffline' instead of 'doDisconnect'):
curl -d "offlineMessage=back_in_a_moment&json=%7B%22offlineMessage%22%3A+%22back_in_a_moment%22%7D&Submit=Mark+this+node+temporarily+offline" http://JENKINS_HOST/computer/NODE_TO_DISCONNECT/toggleOffline
(Running the same command again puts the node status back to normal.)
The above may not apply to you since it sounds like you want to do everything from one jenkins job running on the agent node. And I'm not sure what happens if an agent node disconnects or marks itself offline in the middle of running a job. :)
Still, you might poke around in this Remote Access API doc a bit to see what else is possible with this kind of approach.
Very easy. You create a Master job that runs on the Master, from the master job you call the client job as a build step (it's a new kind of build step and I love it). You need to check that the master job should wait for the client job to finish. Then you can run your script to reconfigure your client and run the second test on the client.
An even better strategy is to have two nodes running on your slave machines. You need to configure two nodes in Jenkins. I used that strategy successfully with a unix slave. The reason was that I needed different environment variables to be set up and I didn't wanted to push that into the jobs. I used ssh clients, so I don't know if it is possible with different client types. Than you might be able to run both tests at the same time or you chain the jobs or use the master strategy mentioned above.
Can I run multiple hudson slaves on a single machine, I mean real slaves with only one build process?
My problem is, I have a slave with 3 build processes, using locks-and-latches (V0.4) to run three different kinds of build jobs. But sometimes I have the problem that more than one build job of one kind runs at the same time, or it blocks the build process from the slave and doesn't run.
Thank you in advance for your insights.
Yes, Hudson should be capable of running multiple slaves on a single machine. I do a limited form of this with my builds so that each job runs on a separate hard drive. In my case, this means I have a master, with a slave that is run on the same machine as the master. Having 3 slaves each with 1 executor could be done instead of one slave with 3 executors could be done, but it shouldn't impact locking so I only see a use for that if you have different physical drives and want more throughput.
I believe locks in both Hudson (i.e. this job is running) and locks-and-latches (this lock is in use) span all slaves & the master for a given hudson setup. So if slave 1 is running a job that holds Lock A, slave 2 won't be able to start a job that holds lock A either. It isn't entirely clear to me if this is the behavior you're seeking.
There is one important note, though:
Supposedly there is currently a bug in the hudson core that sometimes allows multiple jobs to start with the same lock when using the locks-and-latches plugin. I am not an expert on the internals of Hudson locking, nor the locks-and-latches plugin, but if you want a more in-depth explanation there is a conversation that sounds related on the hudson users mailing list (users#hudson.dev.java.net).
here is the archived conversation
The author of the locks-and-latches plugin is usually pretty responsive to questions.