I would like to pass an argument into the sbatch command line.
RHO_COR.sh
#!/bin/bash
#SBATCH -o job-%A_task.out
#SBATCH --job-name=paral_cor
#SBATCH --partition=normal
#SBATCH --time=1-00:00:00
#SBATCH --mem=200G
#SBATCH --cpus-per-task=16
#SBATCH --array=1-10
#Set up whatever package we need to run with
module load gcc/8.1.0 openblas/0.3.3 R
# SET UP DIRECTORIES
OUTPUT="$HOME"/PROJET_M2/data/$(date +"%Y%m%d")_parallel_nodes_test
mkdir -p "$OUTPUT"
export FILENAME="$HOME"/vipailler/PROJET_M2/bin/RHO_COR.R
subset=$((SLURM_ARRAY_TASK_ID))
file="$HOME"/PROJET_M2/raw/truelength2.prok2.uniref2.rares.tsv
#Run the program
echo "Start job :"`date` >> "$OUTPUT"/"$SLURM_ARRAY_TASK_ID".txt
echo "Start job :"`date`
echo PWD $PWD
Rscript $FILENAME --file $file --subset $subset > "$OUTPUT"/"$SLURM_ARRAY_TASK_ID"
wait
echo "Stop job : "`date` >> "$OUTPUT"/"$SLURM_ARRAY_TASK_ID".txt
echo "Stop job : "`date`
I execute this code with :
sbatch --partition normal --array 1-10 RHO_COR.sh
What I would like to get is to use the file argument into the command line above, which looks like sbatch --partition normal --array 1-10 --file name_of_my_file RHO_COR.sh
I don't want to specify the name of my file into the Slurm code, but into the sbatch command line, in order to never change this Slurm code.
Thanks
You can pass an argument after the script as if you were running it directly on the shell like this:
sbatch --partition normal --array 1-10 RHO_COR.sh name_of_my_file
And then the argument will be available inside the shell script as $1
Related
I am trying to use a deepvariant singularity container on my HPC.
However there is something wrong with the way that I am using the container and where it is binding.
This is my code:
#!/bin/bash --login
#SBATCH -J AmyHouseman_deepvariant
#SBATCH -o %x.stdout.%J.%N
#SBATCH -e %x.stderr.%J.%N
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH -p c_compute_wgp
#SBATCH --account=scw1581
#SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=HousemanA#cardiff.ac.uk # Where to send mail
#SBATCH --array=1-33
#SBATCH --time=02:00:00
#SBATCH --time=072:00:00
#SBATCH --mem-per-cpu=32GB
module purge
module load singularity
module load parallel
# Set bash error trapping to exit on first error.
set -eu
WDPATH=/scratch/$USER/$SLURM_ARRAY_JOB_ID
CONTAINER_FILE=deepvariant_1.3.0.sif
MY_CONTAINER_PATH=/scratch/$USER/containers
CONTAINER=$MY_CONTAINER_PATH/$CONTAINER_FILE
if [ "$SLURM_ARRAY_TASK_ID" == "1" ]
then
mkdir -p $MY_CONTAINER_PATH
[ -f "$CONTAINER" ] || ssh cl1 wget -O $CONTAINER https://wotan.cardiff.ac.uk/containers/$CONTAINER_FILE
mkdir $WDPATH
fi
while [ ! -d $WDPATH ]
do
sleep 10
done
cd /scratch/c.c21087028/
sed -n "${SLURM_ARRAY_TASK_ID}p" Polyposis_Exome_Analysis/fastp/All_fastp_input/List_of_33_exome_IDs | parallel -j 1 "singularity run $CONTAINER --model_type=WES \
-ref=Polyposis_Exome_Analysis/bwa/index/HumanRefSeq/GRCh38_latest_genomic.fna \
--reads=Polyposis_Exome_Analysis/samtools/index/indexed_picardbamfiles/{}PE_markedduplicates.bam \
--output_vcf=Polyposis_Exome_Analysis/deepvariant/vcf/{}PE_output.vcf.gz \
--output_gvcf=Polyposis_Exome_Analysis/deepvariant/gvcf/{}PE_output.vcf.gz \
--intermediate_results_dir=Polyposis_Exome_Analysis/deepvariant/intermediateresults/{}PE_output_intermediate"
The error message I get is set:
invalid option: "--"
FATAL: "--model_type": executable file not found in $PATH
I find it a bit strange because I haven't had a problem with any of the other singularity containers I've used before, so not really sure how to go ahead - I've tried adding in the bit in their manual singularity run -B /usr/lib/locale/:/usr/lib/locale/
but I'm still unsure as to why I would need to do this step when I haven't previously on my other tools like bwa, and samtools.
I also know I can get rid of parallel bits as its not actually running in parallel, so I am aware of that.
I hope this makes sense!
Thank you!
Amy
I run Modelsim in the cmd from a python program.
I use the following code which call a tcl script which run the modelsim:
os.system("vsim -c -do top_tb_simulate_reg.tcl " )
The tcl script contain the following:
vsim -voptargs="+acc" +UVM_TESTNAME=test_name +UVM_MAX_QUIT_COUNT=1 +UVM_VERBOSITY=UVM_LOW \
-t 1ps -L unisims_verm -L generic_baseblocks_v2_1_0 -L axi_infrastructure_v1_1_0 \
-L dds_compiler_v6_0_12 -lib xil_defaultlib xil_defaultlib.girobo2_tb_top \
xil_defaultlib.glbl
I want that the value of the +UVM_TESTNAME will be an argument which I passed from the cmd when I execute:
os.system("vsim -c -do top_tb_simulate_reg.tcl " )
How can I do it?
I tried the following with no succees:
Python script:
os.system("vsim -c -do top_tb_simulate_reg.tcl axi_rd_only_test" )
Simulation file (tcl script)
vsim -voptargs="+acc" +UVM_TESTNAME=$argv +UVM_MAX_QUIT_COUNT=1 +UVM_VERBOSITY=UVM_LOW \
-t 1ps -L unisims_verm -L generic_baseblocks_v2_1_0 -L axi_infrastructure_v1_1_0 \
-L dds_compiler_v6_0_12 -lib xil_defaultlib xil_defaultlib.girobo2_tb_top \
xil_defaultlib.glbl
I got the following error:
# ** Error: (vsim-3170) Could not find 'C:/raft/raftortwo/girobo2/ver/sim/work.axi_rd_only_test'.
The problem is that the vsim binary is doing its own processing of the arguments, and that is interfering. While yes, you can probably find a way around this by reading the vsim documentation, the simplest way around this is to pass values via environment variables. They're inherited by a process from its parent process, and are fine for passing most things. (The exception are security tokens, which should always be passed in files with correctly-set permissions, rather than either environment variables or command-line arguments.)
In your python code:
# Store the value in the *inheritable* environment
os.environ["MY_TEST_CASE"] = "axi_rd_only_test"
# Do the call; the environment gets passed over behind the scenes
os.system("vsim -c -do top_tb_simulate_reg.tcl " )
In your tcl code:
# Read out of the inherited environment
set name $env(MY_TEST_CASE)
# Use it! (Could do this as one line, but that's hard to read)
vsim -voptargs="+acc" +UVM_TESTNAME=$name +UVM_MAX_QUIT_COUNT=1 +UVM_VERBOSITY=UVM_LOW \
-t 1ps -L unisims_verm -L generic_baseblocks_v2_1_0 -L axi_infrastructure_v1_1_0 \
-L dds_compiler_v6_0_12 -lib xil_defaultlib xil_defaultlib.girobo2_tb_top \
xil_defaultlib.glbl
Late to the party but I found a great workaround for your obstacle. The do command within Modelsim's TCL instance does accept parameters. See command reference.
vsim -c -do filename.tcl can't take parameters, but you can use vsim -c -do "do filename.tcl params".
In your case this translates to os.system('vsim -c -do "do top_tb_simulate_reg.tcl axi_rd_only_test"'). Your .tcl script will find the parameter passed through the variable $1.
I hope to helps anyone!
I have the shell script:
#!/bin/bash
export LD=$(lsb_release -sd | sed 's/"//g')
export ARCH=$(uname -m)
export VER=$(lsb_release -sr)
# Load the test function
/bin/bash -c "lib/test.sh"
echo $VER
DISTROS=('Arch'
'CentOS'
'Debian'
'Fedora'
'Gentoo')
for I in "${DISTROS[#]}"
do
i=$(echo $I | tr '[:upper:]' '[:lower:]') # convert distro string to lowercase
if [[ $LD == "$I"* ]]; then
./$ARCH/${i}.sh
fi
done
As you can see it should run a shell script, depending on which architecture and OS it is run on. It should first run the script lib/test.sh before it runs this architecture and OS-specific script. This is lib/test.sh:
#!/bin/bash
function comex {
which $1 >/dev/null 2>&1
}
and when I run it on x86_64 Arch Linux with this x86_64/arch.sh script:
#!/bin/bash
if comex atom; then
printf "Atom is already installed!"
elif comex git; then
printf "Git is installed!"
fi
it returned the output:
rolling
./x86_64/arch.sh: line 3: comex: command not found
./x86_64/arch.sh: line 5: comex: command not found
so clearly the comex shell function is not correctly loaded by the time the x86_64/arch.sh script is run. Hence I am confused and wondering what I need to do in order to correctly define the comex function such that it is correctly loaded in this architecture- and OS-dependent final script.
I have already tried using . "lib/test.sh" instead of /bin/bash -c "lib/test.sh" and I received the exact same error. I have also tried adding . "lib/test.sh" to the loop, just before the ./$ARCH/${i}.sh line. This too failed, returning the same error.
Brief answer: you need to import your functions using . or source instead of bash -c:
# Load the test function
source "lib/test.sh"
Longer answer: when you call script with bash -c, a child process is created. This child process sees all exported variables (including functions) from parent process. But not vice versa. So, your script will never see comex function. Instead you need to include script code directly in current script and you do so by using . or source commands.
Part 2. After you "sourced" lib/test.sh, your main script is able to use comex function. But arch scripts won't see this function because it is not exported to them. Your need to export -f comex:
#!/bin/bash
function comex {
which $1 >/dev/null 2>&1
}
export -f comex
I'm having difficulty with a script I'm writing. The script is largely incomplete, but so far I expect it to be able to run containers successfully. When I execute the script I'm given an error with a container ID and "command not found". For example: ./wordpress: line 73: 3c0fba4984f3b70f0eb3f1c15a7b157f4862b9b243657a3d2f7141029fb6641a: command not found
The script I'm using is as follows:
#!/bin/bash
echo "Setting Constants"
MYSQL_ROOT_PASSWORD='password'
MYSQL_DATABASE='wordpress'
WORDPRESS_DB_PASSWORD='password'
WP_PORT='80'
DB_PORT='3306'
EPOCH=$(date +%s) # append EPOCH to container names for uniqueness
#FILE='blogcontainers' # filename containing container IDs
DB_CONTAINER_NAME="myblogdb$EPOCH"
WP_CONTAINER_NAME="myblog$EPOCH"
DB_IMG_NAME='blogdb' # MySQL Docker image
WP_IMG_NAME='blog' # WordPress Docker image
cd ~/myblog
WP_CID_FILE="$PWD/blog.cid"
DB_CID_FILE="$PWD/blogdb.cid"
if [ -f $DB_CID_FILE ]; then
DB_IMG_ID=$(sed -n '1p' $DB_CID_FILE)
else
echo "dbcid not found"
# set to baseline image
DB_IMG_ID="f09a5b2903dc"
fi
if [ -f $WP_CID_FILE ]; then
WP_IMG_ID=$(sed -n '1p' $WP_CID_FILE)
else
echo "wpcid not found"
# set to baseline image
WP_IMG_ID="a8d48bc2313d"
fi
DB_PATH='/var/lib/mysql' # standard MySQL path
WP_PATH='/var/www/html' # standard WordPress path
LOCAL_DB_PATH="/$PWD$DB_PATH"
LOCAL_WP_PATH="/$PWD$WP_PATH"
echo "Starting MySQL Container"
#DB_ID=
$(docker run \
-e MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD \
-e MYSQL_DATABASE=$MYSQL_DATABASE \
-v $LOCAL_WP_PATH:$DB_PATH \
-v /$PWD/.bash_history:$WP_PATH \
--name $DB_CONTAINER_NAME \
-p $DB_PORT:3306 \
--cidfile $DB_CID_FILE \
-d \
$DB_IMG_ID)
echo "Starting WordPress Container"
#WP_ID=
$(docker run \
-e WORDPRESS_DB_PASSWORD=$WORDPRESS_DB_PASSWORD \
--link $DB_CONTAINER_NAME:$DB_IMG_NAME \
-p $WP_PORT:80 \
-v $LOCAL_WP_PATH:$WP_PATH \
-v /$PWD/.bash_history:/root/.bash_history \
--name $WP_CONTAINER_NAME \
--cidfile $WP_CID_FILE \
-d \
$WP_IMG_ID)
echo $WP_CONTAINER_NAME
echo $WP_IMG_ID
echo "reached end"
#echo $WP_ID > $FILE # copy WordPress container ID to file
#echo $DB_ID >> $FILE # append MySQL container ID to file
After executing the code there usually is a MySQL container instance running. For example:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4f2e9ab14c2e f09a5b2903dc "/entrypoint.sh mysql" 2 seconds ago Up 2 seconds 0.0.0.0:3306->3306/tcp myblogdb1449768739
Also, both blog.cid and blogdb.cid are created successfully containing container IDs.
$ cat blog.cid
e6005bcb4dba524b121d02b301fbe421d67d60986c55d554a0e20443df27ed18
$ cat blogdb.cid
4f2e9ab14c2ea5361557a3714477d7758c993af3b08bbc7db529282a41f90959
I've been troubleshooting and searching around for answers, but I think it's time to have another set of eyes take a look at it. As always, any input/criticism are welcome.
You are using $(docker run ...) instead of simply docker run .... The command substitution ($(...)) runs the command, captures the output, and expands to that output. As a result, you are trying to run the output of docker run as a command.
I'm having an issue passing variables to a Bash script using QSub.
Assume I have a Bash script named example. The format of example is the following:
#!/bin/bash
# (assume other variables have been set)
echo $1 $2 $3 $4
So, executing "bash example.sh this is a test" on Terminal (I am using Ubuntu 12.04.3 LTS, if that helps) produces the output "this is a test".
However, when I enter "qsub -v this,is,a,test example.sh", I get no output. I checked the output file that QSub produces, but the line "this is a test" is nowhere to be found.
Any help would be appreciated.
Thank you.
Using PBSPro or SGE, arguments can simply be placed after the script name as may seem intuitive.
qsub example.sh hello world
In Torque, command line arguments can be submitted using the -F option. Your example.sh will look something like this:
#!/bin/bash
echo "$1 $2"
and your command like so:
qsub -F "hello world" example.sh
Alternatively, environment variables can be set using -v with a comma-separated list of variables.
#!/bin/bash
echo "$FOO $BAR"
and your command like so:
qsub -v FOO="hello",BAR="world" example.sh
(This may be better phrased as a comment on #William Hay's answer, but I don't have the reputation to do so.)
Not sure which batch scheduler you are using but on PBSPro or SGE then submitting with qsub example.sh this is a test should do what you want.
The Torque batch scheduler doesn't (AFAIK) allow passing command line arguments to the script this way. You would need to create a script looking something like this.
#!/bin/bash
echo $FOO
Then submit it with a command like:
qsub -v FOO="This is a test" example.sh