I am running this code in tcl:-
set version [exec grep "INTERNAL VERSION:" mojave.log | sed -n -e "s/INTERNAL VERSION: //g" > xor.diff]
set p [exec diff ../log.warning.diff ../log.warning.gold >> xor.diff ]
For the last line it gives the following error after doing some diff:-
> RULE-311 WARNING: Gdsii layer number 85 datatype 0 has already been defined
> TCL-11 WARNING: Command "check quartz drc" is overridden, Quality Of
> TCL-11 WARNING: Command "delete marker quartz" is overridden, Quality Of
> TCL-11 WARNING: Command "import marker quartz" is overridden, Quality Of
> TCL-11 WARNING: Command "mojave! run filter log" is overridden, Quality Of
> TCL-11 WARNING: Command "run quartz gui" is overridden, Quality Of Results
> TCL-11 WARNING: Command "ui! mojave draw rectangle" is overridden, Quality
> TCL-11 WARNING: Command "ui! mojave set_context" is overridden, Quality Of
> TCL-12 WARNING: Overridden command "mojave! run filter log" is used,
child process exited abnormally
while executing
"exec diff ../log.warning.diff ../log.warning.gold 2> xor.diff "
invoked from within
"set p [exec diff ../log.warning.diff ../log.warning.gold 2> xor.diff ]"
(file "test.tcl" line 4)
invoked from within
"source test.tcl"
And it is not writing anything in "xor.diff" file.
exec returns an error if the command returns non-zero or if it sends anything to stderr. There's a very thorough discussion of using catch with exec at http://wiki.tcl.tk/exec
diff (and comm) exits nonzero if any differences are found, so you can redirect the output to use them as "did this change?" tests. For Tcl you'll want to use the `|| :' idiom to ignore the exit status.
P.S. "useless use of grep"... sed -n -e '/INTERNAL VERSION: /s//gp'
Related
A common pattern in GitHub action workflows is to run something like this:
- name: Install and Build 🔧
run: |
npm ci
npm run build
Clearly the intention is to run the second command only if the first command succeeds.
When running on Linux, the question becomes if the shell runs with set -e semantics. This answer suggests that set -e semantics are the default.
I'm trying to find that information in the documentation, but I'm a bit confused how it is specified. The section on exit codes contains the following for shell/sh shells:
Fail-fast behavior using set -eo pipefail: This option is set when shell: bash is explicitly specified. It is not applied by default.
This seems to contradict the other answer (and question!), and would mean that the above pattern actually is invalid, because the second line would be executed even if the first line fails.
Am I just misreading the documentation, or is it really necessary to either always specify set -e manually or add the shell: bash explicitly to get the desired behavior?
Does a GitHub action step use set -e semantics by default?
Yes, it does.
According to jobs.<job_id>.steps[*].shell, the sh and bash invocations do include -e whether specified or unspecified.
unspecified: bash -e {0}
with shell: bash: bash --noprofile --norc -eo pipefail {0}
with shell: sh: sh -e {0}
However, this section specified under Exit codes and error action preference:
bash/sh: Fail-fast behavior using set -eo pipefail: This option is set when shell: bash is explicitly specified. It is not applied by default.
applies to the -o pipefail part for Bash only. It could have been more explicit though.
An issue has been created on the GitHub docs repo to revise this:
https://github.com/github/docs/issues/23853
Problem
I'm not sure if the -N argument is being saved. SGE Cluster. Everything works except for the -N argument.
Snakemake requires a valid -N call
It doesn't set the job name properly.
It always reverts to the default name. This is my call, which has the same results, with or without the -N argument.
snakemake --jobs 100 --drmaa "-V -S /bin/bash -o log/mpileup/mpileupSPLIT -e log/mpileup/mpileupSPLIT -l h_vmem=10G -pe ncpus 1 -N {rule}.{wildcards}.varScan"
The only way I have found to influence the job name is to use --jobname.
snakemake --jobs 100 --drmaa "-V -S /bin/bash -o log/mpileup/mpileupSPLIT -e log/mpileup/mpileupSPLIT -l h_vmem=10G -pe ncpus 1 -N {rule}.{wildcards}.varScan" --jobname "{rule}.{wildcards}.{jobid}"
Background
I've tried a variety of things. Usually I actually just use a cluster configuration file, but that isn't working either, so that's why in the code above, I ditched the file system to make sure it's the '-N' command which isn't being saved.
My usual call is:
snakemake --drmaa "{cluster.clusterSpec}" --jobs 10 --cluster-config input/config.json
1) If I use '-n' instead of '-N', I receive a workflow error:
drmaa.errors.DeniedByDrmException: code 17: ERROR! invalid option argument "-n"
2) If I use '-N', but give it an incorrect wildcard, say {rule.name}:
AttributeError: 'str' object has no attribute 'name'
3) I cannot use both --drmaa AND --cluster:
snakemake: error: argument --cluster/-c: not allowed with argument --drmaa
4) If I specify the {jobid} in the config.json file, then Snakemake doesn't know what to do with it.
RuleException in line 13 of /extscratch/clc/projects/tboyarski/gitRepo-LCR-BCCRC/Snakemake/modules/mpileup/mpileupSPLIT:
NameError: The name 'jobid' is unknown in this context. Please make sure that you defined that variable. Also note that braces not used for variable access have to be escaped by repeating them, i.e. {{print $1}}
EDIT Added #5 w/ Solution
5) I can set the job name using the config.json and just concatenate the jobid on afterwards in my snakemake call. That way I have a generic snakemake call (--jobname "{cluster.jobName}.{jobid}"), and a highly configurable and specific job name ({rule}-{wildcards.sampleMPUS}_chr{wildcards.chrMPUS}) which results in:
mpileupSPLIT-Pfeiffer_chr19.1.e7152298
The 1 is the Snakemake jobid according to the DAG.
The 7152298 is my cluster's job number.
2nd EDIT - Just tried v3.12, same thing. Concatenation must occur in snakemake call.
Alternative solution
I would also be okay with something like this:
snakemake --drmaa "{cluster.clusterSpec}" --jobname "{cluster.jobName}" --jobs 10 --cluster-config input/config.json
With my cluster file like this:
"mpileupSPLIT": {
"clusterSpec": "-V -S /bin/bash -o log/mpileup/mpileupSPLIT -e log/mpileup/mpileupSPLIT -l h_vmem=10G -pe ncpus 1 -n {rule}.{wildcards}.varScan",
"jobName": "{rule}-{wildcards.sampleMPUS}_chr{wildcards.chrMPUS}.{jobid}"
}
Documentation Reviewed
I've read the documentation but I was unable to figure it out.
http://snakemake.readthedocs.io/en/latest/executable.html?-highlight=job_name#cluster-execution
http://snakemake.readthedocs.io/en/latest/snakefiles/configuration.html#snakefiles-cluster-configuration
https://groups.google.com/forum/#!topic/snakemake/whwYODy_I74
System
Snakemake v3.10.2 (Will try newest conda version tomorrow)
Red Hat Enterprise Linux Server release 5.4
SGE Cluster
Solution
Use '--jobname' in your snakemake call instead of '-N' in your qsub parameter submission
Setup your cluster config file to have a targetable parameter for the jobname suffix. In this case these are the overrides for my Snakemake rule named "mpileupSPLIT":
"mpileupSPLIT": {
"clusterSpec": "-V -S /bin/bash -o log/mpileup/mpileupSPLIT -e log/mpileup/mpileupSPLIT -l h_vmem=10G -pe ncpus 1",
"jobName": "{rule}-{wildcards.sampleMPUS}_chr{wildcards.chrMPUS}"
}
Utilize a generic Snakemake call which includes {jobid}. On a cluster (SGE), the 'jobid' variable contains both the Snakemake Job# and the Cluster Job#, both are valuable as the first corresponds to the Snakemake DAG and the later is for cluster logging. (E.g. --jobname "{cluster.jobName}.{jobid}")
EDIT Added solution to resolve post.
I'm executing the command wmic process where Description="java.exe" get caption through my tcl shell and getting below error:-
Node - HAW-TEST-04
ERROR:
Description = Invalid query
while the same commands runs perfectly from command terminal
`C:\>wmic process where Description="java.exe" get caption`
Caption
java.exe
java.exe
java.exe
java.exe
% exec wmic process where Description='java.exe' get caption
Caption
java.exe
%
I used this guide: Using MySQL in R for Windows but it didn't help. I'm already searching for hours on this problem. Since I can't figure out my problem I just can send here the log when trying to install.packages('RMySQL',type='source')
> Warning in install.packages : package ‘RMySQL’ is not available (for
> R version 3.0.2) Installing package into ‘C:/Users/Mr.
> Kox/Documents/R/win-library/3.0’ (as ‘lib’ is unspecified) trying URL
> 'http://cran.rstudio.com/src/contrib/RMySQL_0.9-3.tar.gz' Content type
> 'application/x-gzip' length 165363 bytes (161 Kb) opened URL
> downloaded 161 Kb
>
> * installing *source* package 'RMySQL' ...
> ** Paket 'RMySQL' erfolgreich entpackt und MD5 Summen überprüft
> 0 [main] sh 6324 find_fast_cwd: WARNING: Couldn't compute FAST_CWD pointer. Please report this problem to the public mailing
> list cygwin#cygwin.com checking for $MYSQL_HOME... C:/xampp/mysql
> cygwin warning: MS-DOS style path detected: C:/xampp/mysql
> Preferred POSIX equivalent is: /cygdrive/c/xampp/mysql CYGWIN
> environment variable option "nodosfilewarning" turns off this warning.
> Consult the user's guide for more details about POSIX paths:
> http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
> ** libs Warnung: this package has a non-empty 'configure.win' file, so building only the main architecture
>
> 0 [main] make 6988 find_fast_cwd: WARNING: Couldn't compute FAST_CWD pointer. Please report this problem to the public mailing
> list cygwin#cygwin.com cygwin warning: MS-DOS style path detected:
> C:/PROGRA~1/R/R-30~1.2/etc/x64/Makeconf Preferred POSIX equivalent
> is: /cygdrive/c/PROGRA~1/R/R-30~1.2/etc/x64/Makeconf CYGWIN
> environment variable option "nodosfilewarning" turns off this warning.
> Consult the user's guide for more details about POSIX paths:
> http://cygwin.com/cygwin-ug-net/using.html#using-pathnames gcc -m64 -I"C:/PROGRA~1/R/R-30~1.2/include" -DNDEBUG -I"C:/xampp/mysql"/include -I"d:/RCompile/CRANpkg/extralibs64/local/include" -O2 -Wall -std=gnu99 -mtune=core2 -c RS-DBI.c -o RS-DBI.o RS-DBI.c: In function 'RS_na_set': RS-DBI.c:1219:11: warning: variable 'c' set but not used
> [-Wunused-but-set-variable] gcc -m64
> -I"C:/PROGRA~1/R/R-30~1.2/include" -DNDEBUG -I"C:/xampp/mysql"/include -I"d:/RCompile/CRANpkg/extralibs64/local/include" -O2 -Wall -std=gnu99 -mtune=core2 -c RS-MySQL.c -o RS-MySQL.o RS-MySQL.c: In function 'RS_MySQL_fetch': RS-MySQL.c:657:13: warning: variable
> 'fld_nullOk' set but not used [-Wunused-but-set-variable] RS-MySQL.c:
> In function 'RS_DBI_invokeBeginGroup': RS-MySQL.c:1137:30: warning:
> variable 'val' set but not used [-Wunused-but-set-variable]
> RS-MySQL.c: In function 'RS_DBI_invokeNewRecord': RS-MySQL.c:1158:20:
> warning: variable 'val' set but not used [-Wunused-but-set-variable]
> RS-MySQL.c: In function 'RS_MySQL_dbApply': RS-MySQL.c:1219:38:
> warning: variable 'fld_nullOk' set but not used
> [-Wunused-but-set-variable] gcc -m64 -shared -s -static-libgcc -o
> RMySQL.dll tmp.def RS-DBI.o RS-MySQL.o C:/xampp/mysql/bin/libmySQL.dll
> -Ld:/RCompile/CRANpkg/extralibs64/local/lib/x64 -Ld:/RCompile/CRANpkg/extralibs64/local/lib -LC:/PROGRA~1/R/R-30~1.2/bin/x64 -lR collect2: ld returned 5 exit status ERROR: compilation failed for package 'RMySQL'
> * removing 'C:/Users/Mr. Kox/Documents/R/win-library/3.0/RMySQL'
> * restoring previous 'C:/Users/Mr. Kox/Documents/R/win-library/3.0/RMySQL' Warning in install.packages :
> running command '"C:/PROGRA~1/R/R-30~1.2/bin/x64/R" CMD INSTALL -l
> "C:\Users\Mr. Kox\Documents\R\win-library\3.0"
> C:\Users\MRFFAF~1.KOX\AppData\Local\Temp\RtmpCGe2RS/downloaded_packages/RMySQL_0.9-3.tar.gz'
> had status 1 Warning in install.packages : installation of package
> ‘RMySQL’ had non-zero exit status
>
> The downloaded source packages are in ‘C:\Users\Mr.
> Kox\AppData\Local\Temp\RtmpCGe2RS\downloaded_packages’
I think the main problem is in "collect2: ld returned 5 exit status". Well maybe you guys can help me out. Overall I just want to connect R-Studio with a MySQL database. Thanks.
Found the solution. It's pretty easy.
Go to R-Studio -> Tools -> Options -> R-Version -> and here instead of the current choose "[Default] [32-bit] C:\Program Files\R\R-3.0.2"
Now it works for the issue above. Have fun.
Use RODBC and this guide: http://www.joyofdata.de/blog/make-r-talk-to-mysql-on-windows/
You are using R 64bit apparently - it is important to then use the 64bit connector.
I just wrote a very simple Expect script for wrapping around rsync, but it seems to be giving me trouble. Basically, I am automating the SSH login prompt called from rsync. I also have to pass arguments through rsync to SSH so it doesn't do the host key checking. I am well aware of SSH authentication keys and ssh-keygen, but I have good reasons for doing things this way so no lectures on passing passwords on the command-line.
Script
#!/usr/local/bin/expect -f
if {$argc != 5} {
puts "usage: remoteCopy {remotehost, username, password, localfile, remoteloc}"
exit 1
}
set remotehost [lindex $argv 0]
set username [lindex $argv 1]
set password [lindex $argv 2]
set localfile [lindex $argv 3]
set remoteloc [lindex $argv 4]
set timeout -1
spawn rsync -e \"ssh -q -o StrictHostKeyChecking=no\" $localfile $username#$remotehost:$remoteloc
expect "Password"; send "$password\r"
Here is the complete output from the script:
Output
avoelker#localhost $ ./remoteFileCopy.tcl remotehost remoteuser remotepass ~/localfile /tmp/
spawn rsync -e "ssh -q -o StrictHostKeyChecking=no" /localhost/home/avoelker/localfile remoteuser#remotehost:/tmp/
rsync: Failed to exec "ssh: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(83)
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(434)
send: spawn id exp6 not open
while executing
"send "$password\r""
(file "./remoteFileCopy.tcl" line 17)
avoelker#localhost $
It appears to me that rsync is trying to execute "ssh instead of just ssh, however, copying and pasting the first line of output from Expect's stdout (rsync -e "ssh -q -o StrictHostKeyChecking=no" /localhost/home/avoelker/localfile remoteuser#remotehost:/tmp/) directly into the shell works perfectly fine, so I know it isn't a problem with rsync.
When I remove the -e argument entirely from the spawn rsync line, the Expect script executes without errors, but I would have to add in host key checking to the script, which I do not want to do.
Does someone more experienced with Expect/Tcl know what I am doing wrong?
Drop your backslashes. The spawn line should just say:
spawn rsync -e "ssh -q -o StrictHostKeyChecking=no" $localfile $username#$remotehost:$remoteloc
After all, you don't want rsync to see a command with a quote in it, do you?