gcloud beta lifesciences, JSON pipeline-file rather than options - json

Question

I am trying to run gcloud beta lifesciences because genomics API is deprecated.
There have been so many changes, genomics API vs lifesciences API.
I ran one of my analysis step in google clooud using beta lifesciences.
Here is what I found.
(1) wildcard is not working in command line options (2) It is not easy to set the target directory in command line option, I used env-var for copy.
I am now trying to convert commandline option into JSON format pipeline-file, but it is not easy to understand help page in google cloud. Do you have an idea how to convert following options into JSON file, so I could run it with simpler option?
I used YAML formatted pipeline file in genomics API, but beta lifescienes is totally different.
$ more step03_bwa_mem_genome1.run
#SMALL=
SMALL=chr21.
LIFESCIENCESPATH=/gcloud-shared
#LIFESCIENCESPATH=/mnt
SCRIPTFILENAME=step03_bwa_mem_genome.sh
COHORTID=2_C_222
gcloud beta lifesciences pipelines run \
--logging gs://${BUCKETID}/ExomeSeq/hResults/step03_bwa_mem_genome.${COHORTID}.log \
--regions=asia-northeast1,asia-northeast2,asia-northeast3,asia-east1,asia-east2,asia-south1 \
--boot-disk-size 20 \
--preemptible \
--machine-type n1-standard-1 \
--disk-size "gcloud-shared:10" \
--docker-image asia.gcr.io/thermal-shuttle-199104/centos8-essential-software-genomics-custom-python3:0.4 \
--inputs REFERENCE1=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.amb \
--inputs REFERENCE2=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.ann \
--inputs REFERENCE3=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.bwt \
--inputs REFERENCE4=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.fai \
--inputs REFERENCE5=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.intervals \
--inputs REFERENCE6=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.pac \
--inputs REFERENCE7=gs://${BUCKETID}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.sa \
--inputs SCRIPTFILE=gs://${BUCKETID}/ExomeSeq/${SCRIPTFILENAME} \
--inputs COHORTID=${COHORTID} \
--inputs SAMPLELIST=gs://${BUCKETID}/ExomeSeq/SAMPLELIST.${COHORTID}.lst \
--inputs INPUTFILE1=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_01_1.chr21.fastq.gz \
--inputs INPUTFILE2=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_01_2.chr21.fastq.gz \
--inputs INPUTFILE3=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_02_1.chr21.fastq.gz \
--inputs INPUTFILE4=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_02_2.chr21.fastq.gz \
--inputs INPUTFILE5=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_03_1.chr21.fastq.gz \
--inputs INPUTFILE6=gs://${BUCKETID}/ExomeSeq/hReads/${COHORTID}_03_2.chr21.fastq.gz \
--outputs OUTPUTFILE1=gs://${BUCKETID}/ExomeSeq/hResults/${COHORTID}_01.bam \
--outputs OUTPUTFILE2=gs://${BUCKETID}/ExomeSeq/hResults/${COHORTID}_02.bam \
--outputs OUTPUTFILE3=gs://${BUCKETID}/ExomeSeq/hResults/${COHORTID}_03.bam \
--env-vars REFERENCE1=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.amb,REFERENC
E2=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.ann,REFERENCE3=${LIFESCIENCESPATH}/
ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.bwt,REFERENCE4=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRC
h38.primary_assembly.genome.${SMALL}fa.fai,REFERENCE5=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.ge
nome.${SMALL}fa.intervals,REFERENCE6=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.p
ac,REFERENCE7=${LIFESCIENCESPATH}/ExomeSeq/hReference/GRCh38.primary_assembly.genome.${SMALL}fa.sa,SCRIPTFILE=${LIFESCIE
NCESPATH}/ExomeSeq/${SCRIPTFILENAME},SAMPLELIST=${LIFESCIENCESPATH}/ExomeSeq/SAMPLELIST.${COHORTID}.lst,INPUTFILE1=${LIF
ESCIENCESPATH}/ExomeSeq/hReads/${COHORTID}_01_1.chr21.fastq.gz,INPUTFILE2=${LIFESCIENCESPATH}/ExomeSeq/hReads/${COHORTID
}_01_2.chr21.fastq.gz,INPUTFILE3=${LIFESCIENCESPATH}/ExomeSeq/hReads/${COHORTID}_02_1.chr21.fastq.gz,INPUTFILE4=${LIFESC
IENCESPATH}/ExomeSeq/hReads/${COHORTID}_02_2.chr21.fastq.gz,INPUTFILE5=${LIFESCIENCESPATH}/ExomeSeq/hReads/${COHORTID}_0
3_1.chr21.fastq.gz,INPUTFILE6=${LIFESCIENCESPATH}/ExomeSeq/hReads/${COHORTID}_03_2.chr21.fastq.gz,OUTPUTFILE1=${LIFESCIE
NCESPATH}/ExomeSeq/hResults/${COHORTID}_01.bam,OUTPUTFILE2=${LIFESCIENCESPATH}/ExomeSeq/hResults/${COHORTID}_02.bam,OUTP
UTFILE3=${LIFESCIENCESPATH}/ExomeSeq/hResults/${COHORTID}_03.bam \
--command-line="find ${LIFESCIENCESPATH}; /bin/bash ${LIFESCIENCESPATH}/ExomeSeq/${SCRIPTFILENAME} ${COHORTID} 4"

Answer 1

Thank you for your answer!
Finally, I made an input YML file for gcloud lifesciences from operation descriptions. I needed to understand basic functionality of gcloud lifesciences because I want to make full version of genome analysis pipeline, from FASTQ to snpEff/third party annotation/text extraction in google cloud (whole-exome in a day, whole-genome in 5 days for a single set). I already made it using genomics API, but am trying to upgrade it to use gcloud lifesciences.
I also tried gcsfuse, but setting up google cloud authentication is a little bit tricky.
Thanks,
$ more step03_bwa_mem_genome2.run
#SMALL=
SMALL=chr21.
LIFESCIENCESPATH=/gcloud-shared
#LIFESCIENCESPATH=/mnt
SCRIPTFILENAME=step03_bwa_mem_genome.sh
COHORTID=2_C_222
gcloud beta lifesciences pipelines run \
--logging gs://${BUCKETID}/ExomeSeq/hResults/step03_bwa_mem_genome.${COHORTID}.log \
--pipeline-file step03_bwa_mem_genome1.yml
$ cat step03_bwa_mem_genome1.yml
actions:
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_03_2.chr21.fastq.gz ${INPUTFILE6}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_03_1.chr21.fastq.gz ${INPUTFILE5}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_02_2.chr21.fastq.gz ${INPUTFILE4}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_02_1.chr21.fastq.gz ${INPUTFILE3}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_01_2.chr21.fastq.gz ${INPUTFILE2}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReads/2_C_222_01_1.chr21.fastq.gz ${INPUTFILE1}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/SAMPLELIST.2_C_222.lst ${SAMPLELIST}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/step03_bwa_mem_genome.sh ${SCRIPTFILE}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.sa ${REFERENCE7}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.pac ${REFERENCE6}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.intervals ${REFERENCE5}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.fai ${REFERENCE4}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.bwt ${REFERENCE3}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.ann ${REFERENCE2}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp gs://genconv1/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.amb ${REFERENCE1}
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- -c
- find /gcloud-shared; /bin/bash /gcloud-shared/ExomeSeq/step03_bwa_mem_genome.sh 2_C_222 4
entrypoint: bash
imageUri: asia.gcr.io/thermal-shuttle-199104/centos8-essential-software-genomics-custom-python3:0.4
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp ${OUTPUTFILE1} gs://genconv1/ExomeSeq/hResults/2_C_222_01.bam
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp ${OUTPUTFILE2} gs://genconv1/ExomeSeq/hResults/2_C_222_02.bam
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- commands:
- /bin/sh
- -c
- gsutil -m -q cp ${OUTPUTFILE3} gs://genconv1/ExomeSeq/hResults/2_C_222_03.bam
imageUri: google/cloud-sdk:slim
mounts:
- disk: gcloud-shared
path: /gcloud-shared
- alwaysRun: true
commands:
- /bin/sh
- -c
- gsutil -m -q cp /google/logs/output gs://genconv1/ExomeSeq/hResults/step03_bwa_mem_genome.2_C_222.log
imageUri: google/cloud-sdk:slim
environment:
COHORTID: 2_C_222
INPUTFILE1: /gcloud-shared/ExomeSeq/hReads/2_C_222_01_1.chr21.fastq.gz
INPUTFILE2: /gcloud-shared/ExomeSeq/hReads/2_C_222_01_2.chr21.fastq.gz
INPUTFILE3: /gcloud-shared/ExomeSeq/hReads/2_C_222_02_1.chr21.fastq.gz
INPUTFILE4: /gcloud-shared/ExomeSeq/hReads/2_C_222_02_2.chr21.fastq.gz
INPUTFILE5: /gcloud-shared/ExomeSeq/hReads/2_C_222_03_1.chr21.fastq.gz
INPUTFILE6: /gcloud-shared/ExomeSeq/hReads/2_C_222_03_2.chr21.fastq.gz
OUTPUTFILE1: /gcloud-shared/ExomeSeq/hResults/2_C_222_01.bam
OUTPUTFILE2: /gcloud-shared/ExomeSeq/hResults/2_C_222_02.bam
OUTPUTFILE3: /gcloud-shared/ExomeSeq/hResults/2_C_222_03.bam
REFERENCE1: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.amb
REFERENCE2: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.ann
REFERENCE3: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.bwt
REFERENCE4: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.fai
REFERENCE5: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.intervals
REFERENCE6: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.pac
REFERENCE7: /gcloud-shared/ExomeSeq/hReference/GRCh38.primary_assembly.genome.chr21.fa.sa
SAMPLELIST: /gcloud-shared/ExomeSeq/SAMPLELIST.2_C_222.lst
SCRIPTFILE: /gcloud-shared/ExomeSeq/step03_bwa_mem_genome.sh
resources:
regions:
- asia-northeast1
- asia-northeast2
- asia-northeast3
- asia-east1
- asia-east2
- asia-south1
virtualMachine:
bootDiskSizeGb: 20
disks:
- name: gcloud-shared
sizeGb: 10
machineType: n1-standard-1
preemptible: true

Answer 2

I would generally recommend using something like Cromwell, Nextflow, or Snakemake rather than using either API directly. They have more built in functionality for these types of tasks.
However, the output from gcloud beta lifesciences operations describe <operation name> will include the pipeline definition that gcloud created, which could be used as a starting point. One thing you'll notice there is that --inputs and --outputs automatically create environment variables so the LIFESCIENCESPATH variable and the --env-vars parameter are unnecessary which will simplify the command line significantly.