Is there a way to use AWS Step Functions Input to assembly command string on System Manager block? - json

I am creating a Step Function State machine to everytime an instance starts it copy a file from S3 to an specific folder inside this instance. The origin folder inside S3 bucket has a folder named with this instance ID. The instance ID I am passing as input for the System manager block, but I need to use it to create the command string that will be performed inside the EC2.
For example:
My input is: $.detail.instance-id (lets assume the following ID i-11223344556677889)
The Systems Manager API parameters are:
"CloudWatchOutputConfig": {
"CloudWatchLogGroupName": "myloggroup",
"CloudWatchOutputEnabled": true
},
"DocumentName": "AWS-RunShellScript",
"DocumentVersion": "$DEFAULT",
"InstanceIds.$": "States.Array($)",
"MaxConcurrency": "50",
"MaxErrors": "10",
"Parameters": {
"commands": [
{
"runuser -l ec2-user -c \"aws s3 cp s3://my-bucket/**MY_INSTANCEID**/myfile.xyz /home/ec2-user/myfolder/myfile.xyz\""
}
},
"TimeoutSeconds": 6000
}```
Summing up, I want to turn the line with the command replacing the MY_INSTANCEID by my input $.detail.instance-id, and perform the following command:
"runuser -l ec2-user -c "aws s3 cp s3://my-bucket/i-11223344556677889/myfile.xyz /home/ec2-user/myfolder/myfile.xyz""
Is there a way? I already tried to use the Fn::join withou success.
Thank you in advance,
kind regards,
Winner

It was necessary to use State.Format inside the State.Array so the it worked, and State.Format inside the State.Array cannot have quotes:
"CloudWatchOutputConfig": {
"CloudWatchLogGroupName": "myloggroup",
"CloudWatchOutputEnabled": true
},
"DocumentName": "AWS-RunShellScript",
"DocumentVersion": "$DEFAULT",
"InstanceIds.$": "States.Array($)",
"MaxConcurrency": "50",
"MaxErrors": "10",
"Parameters": {
"commands.$": "States.Array(States.Format('runuser -l ec2-user -c \"aws s3 cp s3://my-bucket/**MY_INSTANCEID**/myfile.xyz /home/ec2-user/myfolder/myfile.xyz\"', $))"
},
"TimeoutSeconds": 6000
}```
Was also necessary to use .$ after command.

Related

AWS CLI events put-targets EcsParameters (structure)

I am trying to get the current Task Definition of an ECS Cluster then update the revision in the cloud bridge event target.
This is what I have so far:
james#LAPTOP:/mnt/c/Users/james$ target_id="xxx_hourly_cron"
james#LAPTOP:/mnt/c/Users/james$ aws events list-targets-by-rule --rule `echo $target_id` > rule-target.json
james#LAPTOP:/mnt/c/Users/james$ cat rule-target.json
{
"Targets": [
{
"Id": "xxx_hourly_cron",
"Arn": "arn:aws:ecs:eu-west-2:000000000000:cluster/xxx-cluster",
"RoleArn": "arn:aws:iam::000000000000:role/ecsEventsRole",
"EcsParameters": {
"TaskDefinitionArn": "arn:aws:ecs:eu-west-2:000000000000:task-definition/xxx-cron:55",
"TaskCount": 1,
"EnableECSManagedTags": false,
"EnableExecuteCommand": false,
"PropagateTags": "TASK_DEFINITION"
}
}
]
}
james#LAPTOP:/mnt/c/Users/james$ aws events put-targets --rule `echo $target_id` --targets --EcsParameters jsonfile?
The last command is where I am struggling:
within the AWS docs I am not to sure what it means by structure I have tried json and I have tried to escape it.
Here is the docs I am looking at:
https://docs.aws.amazon.com/cli/latest/reference/events/put-targets.html
With AWS CLI commands, you can often replace the majority of the arguments supplied to a command with a single JSON file using the --cli-input-json argument. This can make it far easier to work with complex structures as input arguments to CLI commands.
In the above example, you would modify the rule-target.json output to become an input (rule-target-input.json) for the next using something like the following:
{
"Rule": "xxx_hourly_cron",
"Targets": [
{
"Id": "xxx_hourly_cron",
"Arn": "arn:aws:ecs:eu-west-2:000000000000:cluster/xxx-cluster",
"RoleArn": "arn:aws:iam::000000000000:role/ecsEventsRole",
"EcsParameters": {
"TaskDefinitionArn": "arn:aws:ecs:eu-west-2:000000000000:task-definition/xxx-cron:55",
"TaskCount": 1,
"EnableECSManagedTags": false,
"EnableExecuteCommand": false,
"PropagateTags": "TASK_DEFINITION"
}
}
]
}
And then feeding that into the input using something like the following:
aws put-targets --cli-input-json file://rule-target-input.json

Invalid JSON while submitting spark submit job via NiFi

I am trying to submit a spark job where I am setting a date argument in conf property and I am running it through a script in NiFi. However, when I am running the script I am facing an error.
Spark Submit Code in the script:
aws emr add-steps --cluster-id "$1" --steps '[{"Args":["spark-submit","--deploy-mode","cluster","--jars","s3://tvsc-lumiq-edl/jars/ojdbc7.jar","--executor-memory","10g","--driver-memory","10g","--conf","spark.hadoop.yarn.timeline-service.enabled=false","--conf","currDate='\"$5\"'","--class",'\"$2\"','\"$3\"','\"$4\"'],"Type":"CUSTOM_JAR","ActionOnFailure":"CONTINUE","Jar":"command-runner.jar","Properties":"","Name":"Spark application"}]' --region "$6"
and after I run it, I get the below error:
ExecuteStreamCommand[id=5b08df5a-1f24-3958-30ca-2e27a6c4becf] Transferring flow file StandardFlowFileRecord[uuid=00f844ee-dbea-42a3-aba3-0edcabfc50a2,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1607082757752-507103, container=default, section=223], offset=29, length=-1],offset=0,name=6414901712887990,size=0] to nonzero status. Executable command /bin/bash ended in an error:
Error parsing parameter '--steps': Invalid JSON:
[{"Args":["spark-submit","--deploy-mode","cluster","--jars","s3://tvsc-lumiq-edl/jars/ojdbc7.jar","--executor-memory","10g","--driver-memory","10g","--conf","spark.hadoop.yarn.timeline-service.enabled=false","--conf","currDate="Fri
Where am I going wrong?
You can use JSONLint to validate your JSON, which makes it easier to see why its wrong.
In your case, you are wrapping the final 3 values in single quotes ' rather than double quotes "
Your steps JSON should look like:
[{
"Args": [
"spark-submit",
"--deploy-mode",
"cluster",
"--jars",
"s3://tvsc-lumiq-edl/jars/ojdbc7.jar",
"--executor-memory",
"10g",
"--driver-memory",
"10g",
"--conf",
"spark.hadoop.yarn.timeline-service.enabled=false",
"--conf",
"currDate='\"$5\"'",
"--class",
"\"$2\"",
"\"$3\"",
"\"$4\""
],
"Type": "CUSTOM_JAR",
"ActionOnFailure": "CONTINUE",
"Jar": "command-runner.jar",
"Properties": "",
"Name": "Spark application"
}]
Specifically, these 3 lines:
"\"$2\"",
"\"$3\"",
"\"$4\""
Instead of the original:
'\"$2\"',
'\"$3\"',
'\"$4\"'

How to pull the right keys and values from jq into an array in bash shell script

I have a json file that is formatted like so:
{
"ServerName1": {
"localip": "192.168.1.1",
"hostname": "server1"
},
"ServerName2": {
"localip": "192.168.1.2",
"hostname": "server2"
},
"ServerName3": {
"localip": "192.168.1.3",
"hostname": "server3"
}
}
And i am trying to write a shell script that uses Dialog to create a menu to run an ssh connection command. I'm parsing with jq, but can't get past the first object level. We have a lot of servers and this will make connecting to them a lot easier. I have the Dialog statement working fine with static data, but we are trying to populate it with a json file with the rest of the data. So i am killing myself trying to figure out how to get just the localip and hostname either into an array to loop into the Dialog command or something that will effectively do the same thing and al I get it it to do so far is spit out
Servername1 = {"localip":"192.168.1.1","hostname":"server1"}
on each line. I'm a shell script newbie but this is messing with sanity now.
This is the jq command that I've been working with so far:
jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" config.json
This is the Dialog command that works well with static data:
callssh(){
clear
ssh $1#$2
}
## Display Menu ##
dialog --clear --title "SSH Relayer"\
--menu "Please choose which server \n\
with which you would like to connect" 15 50 4 \
"Server 1" "192.168.1.1"\
"Server 2" "192.168.1.2"\
"Server 3" "192.168.1.3"\
Exit "Exit to shell" 2>"${INPUT}"
menuitem=$(<"${INPUT}")
case $menuitem in
"Server 1") callssh $sshuser 192.168.1.1;;
"Server 2") callssh $sshuser 192.168.1.2;;
"Server 3") callssh $sshuser 192.168.1.3;;
Exit) clear
echo "Bye!";;
esac
Thanks for any help or pointing in the right direction.
To create a bash array mapping hostnames to ip addresses based on config.json:
declare -A ip_of
# Emit lines of the form:
# hostname localip (without quotation marks)
function hostname_ip {
local json="$1"
jq -r '.[] | "\(.hostname) \(.localip)"' "$json"
}
while read -r hostname ip ; do
ip_of["$hostname"]="$ip"
done < <(hostname_ip config.json)
You can loop through this bash array like so:
for hostname in "${!ip_of[#]}" ; do
echo hostname=$hostname "=>" ${ip_of[$hostname]}
done
For example, assuming the "dialog" presents the hostnames,
you can replace the case statement by:
callssh "$sshuser" "${ip_of[$menuitem]}"

Get Node attributes via knife

I have a requirement where I need to get hostname, memory, cores, storage, packages installed for multiple nodes(~1k).
I have approached the solution by using knife.
$ knife search node 'hostname:HostName1 OR hostname:HostName2 OR hostname:HostName3' -a hostname -a cpu.cores -a memory.total -a rpm -a filesystem.by_device -F j|jq '.'
And the typical output of this command is like;
{
"results": 3,
"rows": [
{
"MyHostName1": {
"hostname": "MyHostName1",
"cpu.cores": 4,
"memory.total": "15645184kB",
"rpm": {
"loger-multipath": [
{
"version": "0.4.9",
"release": "123.el7",
"arch": "x86_64"
}
],
"python": [
{
"version": "7.19.0",
"release": "19.el7",
"arch": "x86_64"
}
]
},
"filesystem.by_device": {
"/apps/logger/root_my-root": {
"kb_size": "8125880",
"kb_used": "2426760",
"kb_available": "5263308",
"percent_used": "32%",
"mount_options": [
"rw",
"discard",
"data=ordered"
],
"uuid": "87ujrf56-6yu6-654r-yu43-uy67yg43ws67",
"mounts": [
"/"
]
}
}
}
}
}
However, there are details which I do not need;
How can we set the display sequence same as that of the attribute list in the command, i.e. hostname then core, memory…
We get the file system names and their corresponding sizes, however, we are getting all the other tag values as well; how can we get just the file system name and the ( something similar to what we get from the df command; e.g. apps/logger/root_vg-apps: kb_size: 3997376 )
The output of the rpm attribute gives us the rpm package name, architecture, version and release information, how can we concatenate the output of multiple attributes in a single line ( something similar to the output when we run the yum list installed command; e.g loger-multipath.x86_64 0.4.9-123.el7 )
EDIT:
After much googling this is the progress:
knife search node 'HostName1 OR hostname:HostName2' -a cpu.cores -a memory.total -a filesystem.by_device -F j|jq '.rows[]|keys[] as $hostName|"\($hostName),\(.[$hostName]|."cpu.cores"),\(.[$hostName]|."memory.total"),\(.[$hostName]|."filesystem.by_device")"'
And the corresponding output
"HostName1,4,15645184kB,{\"/dev/mapper/root_vg-root\":{\"kb_size\":\"8125880\",\"kb_used\":\"2425220\",\"kb_available\":\"5264848\",\"percent_used\":\"32%\",\"total_inodes\":\"524288\",\"inodes_used\":\"88441\",\"inodes_available\":\"435847\",\"inodes_percent_used\":\"17%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"relatime\",\"seclabel\",\"discard\",\"data=ordered\"],\"uuid\":\"rthd-762c-41affff8-8927-065fsee20853c681\",\"mounts\":[\"/\"]},\"devtmpfs\":{\"kb_size\":\"7810756\",\"kb_used\":\"0\",\"kb_available\":\"7810756\",\"percent_used\":\"0%\",\"total_inodes\":\"1952689\",\"inodes_used\":\"403\",\"inodes_available\":\"1952286\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"devtmpfs\",\"mount_options\":[\"rw\",\"nosuid\",\"seclabel\",\"size=7810756k\",\"nr_inodes=1952689\",\"mode=755\"],\"mounts\":[\"/dev\"]},\"tmpfs\":{\"kb_size\":\"1564520\",\"kb_used\":\"0\",\"kb_available\":\"1564520\",\"percent_used\":\"0%\",\"total_inodes\":\"1955648\",\"inodes_used\":\"1\",\"inodes_available\":\"1955647\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"tmpfs\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"size=1564520k\",\"mode=700\",\"uid=627000\",\"gid=161\"],\"mounts\":[\"/dev/shm\",\"/run\",\"/sys/fs/cgroup\",\"/run/user/0\",\"/run/user/627000\"]},\"/dev/sda1\":{\"kb_size\":\"499656\",\"kb_used\":\"212068\",\"kb_available\":\"250892\",\"percent_used\":\"46%\",\"total_inodes\":\"32768\",\"inodes_used\":\"350\",\"inodes_available\":\"32418\",\"inodes_percent_used\":\"2%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"data=ordered\"],\"uuid\":\"857fgg-b2a2-42d8-9db2-dfrferf7544\",\"mounts\":[\"/boot\"]},\"/dev/mapper/root_vg-var\":{\"kb_size\":\"5029504\",\"kb_used\":\"4142128\",\"kb_available\":\"608848\",\"percent_used\":\"88%\",\"total_inodes\":\"327680\",\"inodes_used\":\"6191\",\"inodes_available\":\"321489\",\"inodes_percent_used\":\"2%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"noacl\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"dfef155456-ab4c-48f4-a7a5-5454sfdf\",\"mounts\":[\"/var\"]},\"/dev/mapper/root_vg-var--tmp\":{\"kb_size\":\"1998672\",\"kb_used\":\"6180\",\"kb_available\":\"1871252\",\"percent_used\":\"1%\",\"total_inodes\":\"131072\",\"inodes_used\":\"21\",\"inodes_available\":\"131051\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"vfghhhht542-ea7c-4c8b-9afd-frfgvbbn\",\"mounts\":[\"/var/tmp\"]},\"/dev/mapper/root_vg-apps\":{\"kb_size\":\"3997376\",\"kb_used\":\"495044\",\"kb_available\":\"3276236\",\"percent_used\":\"14%\",\"total_inodes\":\"262144\",\"inodes_used\":\"3203\",\"inodes_available\":\"258941\",\"inodes_percent_used\":\"2%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nodev\",\"relatime\",\"seclabel\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"fvfbvfbv55444-a813-4d9c-a9ac-7d50cfbfe345\",\"mounts\":[\"/apps\"]},\"/dev/mapper/root_vg-kdump\":{\"kb_size\":\"1998672\",\"kb_used\":\"6144\",\"kb_available\":\"1871288\",\"percent_used\":\"1%\",\"total_inodes\":\"131072\",\"inodes_used\":\"11\",\"inodes_available\":\"131061\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"frgrghg55-9673-47f5-aaac-4g4g4g1g1\",\"mounts\":[\"/kdump\"]},\"/dev/mapper/root_vg-home\":{\"kb_size\":\"1998672\",\"kb_used\":\"6544\",\"kb_available\":\"1870888\",\"percent_used\":\"1%\",\"total_inodes\":\"131072\",\"inodes_used\":\"83\",\"inodes_available\":\"130989\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"frbj4874-fe4d-4f82-ad86-41554ffv\",\"mounts\":[\"/home\"]},\"/dev/mapper/root_vg-tmp\":{\"kb_size\":\"1998672\",\"kb_used\":\"7916\",\"kb_available\":\"1869516\",\"percent_used\":\"1%\",\"total_inodes\":\"131072\",\"inodes_used\":\"51\",\"inodes_available\":\"131021\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"a995fd05-90a8-46a8-a192-0a02f68e476a\",\"mounts\":[\"/tmp\"]},\"/dev/mapper/root_vg-gcis2\":{\"kb_size\":\"20511312\",\"kb_used\":\"3191304\",\"kb_available\":\"16255048\",\"percent_used\":\"17%\",\"total_inodes\":\"1310720\",\"inodes_used\":\"33967\",\"inodes_available\":\"1276753\",\"inodes_percent_used\":\"3%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"relatime\",\"seclabel\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"fb4bb78a-7f33-47f1-87a6-dcbe50bc6349\",\"mounts\":[\"/apps/gcis2\"]},\"sysfs\":{\"fs_type\":\"sysfs\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\",\"seclabel\"],\"mounts\":[\"/sys\"]},\"proc\":{\"fs_type\":\"proc\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\"],\"mounts\":[\"/proc\"]},\"securityfs\":{\"fs_type\":\"securityfs\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\"],\"mounts\":[\"/sys/kernel/security\"]},\"devpts\":{\"fs_type\":\"devpts\",\"mount_options\":[\"rw\",\"nosuid\",\"noexec\",\"relatime\",\"seclabel\",\"gid=5\",\"mode=620\",\"ptmxmode=000\"],\"mounts\":[\"/dev/pts\"]},\"cgroup\":{\"fs_type\":\"cgroup\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\",\"seclabel\",\"net_prio\",\"net_cls\"],\"mounts\":[\"/sys/fs/cgroup/systemd\",\"/sys/fs/cgroup/perf_event\",\"/sys/fs/cgroup/blkio\",\"/sys/fs/cgroup/freezer\",\"/sys/fs/cgroup/cpu,cpuacct\",\"/sys/fs/cgroup/memory\",\"/sys/fs/cgroup/pids\",\"/sys/fs/cgroup/hugetlb\",\"/sys/fs/cgroup/cpuset\",\"/sys/fs/cgroup/devices\",\"/sys/fs/cgroup/net_cls,net_prio\"]},\"pstore\":{\"fs_type\":\"pstore\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\"],\"mounts\":[\"/sys/fs/pstore\"]},\"configfs\":{\"fs_type\":\"configfs\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/sys/kernel/config\"]},\"selinuxfs\":{\"fs_type\":\"selinuxfs\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/sys/fs/selinux\"]},\"systemd-1\":{\"fs_type\":\"autofs\",\"mount_options\":[\"rw\",\"relatime\",\"fd=30\",\"pgrp=1\",\"timeout=0\",\"minproto=5\",\"maxproto=5\",\"direct\",\"pipe_ino=16127\"],\"mounts\":[\"/proc/sys/fs/binfmt_misc\"]},\"debugfs\":{\"fs_type\":\"debugfs\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/sys/kernel/debug\"]},\"hugetlbfs\":{\"fs_type\":\"hugetlbfs\",\"mount_options\":[\"rw\",\"relatime\",\"seclabel\"],\"mounts\":[\"/dev/hugepages\"]},\"mqueue\":{\"fs_type\":\"mqueue\",\"mount_options\":[\"rw\",\"relatime\",\"seclabel\"],\"mounts\":[\"/dev/mqueue\"]},\"binfmt_misc\":{\"fs_type\":\"binfmt_misc\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/proc/sys/fs/binfmt_misc\"]},\"fusectl\":{\"fs_type\":\"fusectl\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/sys/fs/fuse/connections\"]},\"/dev/fd0\":{},\"/dev/sda\":{},\"/dev/sda2\":{\"fs_type\":\"LVM2_member\",\"uuid\":\"zrtcBv-4y6D-LG2a-Wt6M-h18d-K0BQ-zyl11h\"},\"/dev/mapper/root_vg-swap\":{\"fs_type\":\"swap\",\"uuid\":\"vfbvejgbg5456454-93a1-4a67-b33f-gtrbbbn\"},\"/dev/mapper/root_vg-pool0_tmeta\":{},\"/dev/mapper/root_vg-pool0-tpool\":{},\"/dev/mapper/root_vg-pool0\":{},\"/dev/mapper/root_vg-pool0_tdata\":{},\"rootfs\":{\"fs_type\":\"rootfs\",\"mount_options\":[\"rw\"],\"mounts\":[\"/\"]}}"
"HostName2,4,15645184kB,{\"/dev/mapper/root_vg-root\":{\"kb_size\":\"8125880\",\"kb_used\":\"2425220\",\"kb_available\":\"5264848\",\"percent_used\":\"32%\",\"total_inodes\":\"524288\",\"inodes_used\":\"88441\",\"inodes_available\":\"435847\",\"inodes_percent_used\":\"17%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"relatime\",\"seclabel\",\"discard\",\"data=ordered\"],\"uuid\":\"rthd-762c-41affff8-8927-065fsee20853c681\",\"mounts\":[\"/\"]},\"devtmpfs\":{\"kb_size\":\"7810756\",\"kb_used\":\"0\",\"kb_available\":\"7810756\",\"percent_used\":\"0%\",\"total_inodes\":\"1952689\",\"inodes_used\":\"403\",\"inodes_available\":\"1952286\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"devtmpfs\",\"mount_options\":[\"rw\",\"nosuid\",\"seclabel\",\"size=7810756k\",\"nr_inodes=1952689\",\"mode=755\"],\"mounts\":[\"/dev\"]},\"tmpfs\":{\"kb_size\":\"1564520\",\"kb_used\":\"0\",\"kb_available\":\"1564520\",\"percent_used\":\"0%\",\"total_inodes\":\"1955648\",\"inodes_used\":\"1\",\"inodes_available\":\"1955647\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"tmpfs\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"size=1564520k\",\"mode=700\",\"uid=627000\",\"gid=161\"],\"mounts\":[\"/dev/shm\",\"/run\",\"/sys/fs/cgroup\",\"/run/user/0\",\"/run/user/627000\"]},\"/dev/sda1\":{\"kb_size\":\"499656\",\"kb_used\":\"212068\",\"kb_available\":\"250892\",\"percent_used\":\"46%\",\"total_inodes\":\"32768\",\"inodes_used\":\"350\",\"inodes_available\":\"32418\",\"inodes_percent_used\":\"2%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"data=ordered\"],\"uuid\":\"857fgg-b2a2-42d8-9db2-dfrferf7544\",\"mounts\":[\"/boot\"]},\"/dev/mapper/root_vg-var\":{\"kb_size\":\"5029504\",\"kb_used\":\"4142128\",\"kb_available\":\"608848\",\"percent_used\":\"88%\",\"total_inodes\":\"327680\",\"inodes_used\":\"6191\",\"inodes_available\":\"321489\",\"inodes_percent_used\":\"2%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"noacl\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"dfef155456-ab4c-48f4-a7a5-5454sfdf\",\"mounts\":[\"/var\"]},\"/dev/mapper/root_vg-var--tmp\":{\"kb_size\":\"1998672\",\"kb_used\":\"6180\",\"kb_available\":\"1871252\",\"percent_used\":\"1%\",\"total_inodes\":\"131072\",\"inodes_used\":\"21\",\"inodes_available\":\"131051\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"vfghhhht542-ea7c-4c8b-9afd-frfgvbbn\",\"mounts\":[\"/var/tmp\"]},\"/dev/mapper/root_vg-apps\":{\"kb_size\":\"3997376\",\"kb_used\":\"495044\",\"kb_available\":\"3276236\",\"percent_used\":\"14%\",\"total_inodes\":\"262144\",\"inodes_used\":\"3203\",\"inodes_available\":\"258941\",\"inodes_percent_used\":\"2%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nodev\",\"relatime\",\"seclabel\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"fvfbvfbv55444-a813-4d9c-a9ac-7d50cfbfe345\",\"mounts\":[\"/apps\"]},\"/dev/mapper/root_vg-kdump\":{\"kb_size\":\"1998672\",\"kb_used\":\"6144\",\"kb_available\":\"1871288\",\"percent_used\":\"1%\",\"total_inodes\":\"131072\",\"inodes_used\":\"11\",\"inodes_available\":\"131061\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"frgrghg55-9673-47f5-aaac-4g4g4g1g1\",\"mounts\":[\"/kdump\"]},\"/dev/mapper/root_vg-home\":{\"kb_size\":\"1998672\",\"kb_used\":\"6544\",\"kb_available\":\"1870888\",\"percent_used\":\"1%\",\"total_inodes\":\"131072\",\"inodes_used\":\"83\",\"inodes_available\":\"130989\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"frbj4874-fe4d-4f82-ad86-41554ffv\",\"mounts\":[\"/home\"]},\"/dev/mapper/root_vg-tmp\":{\"kb_size\":\"1998672\",\"kb_used\":\"7916\",\"kb_available\":\"1869516\",\"percent_used\":\"1%\",\"total_inodes\":\"131072\",\"inodes_used\":\"51\",\"inodes_available\":\"131021\",\"inodes_percent_used\":\"1%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"relatime\",\"seclabel\",\"discard\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"a995fd05-90a8-46a8-a192-0a02f68e476a\",\"mounts\":[\"/tmp\"]},\"/dev/mapper/root_vg-gcis2\":{\"kb_size\":\"20511312\",\"kb_used\":\"3191304\",\"kb_available\":\"16255048\",\"percent_used\":\"17%\",\"total_inodes\":\"1310720\",\"inodes_used\":\"33967\",\"inodes_available\":\"1276753\",\"inodes_percent_used\":\"3%\",\"fs_type\":\"ext4\",\"mount_options\":[\"rw\",\"relatime\",\"seclabel\",\"stripe=16\",\"data=ordered\"],\"uuid\":\"fb4bb78a-7f33-47f1-87a6-dcbe50bc6349\",\"mounts\":[\"/apps/gcis2\"]},\"sysfs\":{\"fs_type\":\"sysfs\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\",\"seclabel\"],\"mounts\":[\"/sys\"]},\"proc\":{\"fs_type\":\"proc\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\"],\"mounts\":[\"/proc\"]},\"securityfs\":{\"fs_type\":\"securityfs\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\"],\"mounts\":[\"/sys/kernel/security\"]},\"devpts\":{\"fs_type\":\"devpts\",\"mount_options\":[\"rw\",\"nosuid\",\"noexec\",\"relatime\",\"seclabel\",\"gid=5\",\"mode=620\",\"ptmxmode=000\"],\"mounts\":[\"/dev/pts\"]},\"cgroup\":{\"fs_type\":\"cgroup\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\",\"seclabel\",\"net_prio\",\"net_cls\"],\"mounts\":[\"/sys/fs/cgroup/systemd\",\"/sys/fs/cgroup/perf_event\",\"/sys/fs/cgroup/blkio\",\"/sys/fs/cgroup/freezer\",\"/sys/fs/cgroup/cpu,cpuacct\",\"/sys/fs/cgroup/memory\",\"/sys/fs/cgroup/pids\",\"/sys/fs/cgroup/hugetlb\",\"/sys/fs/cgroup/cpuset\",\"/sys/fs/cgroup/devices\",\"/sys/fs/cgroup/net_cls,net_prio\"]},\"pstore\":{\"fs_type\":\"pstore\",\"mount_options\":[\"rw\",\"nosuid\",\"nodev\",\"noexec\",\"relatime\"],\"mounts\":[\"/sys/fs/pstore\"]},\"configfs\":{\"fs_type\":\"configfs\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/sys/kernel/config\"]},\"selinuxfs\":{\"fs_type\":\"selinuxfs\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/sys/fs/selinux\"]},\"systemd-1\":{\"fs_type\":\"autofs\",\"mount_options\":[\"rw\",\"relatime\",\"fd=30\",\"pgrp=1\",\"timeout=0\",\"minproto=5\",\"maxproto=5\",\"direct\",\"pipe_ino=16127\"],\"mounts\":[\"/proc/sys/fs/binfmt_misc\"]},\"debugfs\":{\"fs_type\":\"debugfs\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/sys/kernel/debug\"]},\"hugetlbfs\":{\"fs_type\":\"hugetlbfs\",\"mount_options\":[\"rw\",\"relatime\",\"seclabel\"],\"mounts\":[\"/dev/hugepages\"]},\"mqueue\":{\"fs_type\":\"mqueue\",\"mount_options\":[\"rw\",\"relatime\",\"seclabel\"],\"mounts\":[\"/dev/mqueue\"]},\"binfmt_misc\":{\"fs_type\":\"binfmt_misc\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/proc/sys/fs/binfmt_misc\"]},\"fusectl\":{\"fs_type\":\"fusectl\",\"mount_options\":[\"rw\",\"relatime\"],\"mounts\":[\"/sys/fs/fuse/connections\"]},\"/dev/fd0\":{},\"/dev/sda\":{},\"/dev/sda2\":{\"fs_type\":\"LVM2_member\",\"uuid\":\"zrtcBv-4y6D-LG2a-Wt6M-h18d-K0BQ-zyl11h\"},\"/dev/mapper/root_vg-swap\":{\"fs_type\":\"swap\",\"uuid\":\"vfbvejgbg5456454-93a1-4a67-b33f-gtrbbbn\"},\"/dev/mapper/root_vg-pool0_tmeta\":{},\"/dev/mapper/root_vg-pool0-tpool\":{},\"/dev/mapper/root_vg-pool0\":{},\"/dev/mapper/root_vg-pool0_tdata\":{},\"rootfs\":{\"fs_type\":\"rootfs\",\"mount_options\":[\"rw\"],\"mounts\":[\"/\"]}}"
I know this seems a bit messy; any help is welcome
looking at the json provided in the question, it seems that your query does not use the right values for the key that you specify -- namely, hostname -- you specified hostname:HostName1 where it feels it needs to be hostname:MyHostName1.
also, you can drop the node argument from the search command and specify it in the query, such as node:my.node.name.

Store JSON directly in bash script with variables?

I'm going to preface by saying that "no, find a different way to do it" is an acceptable answer here.
Is there a reliable way to store a short bit of JSON in a bash variable for use in a AWS CLI command running from the same script?
I'll be running a job from Jenkins that's updating an AWS Route53 record, which requires UPSERTing a JSON file with the change in records. Because it's running from Jenkins, there's no local storage where I can keep this file, and I'd really like to avoid needing to do a git checkout every time this project will run (which will be once an hour).
Ideally, storing the data in a variable ($foo) and calling it as part of the change-resource-record-sets command would be most convenient given the Jenkins setup, but I'm unfamiliar with exactly how to quote/store JSON inside bash - can it be done safely?
The specific JSON in this case is the following;
{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"$bar","EvaluateTargetHealth":false}}}]}
As an added complication the DNSName value - $bar - needs to be expanded.
You could use a here-doc:
foo=$(cat <<EOF
{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"$bar","EvaluateTargetHealth":false}}}]}
EOF
)
By leaving EOF in the first line unquoted, the contents of the here-doc will be subject to parameter expansion, so your $bar expands to whatever you put in there.
If you can have linebreaks in your JSON, you can make it a little more readable:
foo=$(cat <<EOF
{
"Comment": "Update DNSName.",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "alex.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "######",
"DNSName": "$bar",
"EvaluateTargetHealth": false
}
}
}
]
}
EOF
)
or even (first indent on each line must be a tab)
foo=$(cat <<-EOF
{
"Comment": "Update DNSName.",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "alex.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "######",
"DNSName": "$bar",
"EvaluateTargetHealth": false
}
}
}
]
}
EOF
)
and to show how that is stored, including quoting (assuming that bar=baz):
$ declare -p foo
declare -- foo="{
\"Comment\": \"Update DNSName.\",
\"Changes\": [
{
\"Action\": \"UPSERT\",
\"ResourceRecordSet\": {
\"Name\": \"alex.\",
\"Type\": \"A\",
\"AliasTarget\": {
\"HostedZoneId\": \"######\",
\"DNSName\": \"baz\",
\"EvaluateTargetHealth\": false
}
}
}
]
}"
Because this expands some shell metacharacters, you could run into trouble if your JSON contains something like `, so alternatively, you could assign directly, but be careful about quoting around $bar:
foo='{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"'"$bar"'","EvaluateTargetHealth":false}}}]}'
Notice the quoting for $bar: it's
"'"$bar"'"
│││ │││
│││ ││└ literal double quote
│││ │└ opening syntactical single quote
│││ └ closing syntactical double quote
││└ opening syntactical double quote
│└ closing syntactical single quote
└ literal double quote
It can be stored safely; generating it is a different matter, since the contents of $bar may need to be encoded. Let a tool like jq handle creating the JSON.
var=$(jq -n --arg b "$bar" '{
Comment: "Update DNSName.",
Changes: [
{
Action: "UPSERT",
ResourceRecordSet: {
Name: "alex.",
Type: "A",
AliasTarget: {
HostedZoneId: "######",
DNSName: $b,
EvaluateTargetHealth: false
}
}
}
]
}')