I'm doing this:
{
"type": "powershell",
"script": "myscript.ps1 -param1 {{user `aaaa`}} -param2 {{user `bbbb`}} -param3 {{user `cccc`}}"
}
I'm adding more params all the time. How do I break this across multiple lines?
I don't want to use a list of env vars because I want the script to explicitly define its params.
I don't want to use hcl because that's still in beta.
Ran into this myself. Packer does support multi-line here documents. And they are serious about the "The lines of the string and the end marker must not be indented."
So a solution (in HCL) would look like:
provisioner "shell-local" {
inline = [ <<EOF
myscript.ps1 \
-param1 {{user `aaaa`}} \
-param2 {{user `bbbb`}} \
-param3 {{user `cccc`}}
EOF
]
}
I'd imagine something similar should work in powershell.
Related
Using jq, is it possible to replace the value of each parameter in the sample JSON with the value of the variable that is the initial value?
In my scenario, Azure DevOps does not carryout any kind of variable substitution on the JSON file, so I need to do it manually. So for example, say $SUBSCRIPTION_ID is set to abc-123, I'd like to use jq to update the JSON file.
I can pull out the values using .parameters[].value, but I can't seem to find a way of setting each individual value.
The main challenge here is that the solution should be reusable, and different JSON files will have different parameters, so I don't think I can use --argjson.
Example
Original JSON
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/parametersTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"subscriptionId": {
"value": "$SUBSCRIPTION_ID"
},
"topicName": {
"value": "$TOPIC_NAME"
}
}
}
Variables
SUBSCRIPTION_ID="abc-123"
TOPIC_NAME="SomeTopic"
Desired JSON
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/parametersTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"subscriptionId": {
"value": "abc-123"
},
"topicName": {
"value": "SomeTopic"
}
}
}
Export those variables so that you can access them from within jq.
export SUBSCRIPTION_ID TOPIC_NAME
jq '.parameters[].value |= (env[.[1:]] // .)' file
//. part is for leaving variables absent in the environment as is, you can drop it if not necessary
Use --argjson; essentially, you are just going to ignore the attempt at parameterizing the JSON and simply replace the values unconditionally.
jq --argjson x "$SUBSCRIPTION_ID" \
--argjson y "$TOPIC_NAME" \
'.parameters.subscriptionId.value = $x; .parameters.topicName.value = $y' \
config.json
Here is a "data-driven" approach based on the contents of the schema and the available environment variables:
export SUBSCRIPTION_ID="abc-123"
export TOPIC_NAME="SomeTopic"
< schema.json jq '.parameters
|= map_values(if .value | (startswith("$") and env[.[1:]])
then .value |= env[.[1:]] else . end)'
Notice that none of the template names appear in the jq program.
If your shell supports it, you could avoid the "export" commands by prefacing the jq command with the variable assignments along the lines of:
SUBSCRIPTION_ID="abc-123" TOPIC_NAME="SomeTopic" jq -f program.jq schema.json
Caveat
Using environment variables to pass in the parameter values may not be such a great idea. Two alternatives would be to provide the name-value pairs in a text file or as a JSON object. See also Using jq as a template engine
[
{
"name":"sandboxserver.tar.gz.part-aa",
"hash":"010d126f8ccf199f3cd5f468a90d5ae1",
"bytes":4294967296,
"last_modified":"2018-10-10T01:32:00.069000",
"content_type":"binary/octet-stream"
},
{
"name":"sandboxserver.tar.gz.part-ab",
"hash":"49a6f22068228f51488559c096aa06ce",
"bytes":397973601,
"last_modified":"2018-10-10T01:32:22.395000",
"content_type":"binary/octet-stream"
},
{
"name":"sandboxserver.tar.gz.part-ac",
"hash":"2c5e845f46357e203214592332774f4c",
"bytes":5179281858,
"last_modified":"2018-10-11T08:20:11.566000",
"content_type":"binary/octet-stream"
}
]
I am getting above JSON as response while listing the objects in cloud object storage using curl -l -X GET. How can I get the object "name" assigned to an array while looping through all the objects.
for example
array[1]="sandboxserver.tar.gz.part- aa"
array[2]="sandboxserver.tar.gz.part- ab"
array[3]="sandboxserver.tar.gz.part- ac"
You can use jq.
jq is a powerful tool that lets you read, filter, and write JSON in bash.
You might need to install it first.
Try this:
I've pasted your json into a file:
~$ cat n1.json
[
{
"name":"sandboxserver.tar.gz.part-aa",
"hash":"010d126f8ccf199f3cd5f468a90d5ae1",
"bytes":4294967296,
"last_modified":"2018-10-10T01:32:00.069000",
"content_type":"binary/octet-stream"
},
{
"name":"sandboxserver.tar.gz.part-ab",
"hash":"49a6f22068228f51488559c096aa06ce",
"bytes":397973601,
"last_modified":"2018-10-10T01:32:22.395000",
"content_type":"binary/octet-stream"
},
{
"name":"sandboxserver.tar.gz.part-ac",
"hash":"2c5e845f46357e203214592332774f4c",
"bytes":5179281858,
"last_modified":"2018-10-11T08:20:11.566000",
"content_type":"binary/octet-stream"
}
]
And then used jq to find the names:
~$ jq -r '.[].name' n1.json
sandboxserver.tar.gz.part-aa
sandboxserver.tar.gz.part-ab
sandboxserver.tar.gz.part-ac
If you don't want to depend on external utility like jq, use can use python + bash combo do the trick.
response="$(cat data.json)"
declare -a array
array=($(python -c "import json,sys; data=[arr['name'] for arr in json.loads(sys.argv[1])]; print('\n'.join(data));" "$response"))
echo "${array[#]}"
Advice: Writing embedded python code may soon become unreadable so you may want to put the python code in a separate script and run the script.
I'm writing a Bash function to get a portion of a JSON object. The API for the function is:
GetSubobject()
{
local Filter="$1" # Filter is of the form .<key>.<key> ... .<key>
local File="$2" # File is the JSON to get the subobject
# Code to get subobject using jq
# ...
}
To illustrate what I mean by a subobject, consider the Bash function call:
GetSubobject .b.x.y example.json
where the file example.json contains:
{
"a": { "p": 1, "q": 2 },
"b":
{
"x":
{
"y": { "j": true, "k": [1,2,3] },
"z": [4,5,6]
}
}
}
The result from the function call would be emitted to stdout:
{
"y": {
"j": true,
"k": [
1,
2,
3
]
}
}
Note that the code jq -r "$Filter" "$File" would not give the desired answer. It would give:
{ "j": true, "k": [1,2,3] }
Please note that the answer I'm looking for needs to be something I can use in the Bash function API above. So, the answer should use the Filter and File variables as show above and not be specific to the example above.
I have come up with a solution; however, it relies on Bash to do part of the job. I am hoping that the solution can be pure jq without reliance on Bash processing.
#!/bin/bash
GetSubobject()
{
local Filter="$1"
local File="$2"
# General case: separate:
# .<key1>.<key2> ... .<keyN-1>.<keyN>
# into:
# Prefix=.<key1>.<key2> ... .<keyN-1>
# Suffix=<keyN>
local Suffix="${Filter##*.}"
local Prefix="${Filter%.$Suffix}"
# Edge case: where Filter = .<key>
# Set:
# Prefix=.
# Suffix=<key>
if [[ -z $Prefix ]]; then
Prefix='.'
Suffix="${Filter#.}"
fi
jq -r "$Prefix|to_entries|map(select(.key==\"$Suffix\"))|from_entries" "$File"
}
GetSubobject "$#"
How would I complete the above Bash function using jq to obtain the desired result, hopefully in a less brute-force way that takes advantage of jq's capabilities without having to do pre-processing in Bash?
Somewhat further simplifying the jq part but with the same general constraints as JawguyChooser's answer, how about the much simpler Bash function
GetSubject () {
local newroot=${1##*.}
jq -r "{$newroot: $1}" "$2"
}
I may be overlooking some nuances of your more-complex Bash processing, but this seems to work for the example you provided.
If I understand what you're trying to do correctly, it doesn't seem possible to me to do it "pure jq" having read the docs (and being a regular jq user myself). The closest I could come to helping here was to simplify the jq part itself:
jq -r "$Prefix| { $Suffix }" "$File"
This has the same behavior as your example (on this limited set of cases):
GetSubobject '.b.x.y' example.json
{
"y": {
"j": true,
"k": [
1,
2,
3
]
}
}
This is really a case of metaprogramming, you want to programmatically operate on a jq program. Well, it makes sense (to me) that jq takes its program as input but doesn't allow you to alter the program itself. bash seems like an appropriate choice for doing the metaprogramming here: to convert a jq program into another one and then run jq using that.
If the goal is to do as little as possible in bash, then maybe the following bash function will fill the bill:
function GetSubobject {
local Filter="$1" # Filter is of the form .<key>.<key> ... .<key>
local File="$2" # File is the JSON to get the subobject
jq '(null|path('"$Filter"')) as $path
| {($path[-1]): '"$Filter"'}' "$File"
}
An alternative would be to pass $Filter in as a string (e.g. --arg filter "$Filter"), have jq do the parsing, and then use getpath.
It would of course be simplest if GetSubobject could be called with the path separated from the field of interest, like this:
GetSubobject .b.x y filename
I'm going to preface by saying that "no, find a different way to do it" is an acceptable answer here.
Is there a reliable way to store a short bit of JSON in a bash variable for use in a AWS CLI command running from the same script?
I'll be running a job from Jenkins that's updating an AWS Route53 record, which requires UPSERTing a JSON file with the change in records. Because it's running from Jenkins, there's no local storage where I can keep this file, and I'd really like to avoid needing to do a git checkout every time this project will run (which will be once an hour).
Ideally, storing the data in a variable ($foo) and calling it as part of the change-resource-record-sets command would be most convenient given the Jenkins setup, but I'm unfamiliar with exactly how to quote/store JSON inside bash - can it be done safely?
The specific JSON in this case is the following;
{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"$bar","EvaluateTargetHealth":false}}}]}
As an added complication the DNSName value - $bar - needs to be expanded.
You could use a here-doc:
foo=$(cat <<EOF
{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"$bar","EvaluateTargetHealth":false}}}]}
EOF
)
By leaving EOF in the first line unquoted, the contents of the here-doc will be subject to parameter expansion, so your $bar expands to whatever you put in there.
If you can have linebreaks in your JSON, you can make it a little more readable:
foo=$(cat <<EOF
{
"Comment": "Update DNSName.",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "alex.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "######",
"DNSName": "$bar",
"EvaluateTargetHealth": false
}
}
}
]
}
EOF
)
or even (first indent on each line must be a tab)
foo=$(cat <<-EOF
{
"Comment": "Update DNSName.",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "alex.",
"Type": "A",
"AliasTarget": {
"HostedZoneId": "######",
"DNSName": "$bar",
"EvaluateTargetHealth": false
}
}
}
]
}
EOF
)
and to show how that is stored, including quoting (assuming that bar=baz):
$ declare -p foo
declare -- foo="{
\"Comment\": \"Update DNSName.\",
\"Changes\": [
{
\"Action\": \"UPSERT\",
\"ResourceRecordSet\": {
\"Name\": \"alex.\",
\"Type\": \"A\",
\"AliasTarget\": {
\"HostedZoneId\": \"######\",
\"DNSName\": \"baz\",
\"EvaluateTargetHealth\": false
}
}
}
]
}"
Because this expands some shell metacharacters, you could run into trouble if your JSON contains something like `, so alternatively, you could assign directly, but be careful about quoting around $bar:
foo='{"Comment":"Update DNSName.","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"alex.","Type":"A","AliasTarget":{"HostedZoneId":"######","DNSName":"'"$bar"'","EvaluateTargetHealth":false}}}]}'
Notice the quoting for $bar: it's
"'"$bar"'"
│││ │││
│││ ││└ literal double quote
│││ │└ opening syntactical single quote
│││ └ closing syntactical double quote
││└ opening syntactical double quote
│└ closing syntactical single quote
└ literal double quote
It can be stored safely; generating it is a different matter, since the contents of $bar may need to be encoded. Let a tool like jq handle creating the JSON.
var=$(jq -n --arg b "$bar" '{
Comment: "Update DNSName.",
Changes: [
{
Action: "UPSERT",
ResourceRecordSet: {
Name: "alex.",
Type: "A",
AliasTarget: {
HostedZoneId: "######",
DNSName: $b,
EvaluateTargetHealth: false
}
}
}
]
}')
I have a JSON file as below.
{
"card":{
"cardName":"10AN10G",
"portSignalRates":[
"10AN10G-1-OTU2",
"10AN10G-1-OTU2E",
"10AN10G-1-TENGIGE",
"10AN10G-1-STM64"
],
"listOfPort":{
"10AN10G-1-OTU2":{
"portAid":"10AN10G-1-OTU2",
"signalType":"OTU2",
"tabNames":[
"PortDetails"
],
"requestType":{
"PortDetails":"PTP"
},
"paramDetailsMap":{
"PortDetails":[
{
"type":"dijit.form.TextBox",
"name":"signalType",
"title":"Signal Rate",
"id":"",
"options":[
],
"label":"",
"value":"OTU2",
"checked":"",
"enabled":"false",
"selected":""
},
{
"type":"dijit.form.TextBox",
"name":"userLabel",
"title":"Description",
"id":"",
"options":[
],
"label":"",
"value":"",
"checked":"",
"enabled":"true",
"selected":""
},
{
"type":"dijit.form.Select",
"name":"Frequency",
"title":"Transmit Frequency"
}
]
}
}
}
}
}
I require the output to be:
signalType:"Signal Rate",
userLabel:"Description",
Frequency:"Transmit Frequency",.. ,.....
I tried with:
grep -oP '(?<=\"title\":\")[^"]*' file > outfile
but this just splits the value of title and returns.
Can I use perl to access elements of the JSON data that I want?
I suggest to use decode_json instead of a regex. First, install the JSON module from CPAN:
sudo perl -MCPAN -e 'install JSON'
Alternatively you can use apt-get on Ubuntu:
sudo apt-get install libjson-pp-perl
Once it is installed, you can use this code:
my $json = '{... your json string ...}';
my $decoded = decode_json($json);
$decoded->{'card'}{'listOfPort'}{'10AN10G-1-OTU2'}{'signalType'}
You can find more details in this article.
Yes you can use the JSON perl module. Of course it meeds to be installed via cpan, cpanm or your system's packaging system. Parse the JSON into a hash and then use that in the normal way from Perl. Here's a quick example:
use JSON;
use IO::All;
use strict;
use warnings;
my $data < io '/tmp/data.json';
my $j = decode_json($data);
use DDP; # for quick debug printing
p $j->{card}{listOfPort}{"10AN10G-1-OTU2"}{paramDetailsMap}{PortDetails}[0]{title}
"Signal Rate"
You might want to use some deep diving techniques to get at the inner values more easily/programmatically (take a look at the Data::Diver module for that) - my example is meant to show only that it is possible and a bit of the mechanics of mapping JSON into a perl hash with the excellent JSON module. The documentation has lots of useful examples.
Perl on the command line:
perl -pe 's/"name":"([^"]+)","title":/"$1":/g' file > outfile
Output (pertinent part):
{"card": ... "signalType":"Signal Rate", ... "userLabel":"Description", ... "Frequency":"Transmit Frequency" ... }