extra backward slash along with quotes on jsonencode a json array object - json

In terraform,I am trying to make a PUT call via curl command using null_resource and executing the command in local_exec provisioner. The body of the request is json array.
The command expects the data to be in string format. I have the data as tuple in locals. I am using jsonencode to serialize the tupple to a string and pass to curl. But on jsonencode, the string formed has an additional \ prefixed before each " in json string.
eg. expected json string is:
"[{"key-1" : "value-1", "key-2": "value-2"}]"
But after jsonencode, the formatted string is as follows:
"[{\"key-1\" : \"value-1\", \"key-2\": \"value-2\"}]"
additional backslash is added along with quotes.
Because of the malformed request body, the API returns Bad Request response.
How can we correctly serialize the tupple/list to correct json string in Terraform and pass to curl command?
This is my null_resource:
resource "null_resource" "call_API" {
for_each = { for x in local.array: x.name => {
name = x.name
vars = jsonencode(x.vars)
} }
provisioner "local-exec" {
command = <<EOF
curl -X PUT ${var.url} -d ${each.value.vars} -H 'Content-Type:application/json' -H 'Authorization:${local.bearer_token}'
EOF
}

I think I understand that you want to pass this variable
"[{"key-1" : "value-1", "key-2": "value-2"}]"
You tried to use EOF?
You can do something like that
variable "body" {
type = "string"
}
variable = <<EOF
"[{"key-1" : "value-1", "key-2": "value-2"}]"
EOF
Or with EOT, honestly I don't remember which one work
EDIT:
If the json is not static and you receive it from other module or others, you can try to expand the variable inside the EOFs
body = <<EOF
${var.json}
EOF

Related

Jenkins : Extract data from Jira curl

I use a curl to get the status of a Jira issue :
result = sh(returnStdout: true, script: """curl -D- -k -u $JIRA_LOGIN:$JIRA_PWD -X GET -H "Content-Type: application/json" $url""")
This returns a String like :
{"expand":"renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations","id":"*****","self":"*****","key":"*****","fields":{"status":{"self":"*****","description":"Issues in development","iconUrl":"****","name":"DEVELOPMENT","id":"10400","statusCategory":{"self":"****","id":4,"key":"indeterminate","colorName":"yellow","name":"In Progress"}}}}
How I can extract the data "DEVELOPMENT" in this string ? I try to parse it in JSON but it's not working :
status_json = new JsonSlurper().parse(result)
with the error "java.io.NotSerializableException: groovy.json.JsonSlurper"
You can check out the JsonSlurper's documentation:
Jira holds all issue fields under the "fields" key in JSON. So, after you get the JSON, you just need to go to the fields.status.name in order to get "DEVELOPMENT" text.
Here's a simple approach:
def jsonSlurper = new JsonSlurper()
def issueObj = jsonSlurper.parseText('...') // Your Jira response will be here
issueObj.fields.status.name // It will give you the status' name.

Error while Passing Json string in scala using Curl

I am trying to post a json string using curl in scala. My curl command works fine if executed from linux box but throes an error(("message": "Must provide query string.",) always from scala.
my working curl command in linux:
curl http://laptpad1811:5000/graphql -H "Content-Type: application/json"
-X POST -d '{"query":"mutation
CreateFileReceivedEvent($createFileReceivedEventInput:
CreateFleReceivedEventInput!) { createFileReceivedEvent(input:
$createFileReceivedEventInput) { clientMutationId }}","variables":
{"createFileReceivedEventInput":
{"clientMutationId":"Test","fileReceivedEvent":{"file":
{"fileTrackingId":"83a86c44-66a5-4de0-9b7f-
c6995877279d","name":"textfile_2017-08-21T15:58:45Z","fileType":
{"code":"textfile"}},"eventTimestamp":"2017-08-
21T15:59:30Z"}}},"operationName":"CreateFileReceivedEvent"}'
My scala code:
step1: copying the entire json string(pay load) to txt file
'{"query":"mutation CreateFileReceivedEvent($createFileReceivedEventInput:
CreateFleReceivedEventInput!) { createFileReceivedEvent(input:
$createFileReceivedEventInput) { clientMutationId }}","variables":
{"createFileReceivedEventInput":
{"clientMutationId":"Test","fileReceivedEvent":{"file":
{"fileTrackingId":"83a86c44-66a5-4de0-9b7f-
c6995877279d","name":"textfile_2017-08-21T15:58:45Z","fileType":
{"code":"textfile"}},"eventTimestamp":"2017-08-
21T15:59:30Z"}}},"operationName":"CreateFileReceivedEvent"}'
step2:
val data=fromFile("/usr/test/data.txt").getLines.mkString
step3:
val cmd = Seq("curl", "http://laptpad1811:5000/graphql", "-H",
"'Content-Type:application/json'" ,"-X", "POST", "-d" , data)
step4:
cmd.!!
I get the below error
String =
"{
"errors": [
{
"message": "Must provide query string.",
"stack": "BadRequestError: Must provide query string.\n
I have tried to change " to ' and mutiple combinations of the json string but I always get the same error.
I suspect that your issue is that sys.process doesn't pass commands through the shell (e.g. bash), so quotes that are necessary in the shell become unnecessary in Scala (and get passed through to the command which in the case of Unix-style utilities will probably result in unexpected behavior).
So try:
val cmd = Seq("curl", "http://laptpad1811:5000/graphql", "-H", "Content-Type: application/json", "-X", "POST", "-d", data)
Likewise remove the single quote wrapping from your text file.
I would however, counsel against spawning a curl in Scala and advise using one of the existing http client libraries (I personally like Gigahorse).

Parsing JSON from shell script using JSON.sh

I'm working on parsing JSON data using JSON.sh. And I wanted to read data from json file (test.json) whose content will be something like,
{
"/home/ukrishnan/projects/test.yml": {
"LOG_DRIVER": "syslog",
"IMAGE": "mysql:5.6"
},
"/home/ukrishnan/projects/mysql/app.xml": {
"ENV_ACCOUNT_BRIDGE_ENDPOINT": "/u01/src/test/sample.txt"
}
}
And I try to parse this JSON using JSON.sh by using,
test_parser=`sh ./lib/JSON.sh < test/test.json`
echo $test_parser
It prints,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog" ["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6" ["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"} ["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt" ["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"} [] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
Whereas, the same command (sh ./lib/JSON.sh < test/test.json), if I run through terminal, it is printing with line breaks,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog"
["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6"
["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"}
["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt"
["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}
[] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
I wanted to read this and assign to bash variables like,
file_name='/home/ukrishnan/projects/test.yml'
key='LOG_DRIVER'
value='syslog'
As I'm almost completely new to shell script and grep or awk, I don't have much idea of how to achieve this. Any help on this would be greatly appreciated.
I wrote a JSON serializer / deserializer for gawk, if you're interested. Save that script and modify it, replacing everything above # === FUNCTIONS === with the following:
#!/usr/bin/gawk -f
# capture JSON string from beginning to end into a scalar variable
{ json = json ORS $0 }
END {
# objectify JSON string to the multilevel array "obj"
deserialize(json, obj)
for (filename in obj) {
print "file_name=" quote(filename)
for (key in obj[filename]) {
# print key="value"
print key "=" quote(obj[filename][key])
}
}
}
Do chmod 755 json.awk and execute it. Output will resemble this:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
ENV_ACCOUNT_BRIDGE_ENDPOINT="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
LOG_DRIVER="syslog"
IMAGE="mysql:5.6"
Hopefully the logic is reasonably easy to follow. If you prefer to output filename=, key=, and value= on every loop iteration, modify the nested for loops accordingly:
for (filename in obj) {
for (key in obj[filename]) {
print "file_name=" quote(filename)
print "key=" quote(key)
print "value=" quote(obj[filename][key])
}
}
That change will result in the following output:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
key="ENV_ACCOUNT_BRIDGE_ENDPOINT"
value="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
key="LOG_DRIVER"
value="syslog"
file_name="/home/ukrishnan/projects/test.yml"
key="IMAGE"
value="mysql:5.6"
Anyway, with that output, you can do something silly in BASH like this to populate and act upon the variables:
#!/bin/bash
./test.awk test5.json | while read -r line; do {
eval $line
[ "${line/=*/}" = "value" ] && {
echo "bash: file_name=$file_name"
echo "bash: key=$key"
echo "bash: value=$value"
echo "------"
}
}; done
It'd probably be more graceful just to do all processing within gawk from start to finish and not mess with the polyglot handoff, though.
Getting back to json.awk, if you prefer to keep json.awk modular for easy reuse in future projects, you could remove everything above # === FUNCTIONS ===, create a separate main.awk containing the code block at the top of this answer, and #include "json.awk" as a helper library pretty much anywhere outside of END {...} (just below the shbang, for example).
JSON.sh (from http://json.org) offers a nice bash friendly means of flattening out a JSON file. Which you've already provided how it looks in your question. So, the flatten form is the format:
[node] tab value
You have to think in UNIX script in extracting the information you want, you'll note the lines you're interested in actually follow this pattern:
["filename","key"] tab ["value"]
In regex notation, we replace:
filename with (.*)
key with (.*)
tab with \t
value with (.*)
We can retrieve the first, second and third matching groups with \1, \2, \3 respectively.
When used in sed we also note that these symbols []() need to be escaped with a backslash \, resulting in the following script:
./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d'
/home/ukrishnan/projects/test.yml,LOG_DRIVER,syslog
/home/ukrishnan/projects/test.yml,IMAGE,mysql:5.6
/home/ukrishnan/projects/mysql/app.xml,ENV_ACCOUNT_BRIDGE_ENDPOINT,/u01/src/test/sample.txt
Now we put the lines in a loop and for each line, we can extract out filename,key,value:
for line in $(./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d')
do
IFS="," read -ra arr <<< $line
filename=${arr[0]}
key=${arr[1]}
value=${arr[2]}
cat <<EOF
filename : $filename
key : $key
value : $value
EOF
done
Which outputs:
filename : /home/ukrishnan/projects/test.yml
key : LOG_DRIVER
value : syslog
filename : /home/ukrishnan/projects/test.yml
key : IMAGE
value : mysql:5.6
filename : /home/ukrishnan/projects/mysql/app.xml
key : ENV_ACCOUNT_BRIDGE_ENDPOINT
value : /u01/src/test/sample.txt

Converting json string in a shell variable to string literal

I have a shell variable which contains json as follows
{
"test" :"test",
"temp":"temp"
}
This is a part of json that I got when parsed from jq, which is a json parser tool.But when I pass this to curl as a part of post request body, it is getting converted to
'{'
'"test"' ':' 'test,'
'"temp"' ':' 'temp'
'}'
But I want it as
'{
"test" : "test",
"temp" : "temp"
}'
VAL=$(echo "$RET" | jq ".pending[$j].value")
VAL is the variable that will contain the json
I need to pass this VAL as request body to curl
curl -X POST -H "$CONTENT_HEADER" -H "$AUTH_HEADER" http://localhost:8080/testApi -d $VAL
The shell is interpreting it as several arguments. Either quote the expansion (as in -d "$VAL") or pipe it instead of saving it to a variable jq '...' | curl ...

Why won't jsontool parse my array?

I'm having trouble parsing this json, which contains an array as its outermost element:
response=[ { "__type": "File", "name": "...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg", "url": "http://files.parse.com/ac3f079b-cacb-49e9-bd74-8325f993f308/...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg" } ]
for blob in $response
do
url=$(echo $blob | json url)
done
But this last json parsing gives a bunch of errors:
json: error: input is not JSON: Bad object at line 2, column 1:
^
Do I need to do something special to turn a JSON array into a bash array, or vice versa?
You should quote the value of reponse to protect it from the shell trying to interpret it:
response='[ { "__type": "File", "name": "...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg", "url": "http://files.parse.com/ac3f079b-cacb-49e9-bd74-8325f993f308/...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg" } ]'
And you can't expect the shell to be able to parse JSON, so for blob in $response isn't going to work out. According to http://trentm.com/json/, using -a should handle the array:
while read url ; do
# use $url here...
done < <(echo "$response" | json -a url)
After putting your json in a file and piping it through jsontool it works just fine, so I'm guessing your file has some weird whitespace which is breaking in jsontool.
Try this instead:
cat your_file.json | sed 's/[[:space:]]\+$//' | json