Powershell & Curl - Using variables inside single-quoted JSON body - json

I am currently trying to automate new user creation in our Zendesk ticketing system using Powershell and Curl. The problem I am running into is that the curl json body is enclosed by single quotes and I need to reference a variable inside that body. Here is what I have:
$Firstname = "Test"
$Lastname = "User"
$email= 'user#test.org'
curl.exe https://mydomain.zendesk.com/api/v2/users.json -H "Content-Type: application/json" -X POST -d '{\"user\": {\"name\": \"$Firstname $Lastname\", \"email\": \"$email\"}}' -v -u myuser:mypass
This works fine if I type in regular text values inside the json, but how can I get it to recognize the variables $Firstname, $Lastname and $email?

Try the following:
$Firstname = "Test"
$Lastname = "User"
$email= 'user#test.org'
$json=#"
{\"user\": {\"name\": \"$Firstname $Lastname\", \"email\": \"$email\"}}
"#
curl.exe https://mydomain.zendesk.com/api/v2/users.json -d $json -H 'Content-Type: application/json' -X POST -v -u myuser:mypass
Using a double-quoted here-string #"<newline>...<newline>"# makes specifying embedded " instances easy (no escaping for the sake of PowerShell's own syntax required), while still expanding variable references - see the docs online or Get-Help about_Quoting_Rules.
You're clearly aware of the - unfortunate - additional need to \-escape the " instances, but just to explain why that is needed:
When passing arguments to an external program, PowerShell, after its own parsing and interpolation, conditionally wraps the resulting arguments in double quotes when concatenating them to form the argument list (a single string) to pass to the external utility. Unfortunately, this can result in embedded " instances getting discarded, and the only way to preserve them reliably is to \-escape them - see this answer for more information.
If you wanted to do it inline with a regular double-quoted string, you'd have to escape the " instances for PowerShell too (as `"), resulting in the awkward combination \`":
"{\`"user\`": {\`"name\`": \`"$Firstname $Lastname\`", \`"email\`": \`"$email\`"}}"
Afterthought:
Ryan himself points out in a comment that using a hashtable to construct the data and then converting it to JSON with ConvertTo-Json and feeding it to curl via stdin is an alternative that avoids the quoting headaches:
# Create data as PS hashtable literal.
$data = #{ user = #{ name = "$Firstname $Lastname"; email = "$adUsername#mydomain.org" } }
# Convert to JSON with ConvertTo-Json and pipe to `curl` via *stdin* (-d '#-')
$data | ConvertTo-Json -Compress | curl.exe mydomain.zendesk.com/api/v2/users.json -d '#-' -H "Content-Type: application/json" -X POST -v -u myuser:mypass

I think we can use here-string as json body for Invoke-RestMethod as below
$bufferTime = 5
$requestBody = #"
{
"size": 0,
"aggs": {
"last_x_min": {
"filter": {
"range": {
"#timestamp": {
"gte": "now-$($bufferTime)m",
"lte": "now"
}
}
},
"aggs": {
"value_agg": {
"avg": {
"field": "time-taken"
}
}
}
}
}
}
"#
$esResponse = Invoke-RestMethod -URI http://locahost:9200 -TimeoutSec 15 -Method Post -ContentType 'application/json' -Body $requestBody
This is the script I use to query Elasticsearch. No need to escape double quotes.

Related

Trying to include a dynamic variable within JSON data of a Bash / Shell POST request to API endpoint

I am trying to submit a POST request using Bash that includes JSON data with a variable equal to that of a random string. I get the string dynamically by parsing a file which can vary in content... and may contain regex characters and also new lines.
I will provide an example string here with a curl request that works successfully with the API I am posting this request to. I can only get the request to go through with the string hardcoded into the JSON data and not while assigning a variable to the string like for instance stringVar and using the variable in the JSON data. I could sure use some help where I am messing this up
my working shell script looks something like this
#!/bin/bash
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' -d '{
"name": "my-project",
"files": [
{
"data": "helloWorldFunction() {\n echo \"hello world\" \n}"
},
],
}'
This works, however I need to change data's value from the string
helloWorldFunction() {\n echo "hello world" \n}
to a variable
I have tried settings the variable in different ways in the JSON content from reading suggestions on other questions. For instance I have tried tried changing my shell script to
#!/bin/bash
stringVar="helloWorldFunction() {\n echo \"hello world\" \n}"
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' -d '{
"name": "my-project",
"files": [
{
"data": "${stringVar}"
},
],
}'
i have tried setting the variable like this
"${stringVar}" | turns into ${stringVar}
"$stringVar" | turns into "$stringVar"
and
"'${stringVar}'"
"'$stringVar'"
both seem to return this error
{"error":{"code":"bad_request","message":"Invalid JSON"}}curl: (3) unmatched brace in URL position 1: {\n
and
stringVar
stringVar
$stringVar
"'"$stringVar"'"
""$stringVar""
${stringVar}
all seem to return this error
{"error":{"code":"bad_request","message":"Invalid JSON"}}
Ahh any help on what I am doing wrong would be great.
Thanks in advance y'all
In order to interpolate the value of a variable, you need to enclose the string in double-quotes("), but the JSON also requires literal double-quotes.
The easiest solution is probably to use a here-document to feed the data into curl's standard input, as in #Gilles Quénot's answer. But you can still pass it in via the command line; you just have to be careful with the quotes.
This is one way:
curl ... -d '{
"name": "my-project",
"files": [
{
"data": "'"$stringVar"'"
}
]
}'
The JSON here is mostly contained within single quotation marks ('...'). But right after opening the pair of literal "s that will enclose the value of data, we close the single quotes and switch our shell quotation mode to double quotes in order to include the value of $stringVar. After closing the double quotes around that expansion, we go back into single quotes for the rest of the JSON, starting with closing the literal double-quotes around the value of data.
In a language that used + for string concatenation, it would look like '... "data": "' + "$stringVar" + '"... ', but in the shell you just put the strings next to each other with no operator to concatenate them.
As an alternative, you could put the whole thing in double-quotes, but then you need backslashes to include the literal double quotes that are part of the JSON:
curl ... -d "{
\"name\": \"my-project\",
\"files\": [
{
\"data\": \"$stringVar\"
}
]
}"
So that requires a lot more changes if you're starting from plain JSON; it also looks messier, IMO.
You can also use a tool that knows how to build JSON and let it worry about quoting etc. Here's one way to build it with jq:
jq -n --arg data "$stringVar" '{
"name": "my-project",
"files": [
{
"data": $data
}
]
}'
Using --arg creates a variable inside jq – I named it data – which can then be included in an expression with the syntax $varname ($data in this case). Despite the similarity of syntax, that's not a shell interpolation; we're passing the literal text $data to jq, and jq itself is what replaces it with the value of the variable (which was passed as the second argument to --arg).
There's another tool called jo, which doesn't manipulate JSON but rather produces it, from input that is easier to generate in the shell. Here's one way to construct the desired object with it:
jo name=my-project files="$(jo -a "$(jo data="$stringVar")")"
Either way you can include the constructed JSON in your curl command line like this:
curl ... -d "$(jo or jq command goes here)"
Do not generate such JSON by hand. Use a tool like jq to do it for you.
#!/bin/bash
stringVar="helloWorldFunction() {\n echo \"hello world\" \n}"
jq -n --arg s "$stringVar" '{name: "my-project", files: [{data: $s}]}' |
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-X POST \
-d #- \
'https://api.example.com/v1/endpoint'
Like this:
#!/bin/bash
stringVar="..."
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' \
-d#/dev/stdin <<EOF
{
"name": "my-project",
"files": [
{
"data": $stringVar
},
],
}
EOF
You then should take care about what you fed in the variable, this have to be valid JSON
As an alternative to jq and curl you could use xidel to generate the JSON and submit the POST-request.
With command-line options:
#!/bin/bash
stringVar='helloWorldFunction() {\n echo "hello world" \n}'
xidel -s --variable var="$stringVar" \
-H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json"
-d '{serialize(
{"name":"my-project","files":array{{"data":$var}}},
{"method":"json"}
)}' \
"https://api.example.com/v1/endpoint" \
-e '$raw'
Or with the x:request() function in-query:
#!/bin/bash
stringVar='helloWorldFunction() {\n echo "hello world" \n}'
xidel -s --variable var="$stringVar" -e '
x:request({
"headers":(
"Authorization: Bearer <MY_API_TOKEN>",
"Content-Type: application/json",
"Accept: application/json"
),
"post":serialize(
{"name":"my-project","files":array{{"data":$var}}},
{"method":"json"}
),
"url":"https://api.example.com/v1/endpoint"
})/raw
'
$raw / /raw returns the raw output, like curl. If the API-endpoint returns JSON, then you can use $json / /json to parse it.

Powershell Nested Json doesn't look like a Json

I am currently trying to prepare a JSON body for an API call, which should look something like this
curl -XPOST -H 'Authorization: Bearer ***API*KEY***' -H 'Content-Type: application/json' http://127.0.0.1:9000/api/alert -d '{
"title": "Other alert",
"description": "alert description",
"type": "external",
"source": "instance1",
"sourceRef": "alert-ref",
"severity": 3,
"tlp": 3,
"artifacts": [
{ "dataType": "ip", "data": "127.0.0.1", "message": "localhost" },
{ "dataType": "domain", "data": "thehive-project.org", "tags": ["home", "TheHive"] },
],
"caseTemplate": "external-alert"
}'
The problem however is that my json body which I create with powershell has weird characters and don't see where the problem is. This is my JSON Body
{
"tlp": 1,
"source": "Test",
"title": "Test Alert1",
"artifacts": "{\r\n \"dataType\": \"ip\",\r\n \"data\": \"127.0.0.1\"\r\n}",
"type": "external",
"sourceRef": "1",
"description": "Test",
"severity": 1
}
I create this JSON body as follows. I have already tried CustomObjects and Hashtables, but always get the same output
$body = #{
title = "$title"
description = "$Alert_Description"
type ="external"
source ="$Source"
sourceRef ="$SourceRef"
severity = $Severity
tlp = $tlp
$artifacts = [PSCustomObject]#{
dataType=ip
data=127.0.0.1}
}| ConvertTo-Json
$JsonBody = $body | ConvertTo-Json
Can somebody give me a hint? I have no clue anymore
First, fix your hashtable so that it creates the intended JSON representation, and only apply ConvertTo-Json once:
$body = [ordered] #{
title = $title
description = $Alert_Description
type = 'external'
source = $Source
sourceRef = $SourceRef
severity = $Severity
tlp = $tlp
artifacts = , [pscustomobject] #{
dataType = 'ip'
data = '127.0.0.1'
}
} | ConvertTo-Json
Variables don't need enclosing in "..." unless you explicitly want to convert their values to strings.
Literal textual information such as ip does require quoting ('...', i.e. a verbatim string literal, is best).
The artifacts property is an array in your sample JSON, so the unary form of ,, the array constructor operator is used to wrap the [pscustomobject] instance in a single-element array.
If you have multiple objects, place , between them; if the array was constructed previously and stored in a variable named, say, $arr, use artifacts = $arr.
Use of [ordered], i.e. the creation of an ordered hashtable isn't strictly necessary, but makes it easier to compare the resulting JSON representation to the input hashtable.
Generally keep in mind that explicit use of a -Depth argument with ConvertTo-Json is situationally required in order to ensure that properties aren't truncated - see this post for more information. However, with the hashtable in your question this happens not to be a problem.
Second, since you're sending the json to an external program, curl (note that in Windows PowerShell you'd have to use curl.exe in order to bypass the built-in curl alias for Invoke-WebRequest, an additional escaping step is - unfortunately - required up to at least v7.1 - though that may change in v7.2: The " instances embedded in the value of $body must manually be escaped as \", due to a long-standing bug in PowerShell:
curl -XPOST -H 'Authorization: Bearer ***API*KEY***' -H 'Content-Type: application/json' `
http://127.0.0.1:9000/api/alert -d ($body -replace '([\\]*)"', '$1$1\"')
Note: With your particular sample JSON, -replace '"', '\"' would do.
See this answer for more information.
Slap a "-Depth 50" on your ConvertTo-Json, that should solve your issue.
Like here: ConvertTo-Json flattens arrays over 3 levels deep

Changing Slack CURL call into PowerShell

I've in the middle of writing a translation of a working Slack script from Bash to Powershell, and I'm stumbled on what is literally the meat of the problem; how to use Invoke-WebRequest to replace curl.
This is the curl command that works successfully in Bash on *nix systems:
curl \
-X POST \
-H "Content-type: application/json" \
--data "{
\"attachments\":
[
{
\"mrkdwn_in\": [\"text\"],
\"color\": \"$COLOUR\",
\"text\": \"$MESSAGE\",
}
]
}" \
https://hooks.slack.com/services/tokenpart1/tokenpart2
Note the $COLOUR and $MESSAGE variables are derived from elsewhere in the script (not the section I'm having trouble with).
I can't get it to work in PowerShell. My translation so far is:
$Body = #{
"attachments" = "[
{
"mrkdwn_in": ["text"],
"color": "$Colour",
"text": "$Message",
]"
}
$BodyJSON = $Body |convertto-JSON
Invoke-WebRequest -Headers -Method Post -Body "$BodyJSON" -Uri "https://hooks.slack.com/services/tokenpart1/tokenpart2" -ContentType application/json
This results in the following error:
At C:\path-to-file-etc.ps1:53 char:11
+ "mrkdwn_in": ["text"],
+ ~~~~~~~~~~~~~~~~~~~~~
Unexpected token 'mrkdwn_in": ["text"],
"color": "$COLOUR",
"text": "$MESSAGE",
}
]"' in expression or statement.
At C:\path-to-file-etc.ps1:53 char:11
+ "mrkdwn_in": ["text"],
+ ~
The hash literal was incomplete.At
C:\path-to-file-etc.ps1:53 char:11
+ "mrkdwn_in": ["text"],
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordEx
ception
+ FullyQualifiedErrorId : UnexpectedToken
Process exited with code 1
Process exited with code 1
Step Notify (PowerShell) failed
I have pretty much zero experience in Powershell. Also because this script has to be able to dropped onto a whole variety of boxes,I won't use any libraries or custom cmdlets or anything like that. Out of box methods or die.
Since you're using ConvertTo-Json anyway, it's simpler to construct the entire input as a (nested) custom object / hashtable, and let ConvertTo-Json handle all JSON formatting:
$Body = [pscustomobject] #{
attachments = , [pscustomobject] #{
mrkdwn_in = , 'text'
color = $Colour
text = $Message
}
}
$BodyJson = $Body | ConvertTo-Json -Depth 3
Note: You could substitute [ordered] for [pscustomobject] to use a hashtable with ordered keys instead; even omitting either would work, though seeing the entries in the resulting JSON in different order may be confusing.
Note the use of ,, the array-construction operator, to ensure that the attachments and mrkdwn_in entries are treated as arrays.
Additionally, since ConvertTo-Json only fully serializes to a default depth of 2 levels, -Depth 3 must be used to ensure that the value of entry mrkdwn_in is serialized as an array.
As for what you tried:
Your immediate problem (in addition to the missing } that Jeff Zeitlin points out in a comment on the question): You've neglected to escape the embedded " chars. in your multi-line string.
Therefore, as documentation topic Get-Help about_Quoting_Rules discusses, you can either use `" to embed double quotes inside "..." or use a here-string (#"<nl>....<n>"#).
Even with the syntax problems fixed, your code wouldn't work as intended, however, because ConvertTo-Json wouldn't preserve the pre-formatted JSON in the attachments entry and instead treat the string value as a string literal that needs escaping; here's a simple example:
#{
foo = "[
1,
2
]"
} | ConvertTo-Json
The above yields:
{
"foo": "[\n 1, \n 2 \n ]"
}

Powershell curl double quotes

I am trying to invoke a curl command in powershell and pass some JSON information.
Here is my command:
curl -X POST -u username:password -H "Content-Type: application/json" -d "{ "fields": { "project": { "key": "key" }, "summary": "summary", "description": "description - here", "type": { "name": "Task" }}}"
I was getting globbing errors and "unmatched braces" and host could not be resolved, etc.
Then I tried prefixing the double quotes in the string with the backtick character, but it could not recognize the - character in the description json field
thanks
EDIT 1:
When I wrote the curl command in a regular batch file, I used double quotes and no single quotes. Also, in the -d string, I escaped all the double quotes with \ and the command worked.
In this case, my curl is actually pointing to curl.exe. I specified the path, just didn't list it here. Also I tried adding single quotes around -d and I got:
curl: option -: is unknown curl: try 'curl --help' or 'curl --manual' for more information
Seems like it cannot recognize the - character in the JSON
Pipe the data into curl.exe, instead of trying to escape it.
$data = #{
fields = #{
project = #{
key = "key"
}
summary = "summary"
description = "description - here"
type = #{
name = "Task"
}
}
}
$data | ConvertTo-Json -Compress | curl.exe -X POST -u username:password -H "Content-Type: application/json" -d "#-"
curl.exe reads stdin if you use #- as your data parameter.
P.S.: I strongly suggest you use a proper data structure and ConvertTo-Json, as shown, instead of building the JSON string manually.
Easy way (for simple testing):
curl -X POST -H "Content-Type: application/json" -d '{ \"field\": \"value\"}'

Converting json string in a shell variable to string literal

I have a shell variable which contains json as follows
{
"test" :"test",
"temp":"temp"
}
This is a part of json that I got when parsed from jq, which is a json parser tool.But when I pass this to curl as a part of post request body, it is getting converted to
'{'
'"test"' ':' 'test,'
'"temp"' ':' 'temp'
'}'
But I want it as
'{
"test" : "test",
"temp" : "temp"
}'
VAL=$(echo "$RET" | jq ".pending[$j].value")
VAL is the variable that will contain the json
I need to pass this VAL as request body to curl
curl -X POST -H "$CONTENT_HEADER" -H "$AUTH_HEADER" http://localhost:8080/testApi -d $VAL
The shell is interpreting it as several arguments. Either quote the expansion (as in -d "$VAL") or pipe it instead of saving it to a variable jq '...' | curl ...