Currently, I'm attempting to call upon an API to run a POST, with JSON data as the body. So I was wondering if anyone would be able to tell me how I need to format the text below inside the variable $postParams. I'm pretty new at working with JSON so I'm having so trouble with this.
Currently, I only have the following and don't know what to do about the second line on.
$postParams = #{name='Example'}
Here's is the entire data I was hoping to add to $postParams. So if you could help me with the 2nd, 4th, and 8th that'd be awesome. Thanks!
{
"name":"Example",
"template":{"name":"Template"},
"url":"http://localhost",
"page":{"name":"Landing Page"},
"smtp":{"name":"Sending Profile"},
"launch_date":"2019-10-08T17:20:00+00:00",
"send_by_date":null,
"groups":[{"name":"test group"}]
}
You'll need a here-string and ConvertFrom-Json.
here-string:
Quotation marks are also used to create a here-string. A here-string is a single-quoted or double-quoted string in which quotation marks are interpreted literally. A here-string can span multiple lines. All the lines in a here-string are interpreted as strings, even though they are not enclosed in quotation marks.
The resulting code:
# Use a PowerShell here string to take JSON as it is
$jsonString = #"
{
"name":"Example",
"template":{"name":"Template"},
"url":"http://localhost",
"page":{"name":"Landing Page"},
"smtp":{"name":"Sending Profile"},
"launch_date":"2019-10-08T17:20:00+00:00",
"send_by_date":null,
"groups":[{"name":"test group"}]
}
"#
# Pipe the string to create a new JSON object
$jsonObject = $jsonString | ConvertFrom-Json
# The resulting JSON object has properties matching the properties in the orig. JSON
$jsonObject.name
$jsonObject.url
# Nested property
$jsonObject.template.name
# Nested property in array
$jsonObject.groups[0].name
I've posted an online version of the above code at tio.run, so you can play around with it.
If you want to update several properties of the $jsonObject you can do the following:
$jsonObject.name = "NEW NAME"
$jsonObject.url = "NEW URL"
$jsonObject | ConvertTo-Json
ConvertTo-Json will take your object and create an appropriate JSON string:
{
"name": "NEW NAME",
"template": {
"name": "Template"
},
"url": "NEW URL",
"page": {
"name": "Landing Page"
},
"smtp": {
"name": "Sending Profile"
},
"launch_date": "2019-10-08T17:20:00+00:00",
"send_by_date": null,
"groups": [
{
"name": "test group"
}
]
}
If you $jsonObject has more than two levels of depth, use the -Depth parameter, otherwise not all object information will be included in the JSON string.
ConvertTo-Json:
-Depth
Specifies how many levels of contained objects are included in the JSON representation. The default value is 2.
Here is a tio.run link to a ConvertTo-Json example.
Hope that helps.
I can't test it currently, but try this.
$postParams = #'
{
"name":"Example",
"template":{"name":"Template"},
"url":"http://localhost",
"page":{"name":"Landing Page"},
"smtp":{"name":"Sending Profile"},
"launch_date":"2019-10-08T17:20:00+00:00",
"send_by_date":null,
"groups":[{"name":"test group"}]
}
'#
Make a hashtable, then convert to JSON:
$Hashtable = #{
Key1 = "Value1"
Key2 = "Value2"
}
$Json = $Hashtable | ConvertTo-Json
I need to transform an array by adding additional objects -
I have:
"user_id":"testuser"
"auth_token":"abcd"
I need:
"key":"user_id"
"value":"testuser"
"key":"auth_token"
"value":"abcd"
I have been using jq but cant figure out how to do it. Do i need to transform this into a multi-dimensional array first?
I have tried multiple jq queries but cant find the most suitable
When i try using jq i get
jq: error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END (Unix shell quoting issues?) at , line 1
Your input is not json, it's just a bunch of what could be thought of as key/value pairs. Assuming your json input actually looked like this:
{
"user_id": "testuser",
"auth_token": "abcd"
}
You could get an array of key/value pair objects using to_entries.
$ jq 'to_entries' input.json
[
{
"key": "user_id",
"value": "testuser"
},
{
"key": "auth_token",
"value": "abcd"
}
]
If on the other hand your input was actually that, you would need to convert it to a format that can be processed. Fortunately you could read it in as a raw string and probably parse using regular expressions or basic string manipulation.
$ jq -Rn '[inputs|capture("\"(?<key>[^\"]+)\":\"(?<value>[^\"]*)\"")]' input.txt
$ jq -Rn '[inputs|split(":")|map(fromjson)|{key:.[0],value:.[1]}]' input.txt
You can use to_entries filter for that.
Here is jqplay example
Robust conversion of key:value lines to JSON.
If the key:value specifications would be valid JSON except for the
missing punctuation (opening and closing braces etc), then a simple and quite robust approach to converting these key:value pairs to a single valid JSON object is illustrated by the following:
cat <<EOF | jq -nc -R '["{" + inputs + "}" | fromjson] | add'
"user_id": "testuser"
"auth_token" : "abcd"
EOF
Output
{
"user_id": "testuser",
"auth_token": "abcd"
}
I am making an API call on one of my vendor's applications via powershell. The response from the API is a JSON file. Only a portion of the JSON file is encoded in BASE64. Is there a way to read the JSON file with powershell and decode the section that is encoded in BASE64? Example of the JSON file:
I can manually copy the "meta" data from the json file and decode it with powershell using the following code:
[System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String("BASE64DATA"))
however when I try to pass in the JSON file instead of the copied text I get an error.
{
results: [
{
"id": "<Plain Text Data>",
"meta": "<BASE64 Encoded Data>",
"info": "<Plain Text Data>"
},
{
"id": "<Plain Text Data>",
"meta": "<BASE64 Encoded Data>",
"info": "<Plain Text Data>"
}
]
}
You can have the json as an object, then convert it, decode and convert back to json.
$Json = Get-Content -Path c:\JsonFilePath.json | ConvertFrom-Json
$NewJson = $Json.Results | Foreach-Object -Process {
$Converted = [System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String($_.meta))
$_.meta = $Converted
}
$NewJson | ConvertTo-Json
Since you're dealing with a RESTful API that returns JSON I recommend using the Invoke-RestMethod cmdlet to pull the data, it will automatically parse the JSON for you into an object.
Below is an example of how this would be done.
$json = Invoke-RestMethod -Uri 'https://something'
$json.results | ForEach-Object { $_.meta = [System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String($_.meta)) }
$json.results
It will connect, pull the JSON data, and convert it to an object in the first command
The second command will convert the base64 encodings to plain text.
The third command will output the data for you to validate that this is working as expected.
I am trying to form a JSON construct using jq that should ideally look like below:-
{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": [
"event"
],
"traffic_including": [
"unattributed_traffic"
],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
"attribution_campaign_id",
"attribution_creative",
"attribution_date_adjusted",
"attribution_date_utc",
"attribution_matched_by",
"attribution_matched_to",
"attribution_network",
"attribution_network_id",
"attribution_seconds_since",
"attribution_site_id",
"attribution_site_id",
"attribution_tier",
"attribution_timestamp",
"attribution_timestamp_adjusted",
"attribution_tracker",
"attribution_tracker_id",
"attribution_tracker_name",
"count",
"custom_dimensions",
"device_id_adid",
"device_id_android_id",
"device_id_custom",
"device_id_idfa",
"device_id_idfv",
"device_id_kochava",
"device_os",
"device_type",
"device_version",
"dimension_count",
"dimension_data",
"dimension_sum",
"event_name",
"event_time_registered",
"geo_city",
"geo_country",
"geo_lat",
"geo_lon",
"geo_region",
"identity_link",
"install_date_adjusted",
"install_date_utc",
"install_device_version",
"install_devices_adid",
"install_devices_android_id",
"install_devices_custom",
"install_devices_email_0",
"install_devices_email_1",
"install_devices_idfa",
"install_devices_ids",
"install_devices_ip",
"install_devices_waid",
"install_matched_by",
"install_matched_on",
"install_receipt_status",
"install_san_original",
"install_status",
"request_ip",
"request_ua",
"timestamp_adjusted",
"timestamp_utc"
]
}
What I have tried unsuccessfully thus far is below:-
json_construct=$(cat <<EOF
{
"api_key": "6AEC90B5-4169-59AF-7AC9-D655F83B4825",
"app_guid": "komacca-s-rewards-app-au-ios-production-cv8tx71",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv"
"columns_order": ["attribution_attribution_action","attribution_campaign","attribution_campaign_id","attribution_creative","attribution_date_adjusted","attribution_date_utc","attribution_matched_by","attribution_matched_to","attributio
network","attribution_network_id","attribution_seconds_since","attribution_site_id","attribution_tier","attribution_timestamp","attribution_timestamp_adjusted","attribution_tracker","attribution_tracker_id","attribution_tracker_name","
unt","custom_dimensions","device_id_adid","device_id_android_id","device_id_custom","device_id_idfa","device_id_idfv","device_id_kochava","device_os","device_type","device_version","dimension_count","dimension_data","dimension_sum","ev
t_name","event_time_registered","geo_city","geo_country","geo_lat","geo_lon","geo_region","identity_link","install_date_adjusted","install_date_utc","install_device_version","install_devices_adid","install_devices_android_id","install_
vices_custom","install_devices_email_0","install_devices_email_1","install_devices_idfa","install_devices_ids","install_devices_ip","install_devices_waid","install_matched_by","install_matched_on","install_receipt_status","install_san_
iginal","install_status","request_ip","request_ua","timestamp_adjusted","timestamp_utc"]
}
EOF)
followed by:-
echo "$json_construct" | jq '.'
I get the following error:-
parse error: Expected separator between values at line 10, column 15
I am guessing it is because of the string literal which spans to multiple lines that jq is unable to parse it.
Use jq itself:
my_formatted_json=$(jq -n '{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
...,
"timestamp_utc"
]
}')
Your input "JSON" is not valid JSON, as indicated by the error message.
The first error is that a comma is missing after the key/value pair: "delivery_format": "csv", but there are others -- notably, JSON strings cannot be split across lines. Once you fix the key/value pair problem and the JSON strings that are split incorrectly, jq . will work with your text. (Note that once your input is corrected, the longest JSON string is quite short -- 50 characters or so -- whereas jq has no problems processing strings of length 10^8 quite speedily ...)
Generally, jq is rather permissive when it comes to JSON-like input, but if you're ever in doubt, it would make sense to use a validator such as the online validator at jsonlint.com
By the way, the jq FAQ does suggest various ways for handling input that isn't strictly JSON -- see https://github.com/stedolan/jq/wiki/FAQ#processing-not-quite-valid-json
Along the lines of chepner's suggestion since jq can read raw text data you could just use a jq filter to generate a legal json object from your script variables. For example:
#!/bin/bash
# whatever logic you have to obtain bash variables goes here
key=XXXXXXXXXX-7AC9-D655F83B4825
guid=XXXXXXXXXXXXXX
# now use jq filter to read raw text and construct legal json object
json_construct=$(jq -MRn '[inputs]|map(split(" ")|{(.[0]):.[1]})|add' <<EOF
api_key $key
app_guid $guid
EOF)
echo $json_construct
Sample Run (assumes executable script is in script.sh)
$ ./script.sh
{ "api_key": "XXXXXXXXXX-7AC9-D655F83B4825", "app_guid": "XXXXXXXXXXXXXX" }
Try it online!
Need your expertise here!
I am trying to load a JSON file (generated by JSON dumps) into redshift using copy command which is in the following format,
[
{
"cookieId": "cb2278",
"environment": "STAGE",
"errorMessages": [
"70460"
]
}
,
{
"cookieId": "cb2271",
"environment": "STG",
"errorMessages": [
"70460"
]
}
]
We ran into the error - "Invalid JSONPath format: Member is not an object."
when I tried to get rid of square braces - [] and remove the "," comma separator between JSON dicts then it loads perfectly fine.
{
"cookieId": "cb2278",
"environment": "STAGE",
"errorMessages": [
"70460"
]
}
{
"cookieId": "cb2271",
"environment": "STG",
"errorMessages": [
"70460"
]
}
But in reality most JSON files from API s have this formatting.
I could do string replace or reg ex to get rid of , and [] but I am wondering if there is a better way to load into redshift seamlessly with out modifying the file.
One way to convert a JSON array into a stream of the array's elements is to pipe the former into jq '.[]'. The output is sent to stdout.
If the JSON array is in a file named input.json, then the following command will produce a stream of the array's elements on stdout:
$ jq ".[]" input.json
If you want the output in jsonlines format, then use the -c switch (i.e. jq -c ......).
For more on jq, see https://stedolan.github.io/jq