Ansible win_shell with JSON to powershell, works fine other way - json

I can not get how to get win_shell to send JSON to a powershell script on a windows server. If I run this:
vm:
annotation: This is a VM
cdrom:
type: client
iso_path: ''
controller_type: ''
state: ''
cluster: ''
- name: "Run script '.ps1'"
win_shell: D:\scripts\outputValues.ps1 -vm "{{vm}}"
register: result
- name: Process win_shell output
set_fact:
fullResult: "{{ result.stdout | from_json }}"
I get this string into powershell that I can not even convert TO JSON:
{u'cdrom': {u'controller_type': u'', u'state': u'', u'type': u'client', u'iso_path': u''}, u'cluster': u'', u'annotation': u'This is a VM'}
If I run the script with this:
win_shell: D:\scripts\outputValues.ps1 -vm "{{vm | to_json}}"
All I get is an open curly bracket '{' into powershell.
It does work the other way. As a test, if I call that same powershell script from win_shell and ignore the input in powershell and simply create an object like this and send it back to ansible, it works fine. Why can I send JSON one way and not the other? I have tried vm|to_json etc.
#Powershell
$vmAttribs = #{"type" = "client"; "iso_path" = ""; "controller_type" =""; "state" =""}
$vmObject = New-Object psobject -Property #{
annotation = 'This is a VM'
cdrom = $vmAttribs
cluster = ""
}
$result = $vmObject | ConvertTo-Json -Depth 8
Write-Output $result
ansible gets:
{
"msg": {
"cdrom": {
"controller_type": "",
"state": "",
"type": "client",
"iso_path": ""
},
"cluster": "",
"annotation": "This is a VM"
},
"changed": false,
"_ansible_verbose_always": true,
"_ansible_no_log": false
}

What you're seeing is a Python structure signifying JSON notation with Unicode strings (e.g. u'my string value'). Obviously, that's not exactly portable to other runtimes like Powershell, and results in an invalid string.
Reading up on Ansible filters, you will want to use {{ vm | to_json }} to ensure that the data structure is stringified to JSON correctly.
Workaround
I want to stress that this is not the ideal way of getting a JSON object out of Ansible, and a proper solution that doesn't involve hacking out the Unicode identifier from the JSON string is desirable to this.
I don't know enough about Ansible to know why the string becomes { when piping vm to to_json, but here's a workaround that might get you going until someone else can chime in with what is going on with Ansible's filtering here.
In your .ps1 script, you can use the following -replace to remove those u characters before the strings:
$vm = $vm -replace "u(?=([`"']).*\1)"
You can test this by running the following in an interactive terminal:
"{u'cdrom': {u'controller_type': u'', u'state': u'', u'type': u'client', u'iso_path': u''}, u'cluster': u'', u'annotation': u'This is a VM'}" -replace "u(?=([`"']).*\1)"
The pattern matches on the u character that comes immediately before a single-quote or double-quote followed by any number of characters (including none) followed by another single-quote or double quote, whichever was matched the first time. -replace does not require a second argument if you are replacing the pattern with an empty string.

Related

Combining JSON items using JMESPath and/or Ansible

I have an Ansible playbook that queries a device inventory API and gets back a JSON result that contains a lot of records following this format:
{
"service_level": "Test",
"tags": [
"Application:MyApp1"
],
"fqdn": "matestsvcapp1.vipcustomers.com",
"ip": "172.20.11.237",
"name": "matestsvcapp1.vipcustomers.com"
}
I then loop through these ansible tasks to query the JSON result for each of the IP addresses I care about:
- name: Set JMESQuery
set_fact:
jmesquery: "Devices[?ip_addresses[?ip.contains(#,'{{ ip_to_query }}' )]].{ip: '{{ ip_to_query }}', tags: tags[], , service_level: service_level }"
- name: Store values
set_fact:
inven_results: "{{ (inven_results| default([])) + (existing_device_info.json | to_json | from_json | json_query(jmesquery)) }}"
I then go on to do other tasks in ansible, pushing this data into other systems, and everything works fine.
However, I just got a request from management that they would like to see the 'service level' represented as a tag in some of the systems I push this data into. Therefore I need to combine the 'tags' and 'service_level' items resulting in something that looks like this:
{
"tags": [
"Application:MyApp1",
"service_level:Test"
],
"fqdn": "matestsvcapp1.vipcustomers.com",
"ip": "172.20.11.237",
"name": "matestsvcapp1.vipcustomers.com"
}
I've tried modifying the JMESPath query to join the results together using the join function, and tried doing it the 'ansible' way, using the combine or map, but I couldn't get either of those to work either.
Any thoughts on the correct way to handle this? Thanks in advance!
Note: 'tags' is a list of strings, and even though it's written in key:value format, it's really just a string.
to add two arrays you use the + operator like this:
ansible localhost -m debug -a 'msg="{{ b + ["String3"] }}"' -e '{"b":["String1", "String2"]}'
result:
localhost | SUCCESS => {
"msg": [
"String1",
"String2",
"String3"
]
}
So if i take your json code as test.json you could run
ansible localhost -m debug -a 'msg="{{ tags + ["service_level:" ~ service_level ] }}"' -e #test.json
Result:
localhost | SUCCESS => {
"msg": [
"Application:MyApp1",
"service_level:Test"
]
}
With this knowledge you can use set_fact to put this new array in a variable for later use.

Pass JSON variable to AzureDevops Release definition

My goal is to pass a JSON object from one machine to another using the AzureDevops pipeline variables.
The process starts with a powershell script that obtains the JSON object and compresses it to:
$json=[{"test":"foo","bar":"hello}].
Please note that it will be always an array.
Now, I set the azure variable with:
Write-Host "##vso[task.setvariable variable=Json]$json"
now the variable is initialized in the release pipeline BUT, the double quotes are not escaped.
That means that when I try to obtain the $(Json) in the next script it fails due to invalid characters of course. My question is, how can escape those double quotes? I have tried adding single quotes to the beginning and the end of the string but it won't work. Thanks!
You may try the following format for the JSON file:
$json = #"
[
{
"op": "add",
"path": "/fields/System.Title",
"value": "Bug22"
},
{
"op": "add",
"path": "/fields/Microsoft.VSTS.Common.Severity",
"value": "3 - Medium"
}
]
"# | ConvertTo-Json -Compress
Have you tried wrapping the pipeline variable with a here-string before using it?
$json = #"
$(Json)
"#

Parsing JSON file in powershell with specific characters

I am trying to get the values in powershell within specific characters. Basically I have a json with thousands of objects like this
"Name": "AllZones_APOPreface_GeographyMatch_FromBRE_ToSTR",
"Sequence": 0,
"Condition": "this.TripOriginLocationCode==\"BRE\"&&this.TripDestinationLocationCode==\"STR\"",
"Action": "this.FeesRate=0.19m;this.ZoneCode=\"Zone1\";halt",
"ElseAction": ""
I want everything within \" \"
IE here I would see that BRE and STR is Zone1
All I need is those 3 things outputted.
I have been searching how to do it with ConvertFrom-Json but no success, maybe I just havent found a good article on this.
Thanks
Start by representing your JSON as a string:
$myjson = #'
{
"Name": "AllZones_APOPreface_GeographyMatch_FromBRE_ToSTR",
"Sequence": 0,
"Condition": "this.TripOriginLocationCode==\"BRE\"&&this.TripDestinationLocationCode==\"STR\"",
"Action": "this.FeesRate=0.19m;this.ZoneCode=\"Zone1\";halt",
"ElseAction": ""
}
'#
Next, create a regular expression that matches everything in between \" and \", that's under 10 characters long (else it'll match unwanted results).
$regex = [regex]::new('\\"(?<content>.{1,10})\\"')
Next, perform the regular expression comparison, by calling the Matches() method on the regular expression. Pass your JSON string into the method parameters, as the text that you want to perform the comparison against.
$matchlist = $regex.Matches($myjson)
Finally, grab the content match group that was defined in the regular expression, and extract the values from it.
$matchlist.Groups.Where({ $PSItem.Name -eq 'content' }).Value
Result
BRE
STR
Zone1
Approach #2: Use Regex Look-behinds for more accurate matching
Here's a more specific regular expression that uses look-behinds to validate each field appropriately. Then we assign each match to a developer-friendly variable name.
$regex = [regex]::new('(?<=TripOriginLocationCode==\\")(?<OriginCode>\w+)|(?<=TripDestinationLocationCode==\\")(?<DestinationCode>\w+)|(?<=ZoneCode=\\")(?<ZoneCode>\w+)')
$matchlist = $regex.Matches($myjson)
### Assign each component to its own friendly variable name
$OriginCode, $DestinationCode, $ZoneCode = $matchlist[0].Value, $matchlist[1].Value, $matchlist[2].Value
### Construct a string from the individual components
'Your origin code is {0}, your destination code is {1}, and your zone code is {2}' -f $OriginCode, $DestinationCode, $ZoneCode
Result
Your origin code is BRE, your destination code is STR, and your zone code is Zone1

jq construct with value strings spanning multiple lines

I am trying to form a JSON construct using jq that should ideally look like below:-
{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": [
"event"
],
"traffic_including": [
"unattributed_traffic"
],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
"attribution_campaign_id",
"attribution_creative",
"attribution_date_adjusted",
"attribution_date_utc",
"attribution_matched_by",
"attribution_matched_to",
"attribution_network",
"attribution_network_id",
"attribution_seconds_since",
"attribution_site_id",
"attribution_site_id",
"attribution_tier",
"attribution_timestamp",
"attribution_timestamp_adjusted",
"attribution_tracker",
"attribution_tracker_id",
"attribution_tracker_name",
"count",
"custom_dimensions",
"device_id_adid",
"device_id_android_id",
"device_id_custom",
"device_id_idfa",
"device_id_idfv",
"device_id_kochava",
"device_os",
"device_type",
"device_version",
"dimension_count",
"dimension_data",
"dimension_sum",
"event_name",
"event_time_registered",
"geo_city",
"geo_country",
"geo_lat",
"geo_lon",
"geo_region",
"identity_link",
"install_date_adjusted",
"install_date_utc",
"install_device_version",
"install_devices_adid",
"install_devices_android_id",
"install_devices_custom",
"install_devices_email_0",
"install_devices_email_1",
"install_devices_idfa",
"install_devices_ids",
"install_devices_ip",
"install_devices_waid",
"install_matched_by",
"install_matched_on",
"install_receipt_status",
"install_san_original",
"install_status",
"request_ip",
"request_ua",
"timestamp_adjusted",
"timestamp_utc"
]
}
What I have tried unsuccessfully thus far is below:-
json_construct=$(cat <<EOF
{
"api_key": "6AEC90B5-4169-59AF-7AC9-D655F83B4825",
"app_guid": "komacca-s-rewards-app-au-ios-production-cv8tx71",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv"
"columns_order": ["attribution_attribution_action","attribution_campaign","attribution_campaign_id","attribution_creative","attribution_date_adjusted","attribution_date_utc","attribution_matched_by","attribution_matched_to","attributio
network","attribution_network_id","attribution_seconds_since","attribution_site_id","attribution_tier","attribution_timestamp","attribution_timestamp_adjusted","attribution_tracker","attribution_tracker_id","attribution_tracker_name","
unt","custom_dimensions","device_id_adid","device_id_android_id","device_id_custom","device_id_idfa","device_id_idfv","device_id_kochava","device_os","device_type","device_version","dimension_count","dimension_data","dimension_sum","ev
t_name","event_time_registered","geo_city","geo_country","geo_lat","geo_lon","geo_region","identity_link","install_date_adjusted","install_date_utc","install_device_version","install_devices_adid","install_devices_android_id","install_
vices_custom","install_devices_email_0","install_devices_email_1","install_devices_idfa","install_devices_ids","install_devices_ip","install_devices_waid","install_matched_by","install_matched_on","install_receipt_status","install_san_
iginal","install_status","request_ip","request_ua","timestamp_adjusted","timestamp_utc"]
}
EOF)
followed by:-
echo "$json_construct" | jq '.'
I get the following error:-
parse error: Expected separator between values at line 10, column 15
I am guessing it is because of the string literal which spans to multiple lines that jq is unable to parse it.
Use jq itself:
my_formatted_json=$(jq -n '{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
...,
"timestamp_utc"
]
}')
Your input "JSON" is not valid JSON, as indicated by the error message.
The first error is that a comma is missing after the key/value pair: "delivery_format": "csv", but there are others -- notably, JSON strings cannot be split across lines. Once you fix the key/value pair problem and the JSON strings that are split incorrectly, jq . will work with your text. (Note that once your input is corrected, the longest JSON string is quite short -- 50 characters or so -- whereas jq has no problems processing strings of length 10^8 quite speedily ...)
Generally, jq is rather permissive when it comes to JSON-like input, but if you're ever in doubt, it would make sense to use a validator such as the online validator at jsonlint.com
By the way, the jq FAQ does suggest various ways for handling input that isn't strictly JSON -- see https://github.com/stedolan/jq/wiki/FAQ#processing-not-quite-valid-json
Along the lines of chepner's suggestion since jq can read raw text data you could just use a jq filter to generate a legal json object from your script variables. For example:
#!/bin/bash
# whatever logic you have to obtain bash variables goes here
key=XXXXXXXXXX-7AC9-D655F83B4825
guid=XXXXXXXXXXXXXX
# now use jq filter to read raw text and construct legal json object
json_construct=$(jq -MRn '[inputs]|map(split(" ")|{(.[0]):.[1]})|add' <<EOF
api_key $key
app_guid $guid
EOF)
echo $json_construct
Sample Run (assumes executable script is in script.sh)
$ ./script.sh
{ "api_key": "XXXXXXXXXX-7AC9-D655F83B4825", "app_guid": "XXXXXXXXXXXXXX" }
Try it online!

Why won't jsontool parse my array?

I'm having trouble parsing this json, which contains an array as its outermost element:
response=[ { "__type": "File", "name": "...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg", "url": "http://files.parse.com/ac3f079b-cacb-49e9-bd74-8325f993f308/...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg" } ]
for blob in $response
do
url=$(echo $blob | json url)
done
But this last json parsing gives a bunch of errors:
json: error: input is not JSON: Bad object at line 2, column 1:
^
Do I need to do something special to turn a JSON array into a bash array, or vice versa?
You should quote the value of reponse to protect it from the shell trying to interpret it:
response='[ { "__type": "File", "name": "...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg", "url": "http://files.parse.com/ac3f079b-cacb-49e9-bd74-8325f993f308/...tfss-ea51ec70-d3a8-45e5-abbf-294f2c2c81f0-myPicture.jpg" } ]'
And you can't expect the shell to be able to parse JSON, so for blob in $response isn't going to work out. According to http://trentm.com/json/, using -a should handle the array:
while read url ; do
# use $url here...
done < <(echo "$response" | json -a url)
After putting your json in a file and piping it through jsontool it works just fine, so I'm guessing your file has some weird whitespace which is breaking in jsontool.
Try this instead:
cat your_file.json | sed 's/[[:space:]]\+$//' | json