Combining Nested Json using PowerShell - json

I have the following Json script:
{
"merchant_info": {
"email": "merchant#example.com",
"first_name": "David",
"last_name": "Larusso",
"business_name": "Mitchell & Murray",
"phone": {
"country_code": "001",
"national_number": "4085551234"
},
"address": {
"line1": "1234 First Street",
"city": "Anytown",
"state": "CA",
"postal_code": "98765",
"country_code": "US"
}
},
"billing_info": [{
"email": "bill-me#example.com",
"first_name": "Stephanie",
"last_name": "Meyers"
}
],
"shipping_info": {
"first_name": "Stephanie",
"last_name": "Meyers",
"address": {
"line1": "1234 Main Street",
"city": "Anytown",
"state": "CA",
"postal_code": "98765",
"country_code": "US"
}
},
"items": [{
"name": "Zoom System wireless headphones",
"quantity": 2,
"unit_price": {
"currency": "USD",
"value": "120"
},
"tax": {
"name": "Tax",
"percent": 8
}
}, {
"name": "Bluetooth speaker",
"quantity": 1,
"unit_price": {
"currency": "USD",
"value": "145"
},
"tax": {
"name": "Tax",
"percent": 8
}
}
],
"discount": {
"percent": 1
},
"shipping_cost": {
"amount": {
"currency": "USD",
"value": "10"
}
},
"note": "Thank you for your business.",
"terms": "No refunds after 30 days."
}
And I want to use PowerShell to get the following Record and export it to CSV:
So far I created the following Script:
$JsonFile = "C:\Users\me\Documents\myfile.json"
$OutputFile = "C:\Users\me\Documents\newtext.csv"
Get-Content -Path $OutputFile
$json = ConvertFrom-Json (Get-Content $JsonFile -Raw)
$json.merchant_info | Select "first_name","last_name",#{Label = "phone"; Expression = {$_.phone.national_number}} |
Export-Csv $OutputFile -NoTypeInformation
I am able to bring values from (Merchant_info, Shipping_info, item) separetely but how do I bring it all in combined like in my screen shot above.

but how do I bring it all in combined like in my screen shot above.
We can only guess; assuming this entire json block is one order, with one merchant and one customer, but multiple items, then each row is an item. So start with that as the input:
Create an output record (PSCustomObject) with the repeated data, and then the individual item data:
$json.items | ForEach-Object {
[PSCustomObject]#{
MerchantInfoFirstName = $json.merchant_info.first_name
MerchantInfoLastName = $json.merchant_info.last_name
MerchantInfoPhoneNumber = $json.merchant_info.phone.national_number
ShippingInfoFirstName = $json.shipping_info.first_name
ShippingInfoLastName = $json.shipping_info.last_name
ItemName = $_.name
ItemQuantity = $_.quantity
}
} | Export-Csv ... etc.

Related

JQ merge WooCommerce REST API responce

Im trying to parse data from woocommerce from linux console.
Need to take only shipping info and item names with quantity.
im making curl request to wp-json/wc/v2/orders/ORDER_ID
then
jq '{order_id:.id,ship_info:.shipping,items: (.line_items[] | {name , quantity} ) }'
If order contains two items, jq will return two objects
example:
{
"order_id": 1234,
"ship_info": {
"first_name": "Sam",
"last_name": "Fisher",
"company": "",
"address_1": "24 Mega Drive",
"address_2": "",
"city": "Eglinton",
"state": "WA",
"postcode": "6032",
"country": "AU",
"phone": ""
},
"items": {
"name": "Black T-shirt",
"quantity": 1
}
}
{
"order_id": 1234,
"ship_info": {
"first_name": "Sam",
"last_name": "Fisher",
"company": "",
"address_1": "24 Mega Drive",
"address_2": "",
"city": "Eglinton",
"state": "WA",
"postcode": "6032",
"country": "AU",
"phone": ""
},
"items": {
"name": "White T-shirt",
"quantity": 1
}
}
I want merge items and use item's name as a key and item's qty as a value. Please advice how to get output like this
{
"order_id": 1234,
"ship_info": {
"first_name": "Sam",
"last_name": "Fisher",
"company": "",
"address_1": "24 Mega Drive",
"address_2": "",
"city": "Eglinton",
"state": "WA",
"postcode": "6032",
"country": "AU",
"phone": ""
},
"items": {
"White T-shirt": "1",
"Black T-shirt": "1"
}
}
With your current jq query you are iterating over the items inside a generated object. That's why you receive one object per item. Rather than merging them afterwards, don't separate them in the first place.
If you changed your query from
jq '{
order_id: .id,
ship_info: .shipping,
items: (.line_items[] | {name, quantity})
}'
to just
jq '{
order_id: .id,
ship_info: .shipping,
items: .line_items
}'
you'd probably already see that .line_items is actually an array.
To transform it according to your desired output, change that line to one of the followings. They should all yield the same result.
items: (.line_items | map({(.name): .quantity}) | add)
items: (INDEX(.line_items[]; .name) | map_values(.quantity))
items: (reduce .line_items[] as $i ({}; .[$i.name] = .quantity))

Compare 2 JSON and retrieve subset from one of them based on condition in Powershell

I have two JSON files abc.json and xyz.json.
Content in abc.json is:
[{"id": "121",
"name": "John",
"location": "europe"
},
{"id": "100",
"name": "Jane",
"location": "asia"
},
{"id": "202",
"name": "Doe",
"location": "america"
}
]
Updated -> Content in xyz.json is:
{
"value": [
{
"id": "111",
"city": "sydney",
"profession": "painter"
},
{
"id": "200",
"city": "istanbul",
"profession": "actor"
},
{
"id": "202",
"city": "seattle",
"profession": "doctor"
}
],
"count": {
"type": "Total",
"value": 3
}
}
I want to get those records of abc.json in when the id in both objects are equal.In this case:
{"id": "202",
"name": "Doe",
"location": "america"
}
I need to do this in Powershell and the version I am using is 5.1.This is what I have tried:
$OutputList = #{}
$abcHash = Get-Content 'path\to\abc.json' | Out-String | ConvertFrom-Json
$xyzHash = Get-Content 'path\to\xyz.json' | Out-String | ConvertFrom-Json
$xyzResp = $xyzHash.value
foreach($item in $xyzResp){
foreach ($record in $abcHash){
if ($item.id -eq $record.id){
$OutputList.Add($record, $null)
}
}
}
Write-Output $OutputList
But on printing the OutputList , I get like this:
Key: #{"id": "202",
"name": "Doe",
"location": "america"
}
Value:
Name:#{"id": "202",
"name": "Doe",
"location": "america"
}
What I require is more of a PSObject like:
id: 202
name:Doe
location:america
I tried using Get-Member cmdlet but could not quite reach there.
Is there any suggestion I could use?
I have corrected your example xyz.json because there was an extra comma in there that should not be there. Also, the example did not have an iten with id 202, so there would be no match at all..
xyz.json
{
"value": [
{
"id": "111",
"city": "sydney",
"profession": "painter"
},
{
"id": "202",
"city": "denver",
"profession": "painter"
},
{
"id": "111",
"city": "sydney",
"profession": "painter"
}
],
"count": {
"type": "Total",
"value": 3
}
}
That said, you can use a simple Where-Object{...} to get the item(s) with matching id's like this:
$abc = Get-Content 'path\to\abc.json' -Raw | ConvertFrom-Json
$xyz = Get-Content 'path\to\xyz.json' -Raw | ConvertFrom-Json
# get the items with matching id's as object(s)
$abc | Where-Object { $xyz.value.id -contains $_.id}
Output:
id name location
-- ---- --------
202 Doe america
Of course you can capture the output first and display as list and/or save to csv, convert back to json and save that.

How to exclude data from json file to convert into csv file powershell

I want to convert my json file into csv -
{
"count": 28,
"value": [
{
"commitId": "65bb6a911872c314a9225815007d74a",
"author": {
"name": "john doe",
"email": "john.doe#gmail.com",
"date": "2020-06-09T17:03:33Z"
},
"committer": {
"name": "john doe",
"email": "john.doe#gmail.com",
"date": "2020-06-09T17:03:33Z"
},
"comment": "Merge pull request 3 from dev into master",
"changeCounts": {
"Add": 6,
"Edit": 0,
"Delete": 0
},
"url": "https://dev.azure.com/",
"remoteUrl": "https://dev.azure.com/"
},
{
"commitId": "bdcb4a1e1e671c15333eacc31aa9795fe32",
"author": {
"name": "john doe",
"email": "john.doe#gmail.com",
"date": "2020-06-09T17:03:33Z"
},
"committer": {
"name": "john doe",
"email": "john.doe#gmail.com",
"date": "2020-06-09T17:03:33Z"
},
"comment": "Updated FGDH",
"changeCounts": {
"Add": 0,
"Edit": 1,
"Delete": 0
},
"url": "https://dev.azure.com/",
"remoteUrl": "https://dev.azure.com/"
}
]
}
I don't want all the fields in my CSV file. I want commitid,commiter.name, commiter.date and comment only.
Get-Content "\commitinfo.json" -Raw | ConvertFrom-Json | Select -Expand value |Export-Csv
"\commitinfo.csv"
with this I get all data , How can I get selected data ?
Another json file -
{
"value": [
{
"id": "5c264dd2-bbcf-4537-8a63-19a0a4d2dc",
"name": "Develop",
"url": "https://dev.azure.com/",
"project": {
"id": "0042dc5c-fd13-4e3c-bfd7-7feb52e287",
"name": "test",
"url": "https://dev.azure.com/",
"state": "wellFormed",
"revision": 11,
"visibility": "private",
"lastUpdateTime": "2020-04-15T04:04:30.01Z"
},
"defaultBranch": "refs/heads/master",
"size": 55438,
"remoteUrl": "https://dev.azure.com",
"sshUrl": "git#ssh.dev.azure.com:v3",
"webUrl": "https://dev.azure.com/"
},
{
"id": "3219e8e2-281d-40ad-81c8-1cecf8",
"name": "automation",
"url": "https://dev.azure.com/",
"project": {
"id": "0e2df786-94a0-42c8-b068-c2656b",
"name": "automation",
"url": "https://dev.azure.com/",
"state": "wellFormed",
"revision": 19,
"visibility": "private",
"lastUpdateTime": "2020-05-02T03:32:50.937Z"
},
"size": 0,
"remoteUrl": "https://dev.azure.com/",
"sshUrl": "git#ssh.dev.azure.com:v3",
"webUrl": "https://dev.azure.com/"
}
]
}
from this file I only want to select projects with size - 0 , CSV should look like this
name size
a 0
b 0
Note: Piping the commands below to Export-Csv is omitted for brevity.
You need to use Select-Object with calculated properties:
(Get-Content commitinfo.json -Raw | ConvertFrom-Json).value |
Select-Object commitid,
#{n='committerName'; e={ $_.committer.name } },
#{n='committerDate'; e={ $_.committer.date } },
comment
In the second case, add Where-Object to filter in the objects of interest:
(Get-Content other.json -Raw | ConvertFrom-Json).value |
Where-Object size -eq 0 |
Select-Object name, size

Splitting nested arrays as separate entities

I have some JSON data which contains attributes and some array elements. I would like to push a given set of fields into the array elements and then separate the arrays as separate entities.
Source data looks like this
[
{
"phones": [
{
"phone": "555-555-1234",
"type": "home"
},
{
"phone": "555-555-5678",
"type": "mobile"
}
],
"email": [
{
"email": "a#b.com",
"type": "work"
},
{
"email": "x#c.com",
"type": "home"
}
],
"name": "john doe",
"year": "2012",
"city": "cupertino",
"zip": "555004"
},
{
"phones": [
{
"phone": "555-666-1234",
"type": "home"
},
{
"phone": "555-666-5678",
"type": "mobile"
}
],
"email": [
{
"email": "a#b.com",
"type": "work"
},
{
"email": "x#c.com",
"type": "home"
}
],
"name": "jane doe",
"year": "2000",
"city": "los angeles",
"zip": "555004"
}
]
I expect a result like this
{
"person": [
{
"name": "john doe",
"year": "2012",
"city": "cupertino",
"zip": "555004"
},
{
"name": "jane doe",
"year": "2000",
"city": "los angeles",
"zip": "555004"
}
],
"phones": [
{
"name": "john doe",
"year": "2012",
"phone": "555-555-1234",
"type": "home"
},
{
"name": "john doe",
"year": "2012",
"phone": "555-555-5678",
"type": "mobile"
},
{
"name": "jane doe",
"year": "2000",
"phone": "555-666-1234",
"type": "home"
},
{
"name": "jane doe",
"year": "2000",
"phone": "555-666-5678",
"type": "mobile"
}
],
"email": [
{
"name": "john doe",
"year": "2012",
"email": "a#b.com",
"type": "work"
},
{
"name": "john doe",
"year": "2012",
"email": "x#c.com",
"type": "home"
},
{
"name": "jane doe",
"year": "2000",
"email": "a#b.com",
"type": "work"
},
{
"name": "jane doe",
"year": "2000",
"email": "x#c.com",
"type": "home"
}
]
}
I have been able to get the desired result, but I can't make it work in a generic way.
experiment on jqterm
The code below achieves the job, but I would like to pass the array of columns to be injected into the child arrays, the name of the primary result and an array containing the array field names.
["phones", "email"] as $children
| ["name", "year"] as $ids
|{person: map(with_entries(
. as $data | select($children|contains([$data.key])|not)
))}
+ {"phones": split_child($children[0];$ids)}
+ {"email": split_child($children[1];$ids)}
It's a lot more easier to achieve this using multiple reduces, like:
def split_data($parent; $ids; $arr_cols):
($arr_cols | map([.])) as $p
| reduce .[] as $in ({}; .[$parent] += [$in | delpaths($p)]
| (reduce $ids[] as $k ({}; . + {($k): $in[$k]}) as $s
| reduce $arr_cols[] as $k (.; .[$k] += [$in[$k][] + $s])
);
split_data("person"; ["name", "year"]; ["phones", "email"])
Here's a straightforward solution to the generic problem (it uses reduce only once, in a helper function). To understand it, it might be helpful to see it as an abstraction of this concrete solution:
{ person: [.[] | {name, year, city, zip} ]}
+ { phones: [.[] | ({name, year} + .phones[]) ]}
+ { email: [.[] | ({name, year} + .email[]) ]}
Helper function
Let's first define a helper function for constructing an object by selecting a set of keys:
def pick($ary):
. as $in
| reduce $ary[] as $k ({};
. + {($k): $in[$k]});
split_data
Here finally is the function that takes as arguments the $parent, $ids, and columns of interest. The main complication is ensuring that the supplemental keys ("city" and "zip") are dealt with in the proper order.
def split_data($parent; $ids; $arr_cols):
(.[0]|keys_unsorted - $arr_cols - $ids) as $extra
| { ($parent): [.[] | pick($ids + $extra)] }
+ ([$arr_cols[] as $k
| {($k): [.[] | pick($ids) + .[$k][]] }] | add) ;
The invocation:
split_data("person"; ["name", "year"]; ["phones", "email"])
produces the desired result.

ConvertFrom-Json "add" null items to array

I am working with a file that has a JSON string in it.
{
"ConfigurationItems": [{
"ActiveDirectory": [{
"Users": [
{ "FirstName": "U1", "LastName": "L1", "Department": "D1", "UserName": "UN1" },
{ "FirstName": "U2", "LastName": "L2", "Department": "D2", "UserName": "UN2" },
{ "FirstName": "U3", "LastName": "L3", "Department": "D3", "UserName": "UN3" },
{ "FirstName": "U4", "LastName": "L4", "Department": "D4", "UserName": "UN4" },
{ "FirstName": "U5", "LastName": "L5", "Department": "D5", "UserName": "UN5" }
]
},
{
"Groups": [
{ "Name": "G1", "Scope": "Global" },
{ "Name": "G2", "Scope": "Global" },
{ "Name": "G3", "Scope": "Global" }
]
},
{
"OU": [
{ "Name": "N1" },
{ "Name": "N2" },
{ "Name": "N3" }
]
}
]
}]
}
I assign this to a variable by doing the following:
$t = Get-content -path $pathtoJSON -raw | ConvertFrom-Json
Now comes the part I cannot explain if I get the count of Users it returns 7
$t.ConfigurationItems.ActiveDirectory.Users.Count
The main issue is when I iterate through this, I will always end up with two $null items in my loop, which throws off my code. How do I ensure I get the correct count from ConvertFrom-Json. For the time being I have worked around this by checking if my array item is not $null
I am using PS 5.1 for this.