ConvertFrom-Json "add" null items to array - json

I am working with a file that has a JSON string in it.
{
"ConfigurationItems": [{
"ActiveDirectory": [{
"Users": [
{ "FirstName": "U1", "LastName": "L1", "Department": "D1", "UserName": "UN1" },
{ "FirstName": "U2", "LastName": "L2", "Department": "D2", "UserName": "UN2" },
{ "FirstName": "U3", "LastName": "L3", "Department": "D3", "UserName": "UN3" },
{ "FirstName": "U4", "LastName": "L4", "Department": "D4", "UserName": "UN4" },
{ "FirstName": "U5", "LastName": "L5", "Department": "D5", "UserName": "UN5" }
]
},
{
"Groups": [
{ "Name": "G1", "Scope": "Global" },
{ "Name": "G2", "Scope": "Global" },
{ "Name": "G3", "Scope": "Global" }
]
},
{
"OU": [
{ "Name": "N1" },
{ "Name": "N2" },
{ "Name": "N3" }
]
}
]
}]
}
I assign this to a variable by doing the following:
$t = Get-content -path $pathtoJSON -raw | ConvertFrom-Json
Now comes the part I cannot explain if I get the count of Users it returns 7
$t.ConfigurationItems.ActiveDirectory.Users.Count
The main issue is when I iterate through this, I will always end up with two $null items in my loop, which throws off my code. How do I ensure I get the correct count from ConvertFrom-Json. For the time being I have worked around this by checking if my array item is not $null
I am using PS 5.1 for this.

Related

Cannot get jq to query json object [duplicate]

This question already has answers here:
How to use jq when the variable has reserved characters?
(3 answers)
Closed 6 months ago.
I have a JSON file that I am trying to query with jq. I am unable to retrieve the observations. I am trying to retieve each of the "observations using the following command and not able to get to the result:
cat sample3.json | jq .dataSets[0].series.0:0:0:0:0.observations.0[0]
I am able to retieve up to the series using:
cat sample3.json | jq .dataSets[0].series
But once I try to drill down further I am getting a compile error:
$ cat sample3.json | jq .dataSets[0].series.0:0:0:0:0
jq: error: syntax error, unexpected LITERAL, expecting end of file (Unix shell quoting issues?) at <top-level>, line 1:
.dataSets[0].series.0:0:0:0:0
jq: 1 compile error
I am not sure what I am doing wrong here....
The input file is:
{
"header": {
"id": "b8be2cd5-33bf-4687-9e81-eb032f6f8a71",
"test": false,
"prepared": "2022-09-01T13:30:57.013+02:00",
"sender": {
"id": "ECB"
}
},
"dataSets": [
{
"action": "Replace",
"validFrom": "2022-09-01T13:30:57.013+02:00",
"series": {
"0:0:0:0:0": {
"attributes": [
0,
null,
0,
null,
null,
null,
null,
null,
null,
null,
null,
null,
0,
null,
0,
null,
0,
0,
0,
0
],
"observations": {
"0": [
1.4529,
0,
0,
null,
null
],
"1": [
1.4472,
0,
0,
null,
null
],
"2": [
1.4591,
0,
0,
null,
null
]
}
}
}
}
],
"structure": {
"links": [
{
"title": "Exchange Rates",
"rel": "dataflow",
"href": "https://sdw-wsrest.ecb.europa.eu:443/service/dataflow/ECB/EXR/1.0"
}
],
"name": "Exchange Rates",
"dimensions": {
"series": [
{
"id": "FREQ",
"name": "Frequency",
"values": [
{
"id": "D",
"name": "Daily"
}
]
},
{
"id": "CURRENCY",
"name": "Currency",
"values": [
{
"id": "AUD",
"name": "Australian dollar"
}
]
},
{
"id": "CURRENCY_DENOM",
"name": "Currency denominator",
"values": [
{
"id": "EUR",
"name": "Euro"
}
]
},
{
"id": "EXR_TYPE",
"name": "Exchange rate type",
"values": [
{
"id": "SP00",
"name": "Spot"
}
]
},
{
"id": "EXR_SUFFIX",
"name": "Series variation - EXR context",
"values": [
{
"id": "A",
"name": "Average"
}
]
}
],
"observation": [
{
"id": "TIME_PERIOD",
"name": "Time period or range",
"role": "time",
"values": [
{
"id": "2022-08-29",
"name": "2022-08-29",
"start": "2022-08-29T00:00:00.000+02:00",
"end": "2022-08-29T23:59:59.999+02:00"
},
{
"id": "2022-08-30",
"name": "2022-08-30",
"start": "2022-08-30T00:00:00.000+02:00",
"end": "2022-08-30T23:59:59.999+02:00"
},
{
"id": "2022-08-31",
"name": "2022-08-31",
"start": "2022-08-31T00:00:00.000+02:00",
"end": "2022-08-31T23:59:59.999+02:00"
}
]
}
]
},
"attributes": {
"series": [
{
"id": "TIME_FORMAT",
"name": "Time format code",
"values": [
{
"name": "P1D"
}
]
},
{
"id": "BREAKS",
"name": "Breaks",
"values": []
},
{
"id": "COLLECTION",
"name": "Collection indicator",
"values": [
{
"id": "A",
"name": "Average of observations through period"
}
]
},
{
"id": "COMPILING_ORG",
"name": "Compiling organisation",
"values": []
},
{
"id": "DISS_ORG",
"name": "Data dissemination organisation",
"values": []
},
{
"id": "DOM_SER_IDS",
"name": "Domestic series ids",
"values": []
},
{
"id": "PUBL_ECB",
"name": "Source publication (ECB only)",
"values": []
},
{
"id": "PUBL_MU",
"name": "Source publication (Euro area only)",
"values": []
},
{
"id": "PUBL_PUBLIC",
"name": "Source publication (public)",
"values": []
},
{
"id": "UNIT_INDEX_BASE",
"name": "Unit index base",
"values": []
},
{
"id": "COMPILATION",
"name": "Compilation",
"values": []
},
{
"id": "COVERAGE",
"name": "Coverage",
"values": []
},
{
"id": "DECIMALS",
"name": "Decimals",
"values": [
{
"id": "4",
"name": "Four"
}
]
},
{
"id": "NAT_TITLE",
"name": "National language title",
"values": []
},
{
"id": "SOURCE_AGENCY",
"name": "Source agency",
"values": [
{
"id": "4F0",
"name": "European Central Bank (ECB)"
}
]
},
{
"id": "SOURCE_PUB",
"name": "Publication source",
"values": []
},
{
"id": "TITLE",
"name": "Title",
"values": [
{
"name": "Australian dollar/Euro"
}
]
},
{
"id": "TITLE_COMPL",
"name": "Title complement",
"values": [
{
"name": "ECB reference exchange rate, Australian dollar/Euro, 2:15 pm (C.E.T.)"
}
]
},
{
"id": "UNIT",
"name": "Unit",
"values": [
{
"id": "AUD",
"name": "Australian dollar"
}
]
},
{
"id": "UNIT_MULT",
"name": "Unit multiplier",
"values": [
{
"id": "0",
"name": "Units"
}
]
}
],
"observation": [
{
"id": "OBS_STATUS",
"name": "Observation status",
"values": [
{
"id": "A",
"name": "Normal value"
}
]
},
{
"id": "OBS_CONF",
"name": "Observation confidentiality",
"values": [
{
"id": "F",
"name": "Free"
}
]
},
{
"id": "OBS_PRE_BREAK",
"name": "Pre-break observation value",
"values": []
},
{
"id": "OBS_COM",
"name": "Observation comment",
"values": []
}
]
}
}
}
The .foo syntax cannot be used if the key name has anything but alphanumeric characters or the underscore, or if the first character of the key name is numeric.
Assuming you are using a recent version of jq,
you can always use the form: ."foo", which is actually an abbreviation of the basic form, .["foo"].
So assuming you're using a sufficiently recent version of jq, your query could begin with:
.dataSets[0].series."0:0:0:0:0"
If you are presenting the jq query on a command line, then you may have to escape the double-quotes appropriately, e.g. in a bash shell, by enclosing the jq query in single-quotes.

How to exclude data from json file to convert into csv file powershell

I want to convert my json file into csv -
{
"count": 28,
"value": [
{
"commitId": "65bb6a911872c314a9225815007d74a",
"author": {
"name": "john doe",
"email": "john.doe#gmail.com",
"date": "2020-06-09T17:03:33Z"
},
"committer": {
"name": "john doe",
"email": "john.doe#gmail.com",
"date": "2020-06-09T17:03:33Z"
},
"comment": "Merge pull request 3 from dev into master",
"changeCounts": {
"Add": 6,
"Edit": 0,
"Delete": 0
},
"url": "https://dev.azure.com/",
"remoteUrl": "https://dev.azure.com/"
},
{
"commitId": "bdcb4a1e1e671c15333eacc31aa9795fe32",
"author": {
"name": "john doe",
"email": "john.doe#gmail.com",
"date": "2020-06-09T17:03:33Z"
},
"committer": {
"name": "john doe",
"email": "john.doe#gmail.com",
"date": "2020-06-09T17:03:33Z"
},
"comment": "Updated FGDH",
"changeCounts": {
"Add": 0,
"Edit": 1,
"Delete": 0
},
"url": "https://dev.azure.com/",
"remoteUrl": "https://dev.azure.com/"
}
]
}
I don't want all the fields in my CSV file. I want commitid,commiter.name, commiter.date and comment only.
Get-Content "\commitinfo.json" -Raw | ConvertFrom-Json | Select -Expand value |Export-Csv
"\commitinfo.csv"
with this I get all data , How can I get selected data ?
Another json file -
{
"value": [
{
"id": "5c264dd2-bbcf-4537-8a63-19a0a4d2dc",
"name": "Develop",
"url": "https://dev.azure.com/",
"project": {
"id": "0042dc5c-fd13-4e3c-bfd7-7feb52e287",
"name": "test",
"url": "https://dev.azure.com/",
"state": "wellFormed",
"revision": 11,
"visibility": "private",
"lastUpdateTime": "2020-04-15T04:04:30.01Z"
},
"defaultBranch": "refs/heads/master",
"size": 55438,
"remoteUrl": "https://dev.azure.com",
"sshUrl": "git#ssh.dev.azure.com:v3",
"webUrl": "https://dev.azure.com/"
},
{
"id": "3219e8e2-281d-40ad-81c8-1cecf8",
"name": "automation",
"url": "https://dev.azure.com/",
"project": {
"id": "0e2df786-94a0-42c8-b068-c2656b",
"name": "automation",
"url": "https://dev.azure.com/",
"state": "wellFormed",
"revision": 19,
"visibility": "private",
"lastUpdateTime": "2020-05-02T03:32:50.937Z"
},
"size": 0,
"remoteUrl": "https://dev.azure.com/",
"sshUrl": "git#ssh.dev.azure.com:v3",
"webUrl": "https://dev.azure.com/"
}
]
}
from this file I only want to select projects with size - 0 , CSV should look like this
name size
a 0
b 0
Note: Piping the commands below to Export-Csv is omitted for brevity.
You need to use Select-Object with calculated properties:
(Get-Content commitinfo.json -Raw | ConvertFrom-Json).value |
Select-Object commitid,
#{n='committerName'; e={ $_.committer.name } },
#{n='committerDate'; e={ $_.committer.date } },
comment
In the second case, add Where-Object to filter in the objects of interest:
(Get-Content other.json -Raw | ConvertFrom-Json).value |
Where-Object size -eq 0 |
Select-Object name, size

POWERSHELL - How to access multilevel child elements in JSON file with condtion

can someone please send me solution or link for PowerShell 5 and 7 how can I access child elements if specific condition is fulfilled for JSON file which I have as output.json. I haven't find it on the net.
I want to retrieve value of the "children" elements if type element has value FILE and to put that into some list. So final result should be [test1.txt,test2.txt]
Thank you!!!
{
"path": {
"components": [
"Packages"
],
"parent": "",
"name": "Packages",
},
"children": {
"values": [
{
"path": {
"components": [
"test1.txt"
],
"parent": "",
"name": "test1.txt",
},
"type": "FILE",
"size": 405
},
{
"path": {
"components": [
"test2.txt"
],
"parent": "",
"name": "test2.txt",
},
"type": "FILE",
"size": 409
},
{
"path": {
"components": [
"FOLDER"
],
"parent": "",
"name": "FOLDER",
},
"type": "DIRECTORY",
"size": 1625
}
]
"start": 0
}
}
1.) The json is incorrect, I assumt that this one is the correct one:
{
"path": {
"components": [
"Packages"
],
"parent": "",
"name": "Packages"
},
"children": {
"values": [
{
"path": {
"components": [
"test1.txt"
],
"parent": "",
"name": "test1.txt"
},
"type": "FILE",
"size": 405
},
{
"path": {
"components": [
"test2.txt"
],
"parent": "",
"name": "test2.txt"
},
"type": "FILE",
"size": 409
},
{
"path": {
"components": [
"FOLDER"
],
"parent": "",
"name": "FOLDER"
},
"type": "DIRECTORY",
"size": 1625
}
],
"start": 0
}
}
2.) The structure is not absolute clear, but for your example this seems to me to be the correct solution:
$element = $json | ConvertFrom-Json
$result = #()
$element.children.values | foreach {
if ($_.type -eq 'FILE') { $result += $_.path.name }
}
$result | ConvertTo-Json
Be aware, that the used construct $result += $_.path.name is fine if you have up to ~10k items, but for very large items its getting very slow and you need to use an arraylist. https://adamtheautomator.com/powershell-arraylist/

Combining Nested Json using PowerShell

I have the following Json script:
{
"merchant_info": {
"email": "merchant#example.com",
"first_name": "David",
"last_name": "Larusso",
"business_name": "Mitchell & Murray",
"phone": {
"country_code": "001",
"national_number": "4085551234"
},
"address": {
"line1": "1234 First Street",
"city": "Anytown",
"state": "CA",
"postal_code": "98765",
"country_code": "US"
}
},
"billing_info": [{
"email": "bill-me#example.com",
"first_name": "Stephanie",
"last_name": "Meyers"
}
],
"shipping_info": {
"first_name": "Stephanie",
"last_name": "Meyers",
"address": {
"line1": "1234 Main Street",
"city": "Anytown",
"state": "CA",
"postal_code": "98765",
"country_code": "US"
}
},
"items": [{
"name": "Zoom System wireless headphones",
"quantity": 2,
"unit_price": {
"currency": "USD",
"value": "120"
},
"tax": {
"name": "Tax",
"percent": 8
}
}, {
"name": "Bluetooth speaker",
"quantity": 1,
"unit_price": {
"currency": "USD",
"value": "145"
},
"tax": {
"name": "Tax",
"percent": 8
}
}
],
"discount": {
"percent": 1
},
"shipping_cost": {
"amount": {
"currency": "USD",
"value": "10"
}
},
"note": "Thank you for your business.",
"terms": "No refunds after 30 days."
}
And I want to use PowerShell to get the following Record and export it to CSV:
So far I created the following Script:
$JsonFile = "C:\Users\me\Documents\myfile.json"
$OutputFile = "C:\Users\me\Documents\newtext.csv"
Get-Content -Path $OutputFile
$json = ConvertFrom-Json (Get-Content $JsonFile -Raw)
$json.merchant_info | Select "first_name","last_name",#{Label = "phone"; Expression = {$_.phone.national_number}} |
Export-Csv $OutputFile -NoTypeInformation
I am able to bring values from (Merchant_info, Shipping_info, item) separetely but how do I bring it all in combined like in my screen shot above.
but how do I bring it all in combined like in my screen shot above.
We can only guess; assuming this entire json block is one order, with one merchant and one customer, but multiple items, then each row is an item. So start with that as the input:
Create an output record (PSCustomObject) with the repeated data, and then the individual item data:
$json.items | ForEach-Object {
[PSCustomObject]#{
MerchantInfoFirstName = $json.merchant_info.first_name
MerchantInfoLastName = $json.merchant_info.last_name
MerchantInfoPhoneNumber = $json.merchant_info.phone.national_number
ShippingInfoFirstName = $json.shipping_info.first_name
ShippingInfoLastName = $json.shipping_info.last_name
ItemName = $_.name
ItemQuantity = $_.quantity
}
} | Export-Csv ... etc.

Json file editing through jq

I have json
{
"file1": [{
"username": "myname",
"groupname": "mypassword",
"environment": [{
"name": "UMASK",
"value": "022"
},
{
"name": "DEBUG",
"value": "2"
}]
}]
}
and want to change the value of DEBUG to 5.
Tried with below command
jq .file1[0].environment sandeep.json |jq '(.[] |select(.name ==
"DEBUG") | .value) |= "5"'
this will return me specific portion of json like
[
{
"name": "UMASK",
"value": "022"
},
{
"name": "DEBUG",
"value": "5"
}
]
but I want to see full json with changed value
{
"file1": [{
"username": "myname",
"groupname": "mypassword",
"environment": [{
"name": "UMASK",
"value": "022"
},
{
"name": "DEBUG",
"value": "5"
}]
}]
}
Please suggest me
It should be:
jq '(.file1[].environment[]|select(.name=="DEBUG").value) |= 5' file.json
Output:
{
"file1": [
{
"username": "myname",
"groupname": "mypassword",
"environment": [
{
"name": "UMASK",
"value": "022"
},
{
"name": "DEBUG",
"value": 5
}
]
}
]
}