Fetch data from JSON with powershell - json

I have without success tried to figure out how to most efficiently fetch data with powershell from the below JSON that is returned to me by a REST API:
{
"$schema": "api:standardResponse",
"links": [
{
"rel": "canonical",
"href": "http://localhost:8501/services/v2/replicats",
"mediaType": "application/json"
},
{
"rel": "self",
"href": "http://localhost:8501/services/v2/replicats",
"mediaType": "application/json"
},
{
"rel": "describedby",
"href": "http://localhost:8501/services/v2/metadata-catalog/replicats",
"mediaType": "application/schema+json"
}
],
"messages": [],
"response": {
"$schema": "ogg:collection",
"items": [
{
"links": [
{
"rel": "parent",
"href": "http://localhost:8501/services/v2/replicats",
"mediaType": "application/json"
},
{
"rel": "canonical",
"href": "http://localhost:8501/services/v2/replicats/RNIMDA00",
"mediaType": "application/json"
}
],
"$schema": "ogg:collectionItem",
"name": "RNIMDA00"
},
{
"links": [
{
"rel": "parent",
"href": "http://localhost:8501/services/v2/replicats",
"mediaType": "application/json"
},
{
"rel": "canonical",
"href": "http://localhost:8501/services/v2/replicats/RNIMDA01",
"mediaType": "application/json"
}
],
"$schema": "ogg:collectionItem",
"name": "RNIMDA01"
},
{
"links": [
{
"rel": "parent",
"href": "http://localhost:8501/services/v2/replicats",
"mediaType": "application/json"
},
{
"rel": "canonical",
"href": "http://localhost:8501/services/v2/replicats/RNIMDA02",
"mediaType": "application/json"
}
],
"$schema": "ogg:collectionItem",
"name": "RNIMDA02"
}
]
}
}
I only need the data from the "name": node, and the data from the "href": node.
I have done some searching and found som examples where the JSON is converted with ConvertFrom-Json and then iterated with a foreach like the pseudo code below:
$users = $response | ConvertFrom-Json
foreach ($user in $users)
{
write-host "$($user.name) has the email: $($user.email)"
}
But I wonder if there is a better way of fetching data with powershell from a object that contains JSON.
Thanks :)

It is not really clear what your desired output should be.
Let's assume you have the JSON data converted in a variable $json using $json = $response | ConvertFrom-Json
Then if you do:
$json.response.items | Select-Object name, #{Name = 'href'; Expression = {$_.links.href}}
You will get objects with a name and a href property, where the href will be an array.
PowerShell would output this on screen as
name href
---- ----
RNIMDA00 {http://localhost:8501/services/v2/replicats, http://localhost:8501/services/v2/replicats/RNIMDA00}
RNIMDA01 {http://localhost:8501/services/v2/replicats, http://localhost:8501/services/v2/replicats/RNIMDA01}
RNIMDA02 {http://localhost:8501/services/v2/replicats, http://localhost:8501/services/v2/replicats/RNIMDA02}
If however you would like to return an object for each of the href values inside the links nodes you can do:
$json.response.items | ForEach-Object {
$name = $_.name
foreach ($link in $_.links) {
[PsCustomObject]#{
name = $name
href = $link.href
}
}
}
which will output
name href
---- ----
RNIMDA00 http://localhost:8501/services/v2/replicats
RNIMDA00 http://localhost:8501/services/v2/replicats/RNIMDA00
RNIMDA01 http://localhost:8501/services/v2/replicats
RNIMDA01 http://localhost:8501/services/v2/replicats/RNIMDA01
RNIMDA02 http://localhost:8501/services/v2/replicats
RNIMDA02 http://localhost:8501/services/v2/replicats/RNIMDA02

jq is what you need.
Just install the windows version, add it to your PATH (depending on how you installed it) and use the following command:
curl -s "https://your-url" | jq -r '.response.items.links.href'

Related

Add nested records of JSON body into one column while exporting to CSV in powershell

I have JSON including multiple nested records. I want to add records with comma separated and store it in a CSV file.
JSON Body
{
"projectVitals": {
"productName": "Enterprise",
"name": "WhiteSource Bolt",
"token": "61a48eab05356f149828c0e",
"creationDate": "2022-10-17 09:08:46",
"lastUpdatedDate": "2023-01-25 06:37:32"
},
"libraries": [
{
"keyUuid": "a89b-40759d783dc3",
"keyId": 145110423,
"type": "NUGET_PACKAGE_MODULE",
"languages": "Nuget",
"references": {
"url": "https://api.nuget.org/packages/system.text.encodings.web.5.0.1.nupkg",
"homePage": "https://github.com/dotnet/runtime",
"genericPackageIndex": "https://api.nuget.org/packages/System.Text.Encodings.Web/5.0.1"
},
"matchType": "SHA1",
"sha1": "05cd84c678cddd1de0c",
"name": "system.text.encodings.web.5.0.1.nupkg",
"artifactId": "system.text.encodings.web.5.0.1.nupkg",
"version": "5.0.1",
"groupId": "System.Text.Encodings.Web",
"licenses": [
{
"name": "MIT",
"url": "http://www.opensource.org/licenses/MIT",
"profileInfo": {
"copyrightRiskScore": "THREE",
"patentRiskScore": "ONE",
"copyleft": "NO",
"royaltyFree": "YES"
},
"references": [
{
"referenceType": "NuGet package (details available in nuget gallery)",
"reference": "https://index.whitesourcesoftware.com/gri/app/reader/resource/content/asString/33131621-c9e5-4c87-ac1d-b988bbef1e0a"
}
]
}
],
"vulnerabilities": []
},
{
"keyUuid": "936f-5daddbcc37b2",
"keyId": 69037902,
"type": "DOT_NET_AS_GENERIC_RESOURCE",
"languages": ".NET",
"references": {
"url": "https://api.nuget.org/packages/system.runtime.interopservices.runtimeinformation.4.3.0.nupkg",
"genericPackageIndex": ""
},
"matchType": "SHA1",
"sha1": "32d3122a48aa379904",
"name": "System.Runtime.InteropServices.RuntimeInformation-4.6.24705.01.dll",
"artifactId": "System.Runtime.InteropServices.RuntimeInformation-4.6.24705.01.dll",
"version": "4.6.24705.01",
"groupId": "System.Runtime.InteropServices.RuntimeInformation",
"licenses": [
{
"name": "Microsoft .NET Library",
"url": "http://microsoft.com/web/webpi/eula/aspnetcomponent_rtw_enu.htm",
"riskLevel": "unknown",
"references": [
{
"referenceType": "Details available in GitHub repository",
"reference": "https://dot.net/"
},
{
"referenceType": "Details available in GitHub repository",
"reference": "https://dotnet.microsoft.com/"
}
]
},
{
"name": "MIT",
"url": "http://www.opensource.org/licenses/MIT",
"profileInfo": {
"copyrightRiskScore": "THREE",
"patentRiskScore": "ONE",
"copyleft": "NO",
"royaltyFree": "YES"
},
"references": [
{
"referenceType": "Details available in GitHub repository",
"reference": "https://dot.net/"
}
]
}
],
"vulnerabilities": []
}
]
}
Powershell Script
$pathToInputJsonFile = "C:\Users\abc\Downloads\test.json"
$pathToOutputCSVFile = "C:\Users\abc\Downloads\License3.csv"
$jsonFileContent = Get-Content -Raw -Path $pathToInputJsonFile | Out-String | ConvertFrom-Json
$libraries = $jsonFileContent.libraries
foreach($obj in $libraries)
{
$LibraryName = $obj.name
$LibraryVersion = $obj.version
$LibraryType = $obj.type
$LibraryLanguage = $obj.languages
$LibraryURL = $obj.references.url
$LicenseName = $obj.licenses.name
$LicenseURL = $obj.licenses.url
[PSCustomObject]#{
LibraryName = $LibraryName
LibraryVersion = $LibraryVersion
LibraryType = $LibraryType
LibraryLanguage = $LibraryLanguage
LibraryURL = $LibraryURL
LicenseName = $LicenseName
LicenseURL = $LicenseURL
} | Export-Csv $pathToOutputCSVFile -notype -Append
}
Actual Result
Expected Result
Use the -join operator to join 1 or more strings together with a given separator:
[PSCustomObject]#{
# ...
LicenseName = $LicenseName -join ', '
LicenseURL = $LicenseURL -join ', '
}

How to get the specific variable value from nested JSON based on the specific variable

Below is the JSON Output got for one of the API call, now need to get the date based on the name value, for example want to iterate the JSON file and look for name variable value which is equal to 1.0 then want to have date which is 2018-12-13T18:04:42-0500.
VERBOSE: {
"paging": {
"pageIndex": 1,
"pageSize": 100,
"total": 2
},
"analyses": [
{
"key": "xxxx",
"date": "2019-06-07T18:04:56-0400",
"events": [
{
"key": "xxxxxx",
"category": "VERSION",
"name": "01.00"
}
]
},
{
"key": "yyyyyy",
"date": "2018-12-13T18:04:42-0500",
"events": [
{
"key": "yyyyyy",
"category": "VERSION",
"name": "1.0"
}
]
}
]
}
Assuming you have the entire JSON text in a string variable, say $Json, would this work for you?
$obj = $Json | ConvertFrom-Json
$obj.analyses | ForEach-Object { if ($_.Events.Name -eq "1.0"){$_.Date}}

Nested Json to CSV powershell

I tried to solve it on my own but my knowlegde of PS and Json is just not so good that I can adapt other Solutions to my Problem.
The Tasks seems quite simple, but my json Structure which I get through a Rest API is kinda stupid for me.
Example JSON (which has more columns like (test-id,test-Name) but excatly the same logic over all columns in Fields):
{
"entities": [
{
"Fields": [
{
"Name": "test-id",
"values": [
{
"value": "1851"
}
]
},
{
"Name": "test-name",
"values": [
{
"value": "01_DUMMY"
}
]
}
],
"Type": "run",
"children-count": 0
},
{
"Fields": [
{
"Name": "test-id",
"values": [
{
"value": "1852"
}
]
},
{
"Name": "test-name",
"values": [
{
"value": "02_DUMMY"
}
]
}
],
"Type": "run",
"children-count": 0
}
],
"TotalResults": 2
}
I tried to following PS Script:
Get-Content $file -raw |
convertfrom-json | select -ExpandProperty entities |
select -ExpandProperty Fields |
select -ExpandProperty Values |
Export-CSV $OutputFile -NoTypeInformation
But my CSV Result Looks like this:
"value"
"1851"
"01_DUMMY"
"N"
I would like to receive following result:
test-id,test-Name,run,children-count
1851,01_DUMMY,run,0
1852,02_DUMMY,run,0

query when object name is a URL

I've written a JQ search that outputs the following, but I cannot work out how to get into the detail and extract specific information from this.
{
"https://www.example.org/rest/relation/node/recording/revision_uid": [
{
"_links": {
"self": {
"href": "https://www.example.org/user/37?_format=hal_json"
},
"type": {
"href": "https://www.example.org/rest/type/user/user"
}
},
"uuid": [
{
"value": "d40684cf-2321-42d2-9194"
}
]
}
],
"https://www.example.org/rest/relation/node/recording/uid": [
{
"_links": {
"self": {
"href": "https://www.example.org/user/37?_format=hal_json"
},
"type": {
"href": "https://www.example.org/rest/type/user/user"
}
},
"uuid": [
{
"value": "d40684cf-2321-42d2-9194"
}
],
"lang": "en"
}
],
"https://www.example.org/rest/relation/node/recording/field_category": [
{
"_links": {
"self": {
"href": "https://www.example.org/simplecategory?_format=hal_json"
},
"type": {
"href": "https://www.example.org/rest/type/taxonomy_term/tags"
}
},
"uuid": [
{
"value": "3fef93d5-926a-41aa-95cb"
}
]
}
],
"https://www.example.org/rest/relation/node/recording/field_part1_speaker": [
{
"_links": {
"self": {
"href": "https://www.example.org/by/speakername?_format=hal_json"
},
"type": {
"href": "https://www.example.org/rest/type/taxonomy_term/author"
}
},
"uuid": [
{
"value": "fb6c806f-fa6a-4aa0-89ef"
}
]
}
]
}
How do I write a query that returns 'https://www.example.org/simplecategory?_format=hal_json'?
And I'd then want a similar query that returns 'https://www.example.org/by/speakername?_format=hal_json'
So jq '._embedded' gets me the data above.
I've then tried various combinations to get to the value of https://www.example.org/rest/relation/node/recording/field_category.
- jq '._embedded.https://www.example.org/rest/relation/node/recording/field_category - but of course the URL has special characters in it.
jq .["https://www.example.org/rest/relation/node/recording/field_category"]
jq ."https://www.example.org/rest/relation/node/recording/field_category$"
I've also messed around with some of JQs built in functions, like flatten and to_entries, from_entries. I've also tried regular expressions but my efforts return Cannot iterate over null (null).
How do I write a query that returns 'https://www.example.org/simplecategory?_format=hal_json'?
If you want to specify the top-level key explicitly, the follow-on query would be:
.["https://www.example.org/rest/relation/node/recording/revision_uid"][]
| ._links.self.href
That is, the entire query would be:
._embedded
| .["https://www.example.org/rest/relation/node/recording/revision_uid"][]
| ._links.self.href
And I'd then want a similar query
An alternative to specifying the top-level key explicitly might be to select the href of interest from the array of all of them:
._embedded
| [.[][]._links.self.href]
This would yield:
[
"https://www.example.org/user/37?_format=hal_json",
"https://www.example.org/user/37?_format=hal_json",
"https://www.example.org/simplecategory?_format=hal_json",
"https://www.example.org/by/speakername?_format=hal_json"
]

Convert JSON to CSV - string manipulation (jq, bash, awk, sed, etc.)

I'm in a dire need of help for a script to basically convert JSON text to CSV text in an attempt to copy users from one AWS Cognito userpool to another.
The export JSON looks like this:
{
"Users": [
{
"Username": "user.name",
"Attributes": [
{
"Name": "sub",
"Value": "some-value"
},
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:jobtitle",
"Value": Director"
},
{
"Name": "custom:user_id",
"Value": "38"
},
{
"Name": "email",
"Value": "foo.bar#email.com"
}
],
"UserCreateDate": some-value,
"UserLastModifiedDate": some-value,
"Enabled": some-value,
"UserStatus": "some-value"
}
[more lines down here]...
] }
Then the CSV file would contain these lines:
,,,,,,,,,foo.bar#email.com,TRUE,,,,,,FALSE,,,Director,,38,FALSE,foo.bar
[more lines down here]...
So, the variables would be like this for JSON:
{
"Users": [
{
"Username": "%USERNAME%",
"Attributes": [
{
"Name": "sub",
"Value": "some-value"
},
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:jobtitle",
"Value": %JOB_TITLE%"
},
{
"Name": "custom:user_id",
"Value": "%USER_ID%"
},
{
"Name": "email",
"Value": %EMAIL%"
}
],
"UserCreateDate": some-value,
"UserLastModifiedDate": some-value,
"Enabled": some-value,
"UserStatus": "some-value"
}
...
]
}
And like this for CSV:
,,,,,,,,,%EMAIL%,TRUE,,,,,,FALSE,,,%JOB_TITLE%,,%USER_ID%,FALSE,%USERNAME%
where %EMAIL%, %JOB_TITLE%, %USER_ID%, and %USERNAME% are variables, everything else should be just string.
Appreciate your help in advanced guys.
Consider first this filter:
.Users[].Attributes
| map(select(.Name | . == "custom:jobtitle" or . == "custom:user_id" or . == "email") )
| from_entries
| [ .email, .["custom:jobtitle"], .["custom:user_id"] ]
| #csv
The trick used here is the use of from_entries to convert the array of Name/Value pairs to an object with the Names as keys.
Assuming valid JSON input along the lines shown in the Q, invoking jq with the -r option would yield:
"foo.bar#email.com","Director","38"
Unfortunately the precise requirements are not so clear to me, but you should be able to adapt the above in accordance with your needs.