Powershell nested json to repeating csv - json

Kind of similar to Convert nested JSON array into separate columns in CSV file but instead of flattened csv (ie discipline_01, discicpline_01) exporting to multiple lines of the csv:
{
"data": [{
"attributes": {
"id": 10011,
"title": "Test procedure",
"slug": "slug",
"url": "http://test.test",
"disciplines": [
"discipline_a",
"discipline_b",
"discipline_c"
]
}
}]
}
export to
"id","title","slug","url","discipline"
"10011","Test procedure","slug","http://test.test","discipline_a"
"10011","Test procedure","slug","http://test.test","discipline_b"
"10011","Test procedure","slug","http://test.test","discipline_c"
Thanks to Export fields with nested values from JSON to CSV I've gotten this far:
$foo = invoke-restmethod $restquery -headers $headers
$foo |
select -ExpandProperty data |
select -ExpandProperty attributes |
select id, title, slug, url, disciplines |
foreach {
$_.disciplines = $_disciplines -join ' '
$_ |
export-csv c:\outfile.csv -notypeinformation
This gives me
"10011","Test procedure","slug","http://test.test","discipline_a discipline_b discipline_c"
But no clue how to get it to:
"id","title","slug","url","discipline"
"10011","Test procedure","slug","http://test.test","discipline_a"
"10011","Test procedure","slug","http://test.test","discipline_b"
"10011","Test procedure","slug","http://test.test","discipline_c"
Any help is appreciated.

While your posted Json is invalid,
this might do:
## Q:\Test\2019\05\31\SO_56401395.ps1
$foo = invoke-restmethod $restquery -headers $headers
$Data = $foo.data.attributes | ForEach-Object {
foreach ($discipline in $_.disciplines}(
[PSCustomObject]#{
id = $_.id
title = $_.title
slug = $_.slug
url = $_.url
discipline = $discipline
}
}
}
$Data | Export-Csv c:\outfile.csv -NoTypeInformation

Related

Extracting fields to CSV from a JSON file with Powershell

i have a json extracted from a API exact like this:
{
"LPEONASVVAP0": {
"LPEONASVVAP0": {
"id": "urn:vcloud:vm:f526d27d-e0f9-4d4f-ae81-4824e397c027",
"name": "LPEONASVVAP0",
"description": "_vm_desc_",
"dateCreated": "2021-04-06T14:56:09.640+0000"
}
},
"WDEONDSVDIS6": {
"WDEONDSVDIS6": {
"id": "urn:vcloud:vm:7ed43492-a7ce-4963-b5bb-5ec2ca89477c",
"name": "WDEONDSVDIS6",
"description": "",
"dateCreated": "2021-04-13T13:44:29.973+0000"
}
},
"WDEONASVSTR0": {
"WDEONASVSTR0": {
"id": "urn:vcloud:vm:7afa34fe-b239-4abe-90df-3f270b44db1f",
"name": "WDEONASVSTR0",
"description": "",
"dateCreated": "2021-03-10T16:17:50.947+0000"
}
},
}
I need extract only fields id, name and description to create a csv with them. I test this but the output file is in blank:
$pathToJsonFile = x
$pathToOutputFile = x
$obj = Get-Content $pathToJsonFile -Raw | ConvertFrom-Json
print $obj
$obj | select id, name, description | Convertto-csv > $pathToOutputFile
You'll need to "discover" the parent property names (eg. 'LPEONASVVAP0') via the psobject hidden memberset. Since the outer and inner properties are named the same, we can re-use the name to get the inner property value:
$obj.psobject.Properties |ForEach-Object {
$_.Value.$($_.Name)
} |Select id,name,description |Export-Csv -NoTypeInformation -Path $pathToOutputFile
Edit: Mathias R. Jessens answer is better written than this, i would do it that way instead of how i posted.
Okay so i copied the json you posted, imported it. Since each array of information is stored like this
"WDEONDSVDIS6": {
"WDEONDSVDIS6": {
i used get-member to iterate each of the arrays and then select the info from that.
Also, you dont need to use Convertto-csv > $pathToOutputFile, use the export-csv command instead.
Below is my code how i would have done it, there is probably a better way but this works :)
$pathToOutputFile = x.csv
$obj = Get-Content example.json -Raw| ConvertFrom-Json
$obj2 = ($obj | Get-Member -MemberType noteproperty).Name
$result = foreach($item in $obj2){
$obj.$item.$item | select id,name,description
}
$result | Export-Csv -Path $pathToOutputFile -Encoding utf8 -NoTypeInformation

How to write a value in json file using if conditions in powershell?

I am trying to add value in key pair json file using Powershell. I am getting an error. Need some guidance on PS Scripting.
Powershell 5.1 version and json file with key pair inside it.
MYJSON File:
{ "Theme":{"Res_List": {
"Method": "POST",
"Name":""}}}
PowerShell Script:
$file_path="C:\Users\RelativeConfig.json"
$json = Get-Content $file_path | Out-String | ConvertFrom-Json
$timestamp = Get-Date -Format ddMMHHmm
$namevalue = 'resource'+$timestamp+'e3'
echo $namevalue ##prints resource16sep2019 I want this to write in json as value for Name Key
$files = $json | Get-Member -MemberType Properties | Select-Object - ExpandProperty Name
$result = Foreach ($file in $json)
{
if ($file.Name -eq "Name") { $file.Value = $namevalue}
}
$result | ConvertTo-Json | Set-Content $json
Error:
No error but json is not updated with new values.
Expected:
{ "Theme":{"Res_List": {
"Method": "POST",
"Name":"resource16sep2019 "}}}
Actual:
{ "Theme":{"Res_List": {
"Method": "POST",
"Name":""}}}
To update your value in json you can do something like below
$bucket_name = "resource16sep2019" ## I want this to write in json as value for Name Key
$json.Theme.Res_List.Name = $bucket_name
It will directly replace the $bucket_name value with the Name value in JSON.
If you want to write this to the respective file use the below command,
$json | ConvertTo-Json -depth 32| set-content $file_path
Hope it helps! Cheers

Nested json extract from powershell

I need to show id_products as a result, the remaining data I have a photo id and url pictures.
Below is the code which I extract data from
$files_ro = "products.csv"
$Ident = Import-Csv -Path $files_ro -Header id_product | select-object -skip 1
foreach ($idka in $ident)
{
$idp = $idka.id_product
$request_n = "http://api.url/"+ $idp +""
foreach($d1 in $request_n)
{
Invoke-WebRequest $d1 |
ConvertFrom-Json |
Select-Object -Expand data |
Select -expand extended_info |
select -expand images |
Select id,url
}
}
files
- product.csv
"id_product"
"21221"
"23526"
"23525"
"24074"
"21302"
"24372"
"21272"
"21783"
"27268"
"21776"
json
{
data: {
id: 21221,
extended_info: {
images: [
{
id: 34380,
url: photos1.jpg
},
{
id: 34381,
url: photos2.jpg
},
{
id: 34382,
url: photos3.jpg
}
],
}
}
}
I would like it to look like this:
id_product,id(images), url
21221,34380,photos1.jpg
21221,34381,photos2.jpg
21221,34382,photos3.jpg
You can help me somehow ?
Your provided JSON is not valid. However, I would use a PSCustomObject to create the desired result:
$json = #'
{
"data": {
"id": 21221,
"extended_info": {
"images": [{
"id": 34380,
"url": "photos1.jpg"
}, {
"id": 34381,
"url": "photos2.jpg"
}, {
"id": 34382,
"url": "photos3.jpg"
}
]
}
}
}
'# | ConvertFrom-Json
$json.data.extended_info.images | ForEach-Object {
[PSCustomObject]#{
id_product = $json.data.id
"id(images)" = $_.id
url = $_.url
}
}
Output:
id_product id(images) url
---------- ---------- ---
21221 34380 photos1.jpg
21221 34381 photos2.jpg
21221 34382 photos3.jpg
To convert the result to CSV, just add | ConvertTo-Csv -NoTypeInformation after the last curly bracket to get the following output:
"id_product","id(images)","url"
"21221","34380","photos1.jpg"
"21221","34381","photos2.jpg"
"21221","34382","photos3.jpg"
As your product.csv already has the same header it isn't necessary to supply one and then skip the first line.
Your single URL $request_n also doesn't need to be iterated with a foreach
I suggest to store the result from the webrequest and convertedfrom-json into var $Json and proceed with Martin Brandls good answer.
## Q:\Test\2019\05\24\SO_56287843.ps1
$files_ro = "products.csv"
$Ident = Import-Csv -Path $files_ro
$Data = foreach ($idka in $ident){
$request_n = "http://api.url/{0}" -f $idka.id_product
$Json = Invoke-WebRequest $request_n | ConvertFrom-Json
# inserted code from Martin Brandl's good answer
$Json.data.extended_info.images | ForEach-Object {
[PSCustomObject]#{
id_product = $json.data.id
"id(images)" = $_.id
url = $_.url
}
}
}
$Data
$Data | Export-Csv ProductImages.csv -NoTypeInformation
#$Data | Out-Gridview

Powershell JSON pipeline expand multiple values into one column csv

I'm trying to automate some data pipelines with Powershell, but I'm kinda stuck with converting a JSON list to a single cell per row in a CSV file. Hope some of you can help me out.
The JSON I get looks like the following:
{"result": [
{
"uid": "1",
"EducationHistory": []
},
{
"uid": "2",
"EducationHistory": []
},
{
"uid": "3",
"EducationHistory": []
},
{
"uid": "4",
"EducationHistory": {
"10466632": {
"euid": 10466632,
"degree": "Highschool",
"educationLevel": null
},
"10466634": {
"euid": 10466634,
"degree": "Law",
"educationLevel": "batchelor"
},
"10466635": {
"euid": 10466635,
"degree": "Law",
"educationLevel": "master"
}
}
},
{
"uid": "5",
"EducationHistory": {
"10482462": {
"euid": 10482462,
"degree": "IT",
"educationLevel": "master"
}
}
}
]
}
What I want to do is collect the educationLevels per uid in one column. So something like this:
uid | educationLevel
----+------------------
1 |
2 |
3 |
4 | barchelor, master
5 | master
Normally I would like Expandproperty to get down to a lower level, but this doesn't work for this case because every EducationHistory entry is behind a euid for that specific entry. Expanding every single one of them like in the example below isn't workable because of the number of records.
So I think I need something of a loop, but I don't know how. Hope you can help me. First post here and a Powershell newbie, so I hope my question is clear. Please let me know if you need more info.
The code for one entry, as example:
$json = Get-content -raw -path C:\TEMP\File.json
(ConvertFrom-Json -InputObject $json).result |
Select-Object uid,
#Expand one of the entries:
#{Name = "Edu.Level";E={$_.EducationHistory | Select-Object -
expandproperty 10466632 |Select-Object -expandpropert degree }} |
Format-Table
$content = Get-Content .\test.json
$result = ($content | ConvertFrom-Json).result
$totalResult = #()
foreach($res in $result) {
$tempArray = #()
if($res.EducationHistory -ne $null) {
$properties = $res.EducationHistory | Get-Member -MemberType NoteProperty
foreach($property in $properties) {
$eduLevel = $res.EducationHistory.$($property.Name).educationLevel
if(![String]::IsNullOrEmpty($eduLevel)) {
$tempArray += $eduLevel
}
}
}
$totalResult += [PSCustomObject]#{
uid = $res.uid
educationLevel = $tempArray -join ", "
}
}
$totalResult
This will output desired result for the input you have provided.
The trickiest part is the value of EducationHistory property. You have to use Get-Member cmdlet (see Get-Help Get-Member) to get the properties of the current object in loop. Then using the name of the property to access the educationLevel.
Your first question, my first answer I believe :) Similar to the last answer. You need to jump through the hoop of finding the object names in EducationalHistory to reference them.
$json = (Get-content C:\TEMP\File.json | ConvertFrom-Json).result
$results = #()
foreach ( $u in $json)
{
foreach ( $h in $u.EducationHistory)
{
$results += $h.PSObject.properties.Name | ForEach-Object{new-object PSObject -property #{ uid=$u.uid; degree=$h.$_.degree}}
}
}
$results | ConvertTo-Csv | select -skip 1

Loading file with array of hashtables failing

Leading on from a previous question, I've been using arrays and hashtables as variables to get values from nested data structures. e.g. Defining $fields and feeding it into a select operator:
$request = 'http://musicbrainz.org/ws/2/artist/5b11f4ce-a62d-471e-81fc-a69a8278c7da?inc=aliases&fmt=json'
$output = Invoke-WebRequest $request | ConvertFrom-Json
$fields = "name",
#{Name='begin'; Expression={$_.{life-span}.begin}},
#{Name='end'; Expression={$_.{life-span}.end}}
$output | select $fields
Giving me:
name begin end
---- ----- ---
Nirvana 1988-01 1994-04-05
However, if I defined the array as a JSON structure for importing (bandFields.json)
{
"fields": [
"name",
{
"Name": "begin",
"Expression": "{$_.{life-span}.begin}"
}, {
"Name": "end",
"Expression": "{$_.{life-span}.end}"
}
]
}
And try to import it a an array (similar to what I'd previously manually specified):
$request = 'http://musicbrainz.org/ws/2/artist/5b11f4ce-a62d-471e-81fc-a69a8278c7da?inc=aliases&fmt=json'
$output = Invoke-WebRequest $request | ConvertFrom-Json
$runConf = Get-Content -Raw -Path bandFields.json | ConvertFrom-Json
$fields = $runConf.fields
$output | select $fields
And I get this:
select : Cannot convert System.Management.Automation.PSObject to one of the following types {System.String, System.Management.Automation.ScriptBlock}.