Nested json extract from powershell - json

I need to show id_products as a result, the remaining data I have a photo id and url pictures.
Below is the code which I extract data from
$files_ro = "products.csv"
$Ident = Import-Csv -Path $files_ro -Header id_product | select-object -skip 1
foreach ($idka in $ident)
{
$idp = $idka.id_product
$request_n = "http://api.url/"+ $idp +""
foreach($d1 in $request_n)
{
Invoke-WebRequest $d1 |
ConvertFrom-Json |
Select-Object -Expand data |
Select -expand extended_info |
select -expand images |
Select id,url
}
}
files
- product.csv
"id_product"
"21221"
"23526"
"23525"
"24074"
"21302"
"24372"
"21272"
"21783"
"27268"
"21776"
json
{
data: {
id: 21221,
extended_info: {
images: [
{
id: 34380,
url: photos1.jpg
},
{
id: 34381,
url: photos2.jpg
},
{
id: 34382,
url: photos3.jpg
}
],
}
}
}
I would like it to look like this:
id_product,id(images), url
21221,34380,photos1.jpg
21221,34381,photos2.jpg
21221,34382,photos3.jpg
You can help me somehow ?

Your provided JSON is not valid. However, I would use a PSCustomObject to create the desired result:
$json = #'
{
"data": {
"id": 21221,
"extended_info": {
"images": [{
"id": 34380,
"url": "photos1.jpg"
}, {
"id": 34381,
"url": "photos2.jpg"
}, {
"id": 34382,
"url": "photos3.jpg"
}
]
}
}
}
'# | ConvertFrom-Json
$json.data.extended_info.images | ForEach-Object {
[PSCustomObject]#{
id_product = $json.data.id
"id(images)" = $_.id
url = $_.url
}
}
Output:
id_product id(images) url
---------- ---------- ---
21221 34380 photos1.jpg
21221 34381 photos2.jpg
21221 34382 photos3.jpg
To convert the result to CSV, just add | ConvertTo-Csv -NoTypeInformation after the last curly bracket to get the following output:
"id_product","id(images)","url"
"21221","34380","photos1.jpg"
"21221","34381","photos2.jpg"
"21221","34382","photos3.jpg"

As your product.csv already has the same header it isn't necessary to supply one and then skip the first line.
Your single URL $request_n also doesn't need to be iterated with a foreach
I suggest to store the result from the webrequest and convertedfrom-json into var $Json and proceed with Martin Brandls good answer.
## Q:\Test\2019\05\24\SO_56287843.ps1
$files_ro = "products.csv"
$Ident = Import-Csv -Path $files_ro
$Data = foreach ($idka in $ident){
$request_n = "http://api.url/{0}" -f $idka.id_product
$Json = Invoke-WebRequest $request_n | ConvertFrom-Json
# inserted code from Martin Brandl's good answer
$Json.data.extended_info.images | ForEach-Object {
[PSCustomObject]#{
id_product = $json.data.id
"id(images)" = $_.id
url = $_.url
}
}
}
$Data
$Data | Export-Csv ProductImages.csv -NoTypeInformation
#$Data | Out-Gridview

Related

Trouble pasring nested values from json files using powershell

Trying to pull the signing_certificate_id field from the below json snippet, but unfortunately, the data is not making it into the excel export. The other info is making it into the csv. (there are cases where these code blocks exist twice in the same json to represent a separate environment.
Json Snippet:
{
"sfg_ping::production::standard_sp_connections": [
{
"name": "hhhh",
"entity_id": "https://hhhh.hhhh.com",
"contact_info": "hhhh#hhhh.com",
"sp_endpoint": "https://hhhh.hhhh.com/",
"sso_initialization_and_signature_policy": {
"sign_assertion_or_response": "assertion",
"sp_trust_model": "unanchored",
"signing_certificate_id": "gggggg",
"sp_initiated": {
"require_signed_authn_requests": false
}
},
Script being used:
Get-ChildItem "C:\Users\hhhh\appdev\targetfolder-json2" -Filter *.json -PipelineVariable file | ForEach-Object {
(Get-Content $_.pspath -Raw | ConvertFrom-Json).PSObject.Properties.Value | Select-Object #(
#{ l = 'filename'; e = { $file.Name }}, 'Name','contact_info', 'signing_certificate_id'
)
} | Export-Csv path\to\myexport.csv -NoTypeInformation
You need to create a calculated property for getting the nested JSON property, similar to what you did for filename:
Get-ChildItem "C:\Users\hhhh\appdev\targetfolder-json2" -Filter *.json -PipelineVariable file | ForEach-Object {
(Get-Content $_.pspath -Raw | ConvertFrom-Json).PSObject.Properties.Value | Select-Object #(
#{ l = 'filename'; e = { $file.Name }}, 'Name','contact_info',
#{ l = 'signing_certificate_id'; e = { $_.sso_initialization_and_signature_policy.signing_certificate_id }}
)
} | Export-Csv path\to\myexport.csv -NoTypeInformation

Extracting fields to CSV from a JSON file with Powershell

i have a json extracted from a API exact like this:
{
"LPEONASVVAP0": {
"LPEONASVVAP0": {
"id": "urn:vcloud:vm:f526d27d-e0f9-4d4f-ae81-4824e397c027",
"name": "LPEONASVVAP0",
"description": "_vm_desc_",
"dateCreated": "2021-04-06T14:56:09.640+0000"
}
},
"WDEONDSVDIS6": {
"WDEONDSVDIS6": {
"id": "urn:vcloud:vm:7ed43492-a7ce-4963-b5bb-5ec2ca89477c",
"name": "WDEONDSVDIS6",
"description": "",
"dateCreated": "2021-04-13T13:44:29.973+0000"
}
},
"WDEONASVSTR0": {
"WDEONASVSTR0": {
"id": "urn:vcloud:vm:7afa34fe-b239-4abe-90df-3f270b44db1f",
"name": "WDEONASVSTR0",
"description": "",
"dateCreated": "2021-03-10T16:17:50.947+0000"
}
},
}
I need extract only fields id, name and description to create a csv with them. I test this but the output file is in blank:
$pathToJsonFile = x
$pathToOutputFile = x
$obj = Get-Content $pathToJsonFile -Raw | ConvertFrom-Json
print $obj
$obj | select id, name, description | Convertto-csv > $pathToOutputFile
You'll need to "discover" the parent property names (eg. 'LPEONASVVAP0') via the psobject hidden memberset. Since the outer and inner properties are named the same, we can re-use the name to get the inner property value:
$obj.psobject.Properties |ForEach-Object {
$_.Value.$($_.Name)
} |Select id,name,description |Export-Csv -NoTypeInformation -Path $pathToOutputFile
Edit: Mathias R. Jessens answer is better written than this, i would do it that way instead of how i posted.
Okay so i copied the json you posted, imported it. Since each array of information is stored like this
"WDEONDSVDIS6": {
"WDEONDSVDIS6": {
i used get-member to iterate each of the arrays and then select the info from that.
Also, you dont need to use Convertto-csv > $pathToOutputFile, use the export-csv command instead.
Below is my code how i would have done it, there is probably a better way but this works :)
$pathToOutputFile = x.csv
$obj = Get-Content example.json -Raw| ConvertFrom-Json
$obj2 = ($obj | Get-Member -MemberType noteproperty).Name
$result = foreach($item in $obj2){
$obj.$item.$item | select id,name,description
}
$result | Export-Csv -Path $pathToOutputFile -Encoding utf8 -NoTypeInformation

How to Get the values from json using powershell

Guys this is my JSON file and I want to create a PowerShell script which will give me result like
I have used method like Get-Content and other but there are some issues with the JSON parsing. Please find what is my requirement I have explained in details below.
MyLocalMachineHome
LocalMachine = Sahil_LocalMachine
Second_MyLocalMachine = Sahil_MylocalMachine
Second_MyLocalMachine = ""
Staging
Second_Staging = Sahil;_Secconf
Staging = Sahil_Staging
third_staging = stsajiii
There is also one functionality which I would like to have if I want to get only variables of "staging".
I was using this function Get-Content -Raw -Path E:\shell\Powershell\1ReleasePipelines.json | ConvertFrom-Json | select -ExpandProperty variables on my original JSON file but somehow there is some kind of limit in storing string which I was getting from this method.
{
"environments": [
{
"id": 3,
"name": "MyLocalMachineHome",
"variableGroups": [],
"variables": {
"LocalMachine": {
"value": "Sahil_LocalMachine"
},
"Second_MyLocalMachine": {
"value": "Sahil_MylocalMachine"
},
"thirf_mylocal": {
"value": ""
}
}
},
{
"id": 7,
"name": "Staging",
"variableGroups": [],
"variables": {
"Second_Staging": {
"value": "Sahil;_Secconf"
},
"Staging": {
"value": "Sahil_Staging"
},
"third_staging": {
"value": "stsajiii"
}
}
}
]
}
If we assume that $json contains your JSON content, you can do the following ugly code:
$environment = 'staging'
$j = $json | ConvertFrom-Json
($j.environments | where name -eq $environment).variables | Foreach-Object {
$CurrentObject = $_
$CurrentObject | Get-Member -MemberType NoteProperty |
Select-Object -Expand Name | Foreach-Object {
$CurrentObject.$_.value
}
}
It appears your issue is that you don't know what variables are going to be contained within your JSON. So you can't easily use Select-Object variable or $object.variable. You need a dynamic approach.
If you know your variables ahead of time, things become simpler. You can store your variable names in an array and loop over them.
$variables = 'Second_Staging','Staging','third_staging'
$environment = 'staging'
$j = $json | ConvertFrom-Json
$jsonVars = ($j.environments | where name -eq $environment).variables
$variables | Foreach-Object {
$jsonVars.$_.value
}
View all the sub-properties of variables with format-list instead of format-table. Since the properties vary, format-table won't show all of them. There's a lot of sloppy object construction in json.
$a = get-content file.json
$a.environments.variables | format-table
LocalMachine Second_MyLocalMachine thirf_mylocal
------------ --------------------- -------------
#{value=Sahil_LocalMachine} #{value=Sahil_MylocalMachine} #{value=}
$a.environments.variables | format-list
LocalMachine : #{value=Sahil_LocalMachine}
Second_MyLocalMachine : #{value=Sahil_MylocalMachine}
thirf_mylocal : #{value=}
Second_Staging : #{value=Sahil;_Secconf}
Staging : #{value=Sahil_Staging}
third_staging : #{value=stsajiii}
Get the staging variables?
$a.environments | where name -eq staging | foreach variables
Second_Staging Staging third_staging
-------------- ------- -------------
#{value=Sahil;_Secconf} #{value=Sahil_Staging} #{value=stsajiii}
cls
start-transcript -path 'C:\E\Devops\PowerShell_Chapters\ABC.txt'
write-output "**********Variables of Release************"
get-content -raw -path 'C:\E\Devops\PowerShell_Chapters\Release.json'| Convertfrom-Json | Select -ExpandProperty variables
$json = get-content -raw -path 'C:\E\Devops\PowerShell_Chapters\Release.json'| Convertfrom-Json | Select -ExpandProperty environments
$EnvirnomentsVariables = get-content -raw -path 'C:\E\Devops\PowerShell_Chapters\Release.json'| Convertfrom-Json | Select -ExpandProperty environments |Select -ExpandProperty name
$ReleaseVariable = get-content -raw -path 'C:\E\Devops\PowerShell_Chapters\Release.json'| Convertfrom-Json | Select -ExpandProperty environments |Select -ExpandProperty variables
$i = 0
foreach($a in $EnvirnomentsVariables)
{
$ABC_Staging = $EnvirnomentsVariables[$i]
#write-output $ABC_Staging
if( $ABC_Staging -match "ABC Staging")
{
write-output "****************Variables of " $EnvirnomentsVariables[$i]*************"
#add-content 'C:\E\Devops\PowerShell_Chapters\ABC.txt' $EnvirnomentsVariables[$i]
# Set-content -path 'C:\E\Devops\PowerShell_Chapters\Sahil.json'| ConvertTo-Json | select $EnvirnomentsVariables[$i]
write-output $ReleaseVariable[$i]
# add-content 'C:\E\Devops\PowerShell_Chapters\ABC.txt' $ReleaseVariable[$i]
# Set-content -path 'C:\E\Devops\PowerShell_Chapters\Sahil.json'| ConvertTo-Json | select $ReleaseVariable[$i]
}
$i = $i + 1
}
stop-transcript

Powershell nested json to repeating csv

Kind of similar to Convert nested JSON array into separate columns in CSV file but instead of flattened csv (ie discipline_01, discicpline_01) exporting to multiple lines of the csv:
{
"data": [{
"attributes": {
"id": 10011,
"title": "Test procedure",
"slug": "slug",
"url": "http://test.test",
"disciplines": [
"discipline_a",
"discipline_b",
"discipline_c"
]
}
}]
}
export to
"id","title","slug","url","discipline"
"10011","Test procedure","slug","http://test.test","discipline_a"
"10011","Test procedure","slug","http://test.test","discipline_b"
"10011","Test procedure","slug","http://test.test","discipline_c"
Thanks to Export fields with nested values from JSON to CSV I've gotten this far:
$foo = invoke-restmethod $restquery -headers $headers
$foo |
select -ExpandProperty data |
select -ExpandProperty attributes |
select id, title, slug, url, disciplines |
foreach {
$_.disciplines = $_disciplines -join ' '
$_ |
export-csv c:\outfile.csv -notypeinformation
This gives me
"10011","Test procedure","slug","http://test.test","discipline_a discipline_b discipline_c"
But no clue how to get it to:
"id","title","slug","url","discipline"
"10011","Test procedure","slug","http://test.test","discipline_a"
"10011","Test procedure","slug","http://test.test","discipline_b"
"10011","Test procedure","slug","http://test.test","discipline_c"
Any help is appreciated.
While your posted Json is invalid,
this might do:
## Q:\Test\2019\05\31\SO_56401395.ps1
$foo = invoke-restmethod $restquery -headers $headers
$Data = $foo.data.attributes | ForEach-Object {
foreach ($discipline in $_.disciplines}(
[PSCustomObject]#{
id = $_.id
title = $_.title
slug = $_.slug
url = $_.url
discipline = $discipline
}
}
}
$Data | Export-Csv c:\outfile.csv -NoTypeInformation

Powershell JSON pipeline expand multiple values into one column csv

I'm trying to automate some data pipelines with Powershell, but I'm kinda stuck with converting a JSON list to a single cell per row in a CSV file. Hope some of you can help me out.
The JSON I get looks like the following:
{"result": [
{
"uid": "1",
"EducationHistory": []
},
{
"uid": "2",
"EducationHistory": []
},
{
"uid": "3",
"EducationHistory": []
},
{
"uid": "4",
"EducationHistory": {
"10466632": {
"euid": 10466632,
"degree": "Highschool",
"educationLevel": null
},
"10466634": {
"euid": 10466634,
"degree": "Law",
"educationLevel": "batchelor"
},
"10466635": {
"euid": 10466635,
"degree": "Law",
"educationLevel": "master"
}
}
},
{
"uid": "5",
"EducationHistory": {
"10482462": {
"euid": 10482462,
"degree": "IT",
"educationLevel": "master"
}
}
}
]
}
What I want to do is collect the educationLevels per uid in one column. So something like this:
uid | educationLevel
----+------------------
1 |
2 |
3 |
4 | barchelor, master
5 | master
Normally I would like Expandproperty to get down to a lower level, but this doesn't work for this case because every EducationHistory entry is behind a euid for that specific entry. Expanding every single one of them like in the example below isn't workable because of the number of records.
So I think I need something of a loop, but I don't know how. Hope you can help me. First post here and a Powershell newbie, so I hope my question is clear. Please let me know if you need more info.
The code for one entry, as example:
$json = Get-content -raw -path C:\TEMP\File.json
(ConvertFrom-Json -InputObject $json).result |
Select-Object uid,
#Expand one of the entries:
#{Name = "Edu.Level";E={$_.EducationHistory | Select-Object -
expandproperty 10466632 |Select-Object -expandpropert degree }} |
Format-Table
$content = Get-Content .\test.json
$result = ($content | ConvertFrom-Json).result
$totalResult = #()
foreach($res in $result) {
$tempArray = #()
if($res.EducationHistory -ne $null) {
$properties = $res.EducationHistory | Get-Member -MemberType NoteProperty
foreach($property in $properties) {
$eduLevel = $res.EducationHistory.$($property.Name).educationLevel
if(![String]::IsNullOrEmpty($eduLevel)) {
$tempArray += $eduLevel
}
}
}
$totalResult += [PSCustomObject]#{
uid = $res.uid
educationLevel = $tempArray -join ", "
}
}
$totalResult
This will output desired result for the input you have provided.
The trickiest part is the value of EducationHistory property. You have to use Get-Member cmdlet (see Get-Help Get-Member) to get the properties of the current object in loop. Then using the name of the property to access the educationLevel.
Your first question, my first answer I believe :) Similar to the last answer. You need to jump through the hoop of finding the object names in EducationalHistory to reference them.
$json = (Get-content C:\TEMP\File.json | ConvertFrom-Json).result
$results = #()
foreach ( $u in $json)
{
foreach ( $h in $u.EducationHistory)
{
$results += $h.PSObject.properties.Name | ForEach-Object{new-object PSObject -property #{ uid=$u.uid; degree=$h.$_.degree}}
}
}
$results | ConvertTo-Csv | select -skip 1