Powershell convertfrom-json | convertto-csv - json

I have a JSON data structured as following (there may be some mistakes here, the data I'm using is fine):
[{
"id": 12345,
"itemName": "some string",
"sellerId": 123,
"seller": "",
"categoryId": ,
"categoryPath": [
{
//more data
},
{
//more data
}
]},
{"id": 12346,
"itemName": "some other string",
"sellerId": 234,
"seller": "",
"categoryId": ,
"categoryPath": [
{
//more data
},
{
//more data
}
]
}]
I would like to convert it to csv so that the selected property names become csv headers and their value (depth 1 only) become data.
e.g
id,itemName,sellerId
12345,"some string",123
12346,"some other string",234
I've tried using hundreds of variations of
cat file.json | convertfrom-json | convertto-csv
but none have worked. All I get is csv data with objects names/types and I can't figure out how to make it use only selected properties of each object from json data.

In short you need to do something like this:
(Get-Content file.json -Raw | ConvertFrom-Json) | Select id,itemName,sellerId | Convertto-CSV -NoTypeInformation
The first problem was that Get-Content was passing individual lines to ConvertFrom-Json which is not what it wants. Using the -Raw switch passes it in its entirety.
The (Get-Content file.json -Raw | ConvertFrom-Json) needs to be in parentheses as that allows us to continue with the pipe. The properties are not accessible without doing this. It looks like it is trying to pass the entire object instead of its individual parts down the pipe.
-NoTypeInformation removes lines like this
#TYPE Selected.System.Management.Automation.PSCustomObject

Related

Convert CSV file to JArray

I have a need to export the json results of a Invoke-WebRequest call to a .CSV file so that non-technical users can edit the data. This is what the json looks like when I save it to a .CSV file:
{
"Forecasts": [
{
"AccountNumber": "12345",
"Period": "2022-02-01T00:00:00",
"Field1": 3998.87968239,
"Field2": 2133.91206875
},
{
"AccountNumber": "12346",
"Period": "2022-03-01T00:00:00",
"Field1": 6741.483,
"Field2": 4007.857
}
]
}
I'm creating the .CSV file with this line of code:
$json.Forecasts | ForEach-Object { $_ | Export-Csv -path $fileName -NoTypeInformation -Append -Force }
And this is working good. After the user edits the data and saves and closes the .CSV file, I need to convert the .CSV data back into a JArray that looks like the original json structure (like above) so I can make another Invoke-WebRequest call (Patch), passing the JArray as the Body.
When I execute this line of code:
$editedJson = Get-Content -Raw -Path "$fileName" | ConvertFrom-CSV | Select-Object -Property AccountNumber,Period,Field1,Field2 | ConvertTo-json
The json looks like this:
[
{
"AccountNumber": "12345",
"Period": "2022-02-01T00:00:00",
"Field1": "3900.87968239",
"Field2": "2100.91206875"
},
{
"AccountNumber": "12346",
"Period": "2022-03-01T00:00:00",
"Field1": "6700.483",
"Field2": "4000.857"
}
]
As you can see, it's missing the "Forecasts" portion of the JArray and I don't know how to get the .CSV data back into this format. Can anyone help with this ?
You simply need to create an object with that property (Forecasts) and attach the imported CSV to said property:
# This line would be where you import the CSV:
# $csv = Import-Csv .....
$csv = #'
"AccountNumber","Period","Field1","Field2"
"12345","2/1/2022 12:00:00 AM","3998.87968239","2133.91206875"
"12346","3/1/2022 12:00:00 AM","6741.483","4007.857"
'# | ConvertFrom-Csv
#{
Forecasts = $csv
} | ConvertTo-Json
If you want to assure that the JSON representation of Forecasts is always an array (even when the CSV only has one row):
#{
Forecasts = [array]$csv
} | ConvertTo-Json

Convert PowerShell JSON to a CSV file

I have not used PowerShell much. I have an API request which returns JSON that I need to turn into a CSV or xlsx file.
$output = Get-SurveyParticipents `
-url "https://orxsurveys.limequery.com/admin/remotecontrol" `
-session $sessionKey `
-id "5133965" `
-start "0" `
-limit "2" `
-unused $False `
-attributes ["completed", "usesleft"]
Write-Host($output | ConvertTo-Json)
{
"id": 1,
"result": [
{
"tid": "6",
"token": "35ddmyQTlNpzLat",
"participant_info": "#{firstname=Hsdfng; lastname=Gsdfh; email=gosdfdsfz.com}"
},
{
"tid": "7",
"token": "nQ_S838LjYT4mR6",
"participant_info": "#{firstname=Ofdlga; lastname=Yadfdfa; email=olsdfdsfivska#axsdfdsfnce.com}"
}
],
"error": null
}
The participant_info doesn't look like a normal JSON structure.
Also I am unsure how to turn this JSON into a CSV. Something like:
$output | ConvertTo-Json | Export-Csv -Path "c:\Scripts"
You don't need to do both Export-Csv and ConvertTo-Json. Export-Csv will convert a stream of powershell objects into a file containing CSV records.
Assuming the objects in the "results" array in your json output are what you want, you might try something like:
$output.result | Export-Csv -NoTypeInformation foo.csv
You are correct to say that participant_info is not Json. You'll have to process this yourself. ConvertFrom-StringData might be helpful.

Only remove/Exclude an attribute from json if it exists

I have following JSON and I would like to remove streets from the JSON object if only it exists under Address which is an array. I am trying to do this in powershell. I can get my script working and remove the streets but I only want to run the exclude line of command if the address has the streets property. Is that possible?
{
"Customer": [{
"id": "123"
}],
"Nationality": [{
"name": "US",
"id": "456"
}],
"address": [{
"$type": "Home",
"name": "Houston",
"streets": [{
"name": "Union",
"postalCode": "10"
}]
},
{
"$type": "Home5",
"name": "Houston5"
},
{
"$type": "Office",
"name": "Hawai",
"streets": [{
"name": "Rock",
"postalCode": "11"
}]
}
]
}
Powershell script
$FileContent = Get-Content -Path "Test.json" -Raw | ConvertFrom-Json
#Only want to run for address objects that contains streets
$FileContent.address = $FileContent.address | Select-Object * -ExcludeProperty streets #Only would like to run if object address has streets property
$FileContent | ConvertTo-Json
Note:
This answer performs the same operation as in the question, only more succinctly, in a single pipeline.
It is benign to run Select-Object * -ExcludeProperty streets against all objects in array address, because the call is an effective no-op for those objects that already lack a streets property (though a copy of such objects is created too).
You need an assignment to modify your objects in-place before outputting them, which requires a ForEach-Object call:
Get-Content -Raw Test.json | ConvertFrom-Json |
ForEach-Object {
[array] $_.address = $_.address | select * -exclude streets; $_
}
Note how each object parsed from the JSON input is first modified via the assignment ($_.address = ...), and then passed out ($_).
A more efficient, but a little more obscure variant:
Get-Content -Raw Test.json | ConvertFrom-Json |
ForEach-Object {
$_.address.ForEach({ $_.psobject.Properties.Remove('streets') }); $_
}
With your sample JSON input, both commands output the following:
Customer Nationality address
-------- ----------- -------
{#{id=123}} {#{name=US; id=456}} {#{$type=Home; name=Houston}, #{$type=Home5; name=Houston5}, #{$type=Office; name=Hawai}}
Note how the objects in the address column no longer have a streets property.
Caveat: Note that ConvertTo-Json limits the serialization depth to 2 by default, which is sufficient in this case, but in other cases you may have to pass a -Depth argument to prevent data loss - see this post.

Exclude the JSON property only if it exists - Powershell

I have two different JSONs and I would like to remove streets from the JSON object if only it exists under Address which is an array. I am trying to do this in powershell. I can get my script working and remove the streets but I only want to run the exclude line of command if the address has the streets property.
{
"Customer": [{
"id": "123"
}],
"address": [{
"$type": "Home",
"name": "Houston",
"streets": [{
"name": "Union",
"postalCode": "10"
}]
},
{
"$type": "Office",
"name": "Hawai",
"streets": [{
"name": "Rock",
"postalCode": "11"
}]
}
]
}
2nd JSON - Do not want to run the exclude line for 2nd JSON because there are no streets
{
"Customer": [{
"id": "123"
}],
"address": [{
"$type": "Home",
"name": "Houston"
},
{
"$type": "Office",
"name": "Hawai"
}
]
}
Powershell script
$FileContent = Get-Content -Path "Test.json" -Raw | ConvertFrom-Json
#Only want to run for address objects that contains streets
$FileContent.address = $FileContent.address | Select-Object * -ExcludeProperty streets #Only run for 1st json and not for 2nd json
$FileContent | ConvertTo-Json
If you want to execute the code only if the address has the member streets you can test for just that:
if (
($FileContent.address | Get-Member -MemberType NoteProperty -Name "streets") -ne $null
){
$FileContent.address = $FileContent.address | Select-Object * -ExcludeProperty streets
}
T-Me's helpful answer is the most robust approach, because it looks for the presence of the property itself rather than non-null values.
If you're willing to assume that the absence of a value also means the absence of the property itself, you can take the following shortcut, which performs better:
$hasAtLeastOneStreet = 0 -ne
(#((Get-Content Test.json -Raw | ConvertFrom-Json).address.streets) -ne $null).Count
.address.streets uses member-access enumeration to extract all streets values, #(...) ensures that the result is an array, -ne $null filters out any $null values from that array, and .Count counts its elements.
Note: This expression should be simpler:
$null -ne (Get-Content Test.json -Raw | ConvertFrom-Json).address.streets
but due to a bug currently cannot - see the bottom section.
To demonstrate (the input strings are compressed, single-line versions of your JSON documents):
'{"Customer":[{"id":"123"}],"address":[{"$type":"Home","name":"Houston","streets":[{"name":"Union","postalCode":"10"}]},{"$type":"Office","name":"Hawai","streets":[{"name":"Rock","postalCode":"11"}]}]}',
'{"Customer":[{"id":"123"}],"address":[{"$type":"Home","name":"Houston"},{"$type":"Office","name":"Hawai"}]}' |
foreach {
"has street values: " +
(0 -ne #(((ConvertFrom-Json $_).address.streets) -ne $null).Count)
}
The above yields, showing that the first JSON document had street values, whereas the second one did not.
has street values: True
has street values: False
Note: You should be able to simplify the test expression to the following, but this doesn't work due to a bug present up to at least PowerShell 7.0:
# !! SHOULD worm, but as of PowerShell 7.0, DOESN'T, due to a bug relating
# to the presence of two or more [pscustomobject] instances in the address array:
$hasAtLeastOneStreet =
$null -ne (Get-Content Test.json -Raw | ConvertFrom-Json).address.streets
Normally, the absence of any streets property values should result in $null, but with two or more [pscustomobject] instances present in the .address array, an array of $null values is unexpectedly returned.
See GitHub issue #13752.

How to save a JSON object to a file using Powershell?

I have converted the following JSON file to powershell representation object.
{
"computer": [
{
"children": [
{
"children": [ {
"children": [ {
"path": "T:\Dropbox\kvaki.html",
"name": "kvaki",
"type": "url",
"url": "http://example.com"
} ],
"path": "T:\Dropbox\",
"name": "Njusha",
"type": "folder"
}, {
"path": "T:\Dropbox\Europa.html",
"name": "Europa",
"type": "url",
"url": "http://example.com"
}, {
"path": "T:\Dropbox\math.html",
"name": "math",
"type": "url",
"url": "http://example.com"
} ],
"path": "T:\Dropbox\",
"name": "Money",
"type": "folder"
}
],
"full_path_on_file_sys": "T:\Dropbox\"
}
]
}
After doing some computations with powershell representation I would like to save it to file as JSON.
But command $jsonRepresentation | ConvertTo-Json | Out-File "D:\dummy_path\file.json" saves it in this way
{
"computer": [
{
"children": " ",
"full_path_on_file_sys": "T:\Dropbox\"
}
]
}
Question: how to achieve correct saving of complex powershell JSON representation?
-depth argument for ConvertTo-Json solves the issue.
$jsonRepresentation | ConvertTo-Json -depth 100 | Out-File "D:\dummy_path\file.json"
Just pipe it to Set-Content, or Out-File:
Get-Process powershell |
ConvertTo-Json |
Set-Content json.txt
If you want to both view the output and save it to file, you can pipe the tee command.
Get-Process powershell | ConvertTo-Json | Tee-Object json.txt
$json.properties.metadata | ConvertTo-Json -Compress
if you are stuck with PowerShell Version 2, the JSON module of Joel Bennett from the 'PowerShell Code Repository' might help.
1) The below command can be used to convert a json to CSV
Example:
Get-Content package.json | Out-String | ConvertFrom-Json | Select parameter1, parameter2, parameter3 | ConvertTo-Csv -NoTypeInformation | Format-Table >> C:\JenkinsWorkspace\Result.csv
Get-Content: This is like "cat" command in linux which will get all data of file "package.json" and converts from Json (Using ConvertFrom-Json function) extracting the details of only required parameters and then converting them into CSV using "ConvertTo-Csv" function without any unwanted Type Headers and formatting them into Table.
2) The above result can also be formatted into proper csv to view it as Excel format without any duplicates and also having Text-To-Column conversion using below command:
Import-Csv "C:\Result.csv" -delimiter "," | Sort-Object _from -Unique | Export-csv "C:\FINAL_REPORT_$date.csv"