Convert JSON to CSV using PowerShell - json

I have a sample JSON-formatted here which converts fine if I use something like: https://konklone.io/json/
I've tried the following code in PowerShell:
(Get-Content -Path $pathToJsonFile | ConvertFrom-Json)
| ConvertTo-Csv -NoTypeInformation
| Set-Content $pathToOutputFile
But the only result I get is this:
{"totalCount":19,"resultCount":19,"hasMore":false,"results":
How do I go about converting this correctly in PowerShell?

By looking at just (Get-Content -Path $pathToJsonFile) | ConvertFrom-Json it looks like the rest of the JSON is going in to a results property so we can get the result I think you want by doing:
((Get-Content -Path $pathToJsonFile) | ConvertFrom-Json).results |
ConvertTo-Csv -NoTypeInformation |
Set-Content $pathToOutputFile
FYI you can do ConvertTo-Csv and Set-Content in one move with Export-CSV:
((Get-Content -Path $pathToJsonFile) | ConvertFrom-Json).results |
Export-CSV $pathToOutputFile -NoTypeInformation

You have to select the results property inside your CSV using the Select-Object cmdlet together with the -expand parameter:
Get-Content -Path $pathToJsonFile |
ConvertFrom-Json |
Select-Object -expand results |
ConvertTo-Csv -NoTypeInformation |
Set-Content $pathToOutputFile

I was getting my json from a REST web api and found that the following worked:
Invoke-WebRequest -method GET -uri $RemoteHost -Headers $headers
| ConvertFrom-Json
| Select-Object -ExpandProperty <Name of object in json>
| ConvertTo-Csv -NoTypeInformation
| Set-Content $pathToOutputFile
I end up with a perfectly formatted csv file

Trying to use Mark Wrang's answer failed for me. While Piemol's comment from Jan 30 '19 solved a basic problem with Mark Wrang's answer, it also didn't work for me.
JSON strings do not always represent rectangular data sets. They may contain ragged data. For example, the Power BI activities log outputs JSON that contains different members depending on variables like what activities occurred in the requested data or what features were available at the time.
Using Piemol's comment, I processed this JSON:
[
{
"a": "Value 1",
"b": 20,
"g": "Arizona"
},
{
"a": "Value 2",
"b": 40,
"c": "2022-01-01T11:00:00Z"
},
{
"a": "Value 3",
"d": "omicron",
"c": "2022-01-01T12:00:00Z"
},
{
"a": "Value 4",
"b": 60,
"d": "delta",
"e": 14,
"c": "2022-01-01T13:00:00Z"
}
]
The script produced this CSV:
"a","b","g"
"Value 1","20","Arizona"
"Value 2","40",
"Value 3",,
"Value 4","60",
Notice that columns c, d, and e are missing. It appears that Export-CSV uses the first object passed to determine the schema for the CSV to output.
To handle this, use the UnifyProperties function:
function UnifyProperties {
$Names = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::OrdinalIgnoreCase)
$InputCollected = #($Input)
$InputCollected.ForEach({
foreach ($Name in $_.psobject.Properties.Name) { $Null = $Names.Add($Name) }
})
$inputCollected | Select-Object #($Names)
}
$pathToInputFolder = (New-Object -ComObject Shell.Application).NameSpace('shell:Downloads').Self.Path + "\" + "PowerBIActivities\combined\"
$pathToInputFile = $pathToInputFolder + "Activities.json"
$pathToOutputFile = $pathToInputFolder + "Activities.csv"
$content = Get-Content -Path $pathToInputFile -Raw
$psObj = ConvertFrom-Json -InputObject $content
$psObj | UnifyProperties | Export-CSV $pathToOutputFile -NoTypeInformation

Related

Powershell convertto-json gives a bad format

Hello I'm trying to use convertto-json to create a json file from another output and save it but I'm getting a bad json format
Here is my function
$output.value | ForEach-Object {
"{"+"id"+':'+ $_.id+','+"outcome" +':'+ '"'+$_.results.outcome+'"'+','+"idtc"+':' +$_.testCaseReference.id + "}"
} | ConvertTo-Json| Out-File -FilePath .\tfs.json
Here is the outcome of the tfs.json
[
"{id:11431,outcome:\"passed\",idtc:70155}",
"{id:11432,outcome:\"unspecified\",idtc:70156}",
"{id:11433,outcome:\"failed\",idtc:70157}",
"{id:11434,outcome:\"unspecified\",idtc:70158}"
]
Can anyone help why it does appear in that way instead of
[
{"id":11431,"outcome":"passed","idtc":70155"},
{"id":11432,"outcome":"unspecified","idtc":70156"},
{"id":11433,"outcome":"failed","idtc":70157"},
{"id":11434,"outcome":"unspecified","idtc":70158"}
]
The output you're receiving from ConvertTo-JSON is pretty much expected. You're passing a string to the cmdlet hence it is returning a string as JSON.
Try this instead:
$output.value | ForEach-Object {
[pscustomobject]#{
id = $_.id
outcome = $_.results.outcome
idtc = $_.testCaseReference.id
}
} | ConvertTo-Json | Out-File -FilePath .\tfs.json

Extracting fields to CSV from a JSON file with Powershell

i have a json extracted from a API exact like this:
{
"LPEONASVVAP0": {
"LPEONASVVAP0": {
"id": "urn:vcloud:vm:f526d27d-e0f9-4d4f-ae81-4824e397c027",
"name": "LPEONASVVAP0",
"description": "_vm_desc_",
"dateCreated": "2021-04-06T14:56:09.640+0000"
}
},
"WDEONDSVDIS6": {
"WDEONDSVDIS6": {
"id": "urn:vcloud:vm:7ed43492-a7ce-4963-b5bb-5ec2ca89477c",
"name": "WDEONDSVDIS6",
"description": "",
"dateCreated": "2021-04-13T13:44:29.973+0000"
}
},
"WDEONASVSTR0": {
"WDEONASVSTR0": {
"id": "urn:vcloud:vm:7afa34fe-b239-4abe-90df-3f270b44db1f",
"name": "WDEONASVSTR0",
"description": "",
"dateCreated": "2021-03-10T16:17:50.947+0000"
}
},
}
I need extract only fields id, name and description to create a csv with them. I test this but the output file is in blank:
$pathToJsonFile = x
$pathToOutputFile = x
$obj = Get-Content $pathToJsonFile -Raw | ConvertFrom-Json
print $obj
$obj | select id, name, description | Convertto-csv > $pathToOutputFile
You'll need to "discover" the parent property names (eg. 'LPEONASVVAP0') via the psobject hidden memberset. Since the outer and inner properties are named the same, we can re-use the name to get the inner property value:
$obj.psobject.Properties |ForEach-Object {
$_.Value.$($_.Name)
} |Select id,name,description |Export-Csv -NoTypeInformation -Path $pathToOutputFile
Edit: Mathias R. Jessens answer is better written than this, i would do it that way instead of how i posted.
Okay so i copied the json you posted, imported it. Since each array of information is stored like this
"WDEONDSVDIS6": {
"WDEONDSVDIS6": {
i used get-member to iterate each of the arrays and then select the info from that.
Also, you dont need to use Convertto-csv > $pathToOutputFile, use the export-csv command instead.
Below is my code how i would have done it, there is probably a better way but this works :)
$pathToOutputFile = x.csv
$obj = Get-Content example.json -Raw| ConvertFrom-Json
$obj2 = ($obj | Get-Member -MemberType noteproperty).Name
$result = foreach($item in $obj2){
$obj.$item.$item | select id,name,description
}
$result | Export-Csv -Path $pathToOutputFile -Encoding utf8 -NoTypeInformation

How to Get the values from json using powershell

Guys this is my JSON file and I want to create a PowerShell script which will give me result like
I have used method like Get-Content and other but there are some issues with the JSON parsing. Please find what is my requirement I have explained in details below.
MyLocalMachineHome
LocalMachine = Sahil_LocalMachine
Second_MyLocalMachine = Sahil_MylocalMachine
Second_MyLocalMachine = ""
Staging
Second_Staging = Sahil;_Secconf
Staging = Sahil_Staging
third_staging = stsajiii
There is also one functionality which I would like to have if I want to get only variables of "staging".
I was using this function Get-Content -Raw -Path E:\shell\Powershell\1ReleasePipelines.json | ConvertFrom-Json | select -ExpandProperty variables on my original JSON file but somehow there is some kind of limit in storing string which I was getting from this method.
{
"environments": [
{
"id": 3,
"name": "MyLocalMachineHome",
"variableGroups": [],
"variables": {
"LocalMachine": {
"value": "Sahil_LocalMachine"
},
"Second_MyLocalMachine": {
"value": "Sahil_MylocalMachine"
},
"thirf_mylocal": {
"value": ""
}
}
},
{
"id": 7,
"name": "Staging",
"variableGroups": [],
"variables": {
"Second_Staging": {
"value": "Sahil;_Secconf"
},
"Staging": {
"value": "Sahil_Staging"
},
"third_staging": {
"value": "stsajiii"
}
}
}
]
}
If we assume that $json contains your JSON content, you can do the following ugly code:
$environment = 'staging'
$j = $json | ConvertFrom-Json
($j.environments | where name -eq $environment).variables | Foreach-Object {
$CurrentObject = $_
$CurrentObject | Get-Member -MemberType NoteProperty |
Select-Object -Expand Name | Foreach-Object {
$CurrentObject.$_.value
}
}
It appears your issue is that you don't know what variables are going to be contained within your JSON. So you can't easily use Select-Object variable or $object.variable. You need a dynamic approach.
If you know your variables ahead of time, things become simpler. You can store your variable names in an array and loop over them.
$variables = 'Second_Staging','Staging','third_staging'
$environment = 'staging'
$j = $json | ConvertFrom-Json
$jsonVars = ($j.environments | where name -eq $environment).variables
$variables | Foreach-Object {
$jsonVars.$_.value
}
View all the sub-properties of variables with format-list instead of format-table. Since the properties vary, format-table won't show all of them. There's a lot of sloppy object construction in json.
$a = get-content file.json
$a.environments.variables | format-table
LocalMachine Second_MyLocalMachine thirf_mylocal
------------ --------------------- -------------
#{value=Sahil_LocalMachine} #{value=Sahil_MylocalMachine} #{value=}
$a.environments.variables | format-list
LocalMachine : #{value=Sahil_LocalMachine}
Second_MyLocalMachine : #{value=Sahil_MylocalMachine}
thirf_mylocal : #{value=}
Second_Staging : #{value=Sahil;_Secconf}
Staging : #{value=Sahil_Staging}
third_staging : #{value=stsajiii}
Get the staging variables?
$a.environments | where name -eq staging | foreach variables
Second_Staging Staging third_staging
-------------- ------- -------------
#{value=Sahil;_Secconf} #{value=Sahil_Staging} #{value=stsajiii}
cls
start-transcript -path 'C:\E\Devops\PowerShell_Chapters\ABC.txt'
write-output "**********Variables of Release************"
get-content -raw -path 'C:\E\Devops\PowerShell_Chapters\Release.json'| Convertfrom-Json | Select -ExpandProperty variables
$json = get-content -raw -path 'C:\E\Devops\PowerShell_Chapters\Release.json'| Convertfrom-Json | Select -ExpandProperty environments
$EnvirnomentsVariables = get-content -raw -path 'C:\E\Devops\PowerShell_Chapters\Release.json'| Convertfrom-Json | Select -ExpandProperty environments |Select -ExpandProperty name
$ReleaseVariable = get-content -raw -path 'C:\E\Devops\PowerShell_Chapters\Release.json'| Convertfrom-Json | Select -ExpandProperty environments |Select -ExpandProperty variables
$i = 0
foreach($a in $EnvirnomentsVariables)
{
$ABC_Staging = $EnvirnomentsVariables[$i]
#write-output $ABC_Staging
if( $ABC_Staging -match "ABC Staging")
{
write-output "****************Variables of " $EnvirnomentsVariables[$i]*************"
#add-content 'C:\E\Devops\PowerShell_Chapters\ABC.txt' $EnvirnomentsVariables[$i]
# Set-content -path 'C:\E\Devops\PowerShell_Chapters\Sahil.json'| ConvertTo-Json | select $EnvirnomentsVariables[$i]
write-output $ReleaseVariable[$i]
# add-content 'C:\E\Devops\PowerShell_Chapters\ABC.txt' $ReleaseVariable[$i]
# Set-content -path 'C:\E\Devops\PowerShell_Chapters\Sahil.json'| ConvertTo-Json | select $ReleaseVariable[$i]
}
$i = $i + 1
}
stop-transcript

how to prevent PowerShell adding \ to a string

I'm currently working with Powershell editing JSONs
My JSON looks like this:
{
"value": ["E_"]
}
I only want to add a specific number after E_.
The result should look like this:
{
"value": ["E_5"]
}
My PS Script:
$tdePath = 'C:\temp\tde.csv'
$dirName = Get-ChildItem -Path $rootDir -Filter TDE_Config.json -Recurse -ErrorAction SilentlyContinue -Force | Select-Object DirectoryName | Export-Csv $tdePath -NoTypeInformation
$importTDEJson = Import-Csv $tdePath -Encoding UTF8 | % {
[String]$nr = $_.DirectoryName.Split('\')[6].split(' ')[4]
$full_path = $_.DirectoryName + "\TDE_Config.json"
[int]$nr2 = $nr -as [int] #Convert String to Int to convert 004 -> 4
$a = Get-Content $full_path -raw | ConvertFrom-Json
$a.value= "[`"E_" + $nr2 + "`"]"
$a | ConvertTo-Json | set-content $full_path -Force
}
Only excuting "[`"E_" + $nr2 + "`"]" returns the value I need.
For example if $nr2 = 145 it returns in ps console
["E_145"]
But my JSON looks like this:
{
"value": "[\"E_145\"]"
}
Why is powershell adding \ to my string?
How can I prevent powershell adding \ to my string?
You are mixing Json by hand and Json by ConvertTo-Json.
Powershell is doing javascript escaping of the backslashes you insert. Just set $a.value like this:
$a.value= #("E_$nr2")
That will create an array and the Json conversion will do the rest for you.
EDIT:
Here is a proof of concept:
$a = [pscustomobject]#{
value = 'Something else'
}
$nr2 = 145
$a.value= #("E_$nr2")
$a | ConvertTo-Json
outputs:
{
"value": [
"E_145"
]
}

Convert CSV to JSON and JSON to CSV using PowerShell

I have a CSV file that I am trying to convert to JSON using PowerShell.
The CSV file contains the following data.
web_url.csv
wikipedia,https://en.wikipedia.org/wiki/%s
wolframalpha,http://www.wolframalpha.com/input/?i=%s
drive,http://www.drive.google.com/
I would like to convert to json in the following format. Similarly how do you convert this json back to original csv in the format shown above?
web_url.json
{
"wikipedia": "https://en.wikipedia.org/wiki/%s",
"wolframalpha": "http://www.wolframalpha.com/input/?i=%s",
"drive": "http://www.drive.google.com/"
}
When I run the command,
Get-Content -path web_url.csv | ConvertFrom-Csv -Delimiter ',' |
ConvertTo-Json
it returns the following output which is not what I want.
[
{
"wikipedia": "wolframalpha",
"https://en.wikipedia.org/wiki/%s": "http://www.wolframalpha.com/input/?i=%s"
},
{
"wikipedia": "drive",
"https://en.wikipedia.org/wiki/%s": "http://www.drive.google.com/"
}
]
# PowerShell script
import-csv "SampleInput.csv" | ConvertTo-Json | Add-Content -Path "output.json"
Your csv doen't look like a "proper" csv to me: columns are swapped with rows. If you have control over input file, you may fix it there already:
#'
wikipedia,wolframalpha,drive
https://en.wikipedia.org/wiki/%s,http://www.wolframalpha.com/input/?i=%s,http://www.drive.google.com/
'# | ConvertFrom-Csv | ConvertTo-Json
If that is not possible, you just have to perform some extra steps to get what you need:
$propertyList = #'
wikipedia,https://en.wikipedia.org/wiki/%s
wolframalpha,http://www.wolframalpha.com/input/?i=%s
drive,http://www.drive.google.com/
'# | ConvertFrom-Csv -Header Name, Value
$properties = [ordered]#{}
foreach ($property in $propertyList) {
$properties.Add($property.Name, $property.Value)
}
New-Object PSObject -Property $properties | ConvertTo-Json
And back, again - some extra work is required:
(#'
{
"wikipedia": "https://en.wikipedia.org/wiki/%s",
"wolframalpha": "http://www.wolframalpha.com/input/?i=%s",
"drive": "http://www.drive.google.com/"
}
'# | ConvertFrom-Json).PSObject.Properties |
Select-Object Name, Value |
ConvertTo-Csv -NoTypeInformation |
Select-Object -Skip 1
what about this ?
$csv=#"
wikipedia,https://en.wikipedia.org/wiki/%s
wolframalpha,http://www.wolframalpha.com/input/?i=%s
drive,http://www.drive.google.com/
"#
$obj= $csv |convertfrom-csv -delim ',' -Header "name","url"
#$obj
$json=$obj |convertto-json
#$json
$csv2 =$json |convertfrom-json |select -expand syncroot |convertto-csv -NoTypeInformation -Delimiter ','
#$csv2