Powershell - json to text or csv - json

I have a folder with hundreds of json files in it & need to read them & create an output file with the various fields & values in it.
{
"id": "02002010",
"booktitle": "",
"pagetitle": "Demo Page",
"parent": "02002000",
"img": [
{
"imgfile": "02A.png",
"imgname": "02A.png"
}
],
"fmt": "",
"entries": [
{
"itemid": "1",
"partnumber": "1234567",
"partdescription": "Washer",
"partqty": "2",
"Manufacturer": "ACME",
"partdescriptionlocal": "Washer"
},
{
"itemid": "2",
"partnumber": "98765-B",
"partdescription": "Screw",
"partqty": "8",
"Vendor": "Widget Inc",
"TYPE": "Galv",
"partdescriptionlocal": "Screw"
}]
}
The json files will have generally the same structure, except that the "entries" may contain various fields in it that may not be the same from one entry to the next, or one json file to the next. Some may have fields within entry that I do not know the name of. There will be a few common fields in each "entries" section, but they could vary, and could be in a different order than what is shown.
I would like to write the output to a text/csv file that would be delimited that could then be imported into Excel. One column header with all fields listed. As new "entries" fields are found, tack them on to the end of each row & add to the header also.

you mean to do something like this?
$json = gc C:\temp\file.json | ConvertFrom-Json
$props = $json.entries | % {$_ | gm -MemberType NoteProperty} | select -exp name -Unique
$results = #()
foreach ($entry in $json.entries) {
$obj = $json | select *
foreach ($prop in $props) {
$obj | Add-Member -MemberType NoteProperty -Name $prop -Value $($entry | select -exp $prop -ea 0)
}
$results += $obj
}
$results | epcsv C:\temp\file.csv -NoTypeInformation -Encoding ASCII
$results

Related

Seperating Json objects based on keyvalue

currently i am working on fetching Azure ad application expiry status on that i have pulled around 1700 ad applications and i have added one key pair(status) to the json object based on the secret expiry date
1) valid 2) Expired 3) expiring soon
so i have extracted all applications to a json file now i need to split single file into 3 files based on status as mentioned below
[
{
"DisplayName": "Reporter-dev",
"ObjectId": null,
"ApplicationId": {
"value": "62838283828288282828828288282828",
"Guid": "62838283828288282828828288282828"
},
"KeyId": "62838283828288282828828288282828",
"Type": "Password",
"StartDate": {
"value": "/Date(1590537256000)/",
"DateTime": "27 May 2020 05:24:16"
},
"EndDate": {
"value": "/Date(1653609256000)/",
"DateTime": "27 May 2022 05:24:16"
},
"Ownername": "shetty#gmail.com",
"Status": "Valid"
},
{
"DisplayName": "azure-cli-2018",
"ObjectId": null,
"ApplicationId": {
"value": "52388282828828288273673282932739223",
"Guid": "52388282828828288273673282932739223"
},
"KeyId": "52388282828828288273673282932739223",
"Type": "Password",
"StartDate": {
"value": "/Date(1568849784000)/",
"DateTime": "19 September 2019 05:06:24"
},
"EndDate": {
"value": "/Date(1600472184000)/",
"DateTime": "19 September 2020 05:06:24"
},
"Ownername": "joseph#gmail.com",
"Status": "Expired"
},
{
"DisplayName": "azure-cli-2019",
"ObjectId": null,
"ApplicationId": {
"value": "26382882828828282882828282828",
"Guid": "26382882828828282882828282828"
},
"KeyId": "26382882828828282882828282828",
"Type": "Password",
"StartDate": {
"value": "/Date(1576143476000)/",
"DateTime": "12 December 2019 15:07:56"
},
"EndDate": {
"value": "/Date(1607765876000)/",
"DateTime": "12 December 2020 15:07:56"
},
"Ownername": "zzzzzzzzz#gmail.com",
"Status": "About to Expire"
}
]
The below will split out the JSON based on the status and convert the data back to JSON. Change $JSONPath, $ValidPath, $ExpiredPath, $ExpiredSoonPath to the paths you require, the ones currently populated are what I have used for testing.
The contents of $JSONPath must have valid JSON to be able to work, whilst this is probably not the most efficient nor elegant it should do what you need.
$JSONPath = "C:\PS\JT\JSON.txt"
$JSONObj = Get-Content $JSONPath | ConvertFrom-Json
$ValidPath = "C:\PS\JT\Valid.txt"
$ExpiredPath = "C:\PS\JT\Expired.txt"
$ExpireSoonPath = "C:\PS\JT\ExpireSoon.txt"
$JSONObj | Where {$_.Status -eq "Valid"} | ConvertTo-Json | Out-File $ValidPath
$JSONObj | Where {$_.Status -eq "Expired"} | ConvertTo-Json | Out-File $ExpiredPath
$JSONObj | Where {$_.Status -eq "About to Expire"} | ConvertTo-Json | Out-File $ExpireSoonPath
Something very simple like the below example should work in theory but since you won't show your code there's a bunch of guess work here...
$collection = $json | ConvertFrom-Json
$valid = #(); $exp = #(); $soon = #(); $unknown = #()
foreach($item in $collection) {
switch ($item.Status)
{
'Valid' { $valid += $item }
'Expired' { $exp += $item }
'About to Expire' { $soon += $item }
Default { $unknown += $item}
}
}
$valid | Out-File .\Valid.txt -Append
$exp | Out-File .\Expired.txt -Append
$soon | Out-File .\AboutToExpire.txt -Append
$unknown | Out-File .\Unknown.txt -Append
Or if json is the desired output as in the other example;
$valid | ConvertTo-Json | Out-File .\Valid.txt -Append
etc, etc
Adding to the other answers, here is another solutions using pipelines.
Get-Content -Path .\sample.json -Raw | ConvertFrom-Json |
ForEach-Object { $_ | ConvertTo-Json | Out-File -FilePath "$($_.Status).json" }
Explanation
Parse JSON content using Get-Content, also making sure we use the -Raw switch to return the contents as one string with the newlines preserved. We don't need the contents as an array of strings(default behaviour).
Pipe to ConvertFrom-Json to deserialize the JSON into a System.Management.Automation.PSCustomObject.
Iterate over the objects using Foreach-Object and convert the object to a JSON file using ConvertTo-Json and Out-File. Note the default depth for ConvertTo-Json is 2, so if your JSON files end up having deeper levels, then you will need to specify a larger depth using the -Depth switch.
If its easier understand, you can also use a regular foreach loop here:
$json = Get-Content -Path .\sample.json -Raw | ConvertFrom-Json
foreach ($object in $json) {
$object | ConvertTo-Json | Out-File -FilePath "$($object.Status).json"
}

Converting XML to JSON with Powershell

I am trying to convert an xml file to json for further processing.
This is actually first time I use powershell so I need some help.
I get an xml file with this structure:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="https://www.something..."?>
<tabela>
<naslov>Robno knjigovodstvo - pregled automatskog stanja zaliha artikala</naslov>
<dataset>qu1RobaZalihe</dataset>
<datum_kreiranja>12.02.2021</datum_kreiranja>
<zaglavlje><sifra>Šifra</sifra><naziv>Naziv</naziv><katbroj>Kataloški broj</katbroj><cfJedmj>JedMj</cfJedmj><stanje>Stanje</stanje><cijena>Nabavna cijena</cijena><datum>Datum dokumenta</datum></zaglavlje>
<redovi>
<red>
<sifra>1</sifra>
<naziv>CORE HIT</naziv>
<katbroj>A4B0000006606</katbroj>
<cfJedmj>KOM</cfJedmj>
<stanje>1</stanje>
<cijena>100,00</cijena>
<datum>01.01.2021</datum>
</red>
<red>
<sifra>2</sifra>
<naziv>CORE HIT</naziv>
<katbroj>A4B0000008340</katbroj>
<cfJedmj>KOM</cfJedmj>
<stanje>8</stanje>
<cijena>100,00</cijena>
<datum>01.01.2021</datum>
</red>
...
</redovi>
</tabela>
I need to ger an array of objets separated with comma like this:
[
{
"sifra": "1",
"naziv": "CORE HIT 7050",
"jed_mj": "KOM",
"kolicina": "1",
"skladiste": "1",
"pn": "TNX:A4B0000006606"
},
{
"sifra": "2",
"naziv": "CORE HIT 7020",
"jed_mj": "KOM",
"kolicina": "8",
"skladiste": "1",
"pn": "TNX:A4B0000008340"
},
{
"sifra": "3",
"naziv": "SYSTEM SW LICENCE AND CD-ROM",
"jed_mj": "KOM",
"kolicina": "1",
"skladiste": "1",
"pn": "TNX:A4B0000007760"
},
...
]
... but instead I get only objects like this:
{
"sifra": "1",
"naziv": "CORE HIT 7050",
"pn": "TNX:A4B0000006606",
"jed_mj": "KOM",
"kolicina": "1",
"skladiste": "1"
}
{
"sifra": "2",
"naziv": "CORE HIT 7020",
"pn": "TNX:A4B0000008340",
"jed_mj": "KOM",
"kolicina": "8",
"skladiste": "1"
}
{
"sifra": "3",
"naziv": "SYSTEM SW LICENCE AND CD-ROM",
"pn": "TNX:A4B0000007760",
"jed_mj": "KOM",
"kolicina": "1",
"skladiste": "1"
}
So far I managed to figure out some things but not all :)
Code I use to get this is following:
[xml]$xml = Get-Content xml-skladiste.XML
foreach ($atr in $xml.tabela.redovi.red)
{
$prop = [ordered]#{
'sifra' = $atr.sifra
'naziv' = $atr.naziv
'pn' = $atr.katbroj
'jed_mj' = $atr.cfJedmj
'kolicina' = $atr.stanje
'skladiste' = "1"
}
New-Object -Type PSCustomObject -Property $prop | ConvertTo-Json | Add-Content -Path "output.json"
}
What I have to add in order to get desirable structure?
You are currently creating multiple JSON documents, one for each element of the array $xml.tabela.redovi.red, joined together in a single file.
Move ConvertTo-Json out of the loop to fix the problem. This way you separate the creation of the data from conversion to JSON so ConvertTo-JSON can operate on the complete data structure.
[xml]$xml = Get-Content xml-skladiste.XML
$array = foreach ($atr in $xml.tabela.redovi.red)
{
$prop = [ordered]#{
'sifra' = $atr.sifra
'naziv' = $atr.naziv
'pn' = $atr.katbroj
'jed_mj' = $atr.cfJedmj
'kolicina' = $atr.stanje
'skladiste' = "1"
}
New-Object -Type PSCustomObject -Property $prop
}
$array | ConvertTo-Json | Set-Content -Path "output.json"
Explanation:
$array = foreach ... captures the output of the foreach loop in the $array variable, automatically creating an array.
New-Object ... produces the output that is captured. Note that we didn't have to use Write-Output because of PowerShell's implicit output behaviour (read more about it).
$array | ConvertTo-Json ... passes the whole array to a single invocation of ConvertTo-Json and finally writes it to the output file.

Replace column value with null in json file using Powershell

Here is how my json file looks like :
{
"count": 12,
"name": "Daily Ticket",
"columnNames": [
"User",
"Channel",
"Date",
"# of Closed Incidents",
"Open",
"Response",
"Remark",
"Closed"
],
"rows": [
[
"abc",
"Service Web",
"\u00272020-06-13 00:00:00\u0027",
"1",
"0",
"0",
"this is a text,please replace with null",
"1"
],
[
"xyz",
"Email",
"\u00272020-06-13 00:00:00\u0027",
"21",
"1",
"0",
"this is a text,please replace with null",
"7"
]
]
}
I want to replace all the values in columns of Remark with null and convert into a csv file using powershell. Please help to achieve this.
I want column names as header and rows as rows separated with comma in csv.
My output csv file should look like below one:
User,Channel,Date,# of Closed Incidents,Open,Response,Remark,Closed
abc,Service Web,\u00272020-06-13 00:00:00\u0027,1,0,0,,1
xyz,Email,\u00272020-06-13 00:00:00\u0027,1,0,0,,1
To convert this json into a CSV file is not that difficult.
Just load the JSON, convert it into an object and loop through the properties building an array of new objects you can save as CSV:
$json = Get-Content -Path 'D:\Test\DailyTicket.json' -Raw | ConvertFrom-Json
$headers = $json.columnNames
$result = foreach ($row in $json.rows) {
# just a precaution to not run into index errors when there are
# more items in the array than there are headers or vice-versa
$items = [math]::Min($row.Count, $headers.Count)
# create a new empty (ordered) hashtable
$hash = [ordered]#{}
for ($i = 0; $i -lt $items; $i++) {
# fill the hashtable, except for iten 'Remark'
$hash[$headers[$i]] = if ($headers[$i] -ne 'Remark') { $row[$i] } else { $null }
}
# If you insist on keeping the apostrophe characters in the date field in unicode format `\u0027`
# $hash['Date'] = $hash['Date'] -replace "'", '\u0027'
# output a PSObject to be collected in array $result
[PsCustomObject]$hash
}
# output on screen
$result | Format-Table -AutoSize
# output to CSV file
$result | Export-Csv -Path 'D:\Test\DailyTicket.csv' -NoTypeInformation
Resulting CSV file:
"User","Channel","Date","# of Closed Incidents","Open","Response","Remark","Closed"
"abc","Service Web","'2020-06-13 00:00:00'","1","0","0",,"1"
"xyz","Email","'2020-06-13 00:00:00'","21","1","0",,"7"

Go over json data in Powershell

I have a JSON file that looks like this:
$jsondata = '{
"ips": {
"10.20.30.40": [
{
"rhost": "DNS Name1.",
"rdata": [
"10.20.30.40"
],
"rrtype": "A (1)",
"ttl": 86400,
"geo": null,
"source": "DNSProvider1"
}
],
"40.50.60.70": [
{
"rhost": "DNS Name2.",
"rdata": [
"40.50.60.70"
],
"rrtype": "A (1)",
"ttl": 86400,
"geo": null,
"source": "DNSProvider1"
}
]
}
}'
I want to get all the TTLs (for example) of every IP address in the list.
I converted this JSON to Powershell PSCustomObject:
$obj = $jsondata | convertFrom-Json
and now I want to get all the TTLs, I tried to get the list of the IPs (as a start):
foreach ($ip in $a.ips) {write-host $ip }
and I'm not getting strings as a result, that's why I (probably) can't go inside and get the TTLs.
So my question: how can I get all the IPs as strings?
I believe that once I'll get an answer for that, I'll understand how I can go over all the IPs in the list.
Thanks!
foreach($ip in $obj.ips | Get-Member -MemberType NoteProperty)
{
Write-Host -Verbose ("IP Address {0} has TTL {1}" -f $ip.Name, $obj.ips."$($ip.Name)".ttl)
}
Get-Member will get you the name of the property (which is the ip address) and not the value.
Thanks Rubanov, that really helped!
And just to document the whole answer:
foreach($ip in $obj.ips | Get-Member -MemberType NoteProperty)
{
Write-Host -Verbose $obj.ips.$($ip.Name).ttl
}
Or:
($obj.ips | Get-Member -MemberType NoteProperty).Name | % {$obj.ips.$_.ttl}

PowerShell : retrieve JSON object by field value

Consider JSON in this format :
"Stuffs": [
{
"Name": "Darts",
"Type": "Fun Stuff"
},
{
"Name": "Clean Toilet",
"Type": "Boring Stuff"
}
]
In PowerShell 3, we can obtain a list of Stuffs :
$JSON = Get-Content $jsonConfigFile | Out-String | ConvertFrom-Json
Assuming we don't know the exact contents of the list, including the ordering of the objects, how can we retrieve the object(s) with a specific value for the Name field ?
Brute force, we could iterate through the list :
foreach( $Stuff in $JSON.Stuffs ) {
But I am hopeful there exists a more direct mechanism ( similar to Lync or Lambda expressions in C# ).
$json = #"
{
"Stuffs":
[
{
"Name": "Darts",
"Type": "Fun Stuff"
},
{
"Name": "Clean Toilet",
"Type": "Boring Stuff"
}
]
}
"#
$x = $json | ConvertFrom-Json
$x.Stuffs[0] # access to Darts
$x.Stuffs[1] # access to Clean Toilet
$darts = $x.Stuffs | where { $_.Name -eq "Darts" } #Darts
I just asked the same question here: https://stackoverflow.com/a/23062370/3532136
It has a good solution. I hope it helps ^^.
In resume, you can use this:
The Json file in my case was called jsonfile.json:
{
"CARD_MODEL_TITLE": "OWNER'S MANUAL",
"CARD_MODEL_SUBTITLE": "Configure your download",
"CARD_MODEL_SELECT": "Select Model",
"CARD_LANG_TITLE": "Select Language",
"CARD_LANG_DEVICE_LANG": "Your device",
"CARD_YEAR_TITLE": "Select Model Year",
"CARD_YEAR_LATEST": "(Latest)",
"STEPS_MODEL": "Model",
"STEPS_LANGUAGE": "Language",
"STEPS_YEAR": "Model Year",
"BUTTON_BACK": "Back",
"BUTTON_NEXT": "Next",
"BUTTON_CLOSE": "Close"
}
Code:
$json = (Get-Content "jsonfile.json" -Raw) | ConvertFrom-Json
$json.psobject.properties.name
Output:
CARD_MODEL_TITLE
CARD_MODEL_SUBTITLE
CARD_MODEL_SELECT
CARD_LANG_TITLE
CARD_LANG_DEVICE_LANG
CARD_YEAR_TITLE
CARD_YEAR_LATEST
STEPS_MODEL
STEPS_LANGUAGE
STEPS_YEAR
BUTTON_BACK
BUTTON_NEXT
BUTTON_CLOSE
Thanks to mjolinor.
David Brabant's answer led me to what I needed, with this addition:
x.Stuffs | where { $_.Name -eq "Darts" } | Select -ExpandProperty Type
Hows about this:
$json=Get-Content -Raw -Path 'my.json' | Out-String | ConvertFrom-Json
$foo="TheVariableYourUsingToSelectSomething"
$json.SomePathYouKnow.psobject.properties.Where({$_.name -eq $foo}).value
which would select from json structured
{"SomePathYouKnow":{"TheVariableYourUsingToSelectSomething": "Tada!"}
This is based on this accessing values in powershell SO question
. Isn't powershell fabulous!
In regards to PowerShell 5.1 ...
Operating off the assumption that we have a file named jsonConfigFile.json with the following content from your post:
{
"Stuffs": [
{
"Name": "Darts",
"Type": "Fun Stuff"
},
{
"Name": "Clean Toilet",
"Type": "Boring Stuff"
}
]
}
This will create an ordered hashtable from a JSON file to help make retrieval easier:
$json = [ordered]#{}
(Get-Content "jsonConfigFile.json" -Raw | ConvertFrom-Json).PSObject.Properties |
ForEach-Object { $json[$_.Name] = $_.Value }
$json.Stuffs will list a nice hashtable, but it gets a little more complicated from here. Say you want the Type key's value associated with the Clean Toilet key, you would retrieve it like this:
$json.Stuffs.Where({$_.Name -eq "Clean Toilet"}).Type
It's a pain in the ass, but if your goal is to use JSON on a barebones Windows 10 installation, this is the best way to do it as far as I've found.
This is my json data:
[
{
"name":"Test",
"value":"TestValue"
},
{
"name":"Test",
"value":"TestValue"
}
]
Powershell script:
$data = Get-Content "Path to json file" | Out-String | ConvertFrom-Json
foreach ($line in $data) {
$line.name
}