Parse json values into a single row - json

{
"value": [{
"ExternalKey": "12345",
"PortfolioId": "ABC",
"InceptionDate": null,
"TerminationDate": null,
"Version": 2,
"SourceRef": "ABC",
"MasterSeries": [{
"Points": [{
"Date": "1900-01-01T00:00:00+00:00",
"Name": "XYZ Name",
"Type": "IP",
"Status": "Active",
"Version": 1
}],
"ExternalKey": "12345",
"Version": 1,
"IsDeleted": false,
"SourceRef": "ABC"
}]
}]
}
I tried to loop through each level and manually selecting the keys
ForEach($d in $json.value) { $row=$d.column1+","+ $d.column2 ..
$rootProps = $d| Get-Member | Where-Object { $_.MemberType -match "Property"}
$rootProps | ForEach-Object {
if($_.Name -eq "MasterSeries") {
$d | select -Expand MasterSeries | select-object * -ExcludeProperty Points | ForEach { $row = $_.column1 + "," $_.column2 ..
}
}
and so on..... but when I export it to Out-File "Export.csv" and try importing it gives me a UTF error. Is there a better way to achieve?

If this helps any, here's the typical way to loop through json properties in powershell 5 using the hidden psobject.properties. You might search for "flatten json".
$a = get-content file.json | convertfrom-json
$a.value[0].psobject.properties | select name,value
Name Value
---- -----
ExternalKey 12345
PortfolioId ABC
InceptionDate
TerminationDate
Version 2
SourceRef ABC
MasterSeries

For this you will need to create a recursive function.
To make things easier, I recommend you to use the ConvertFrom-Json -AsHashTable parameter which will return [HashTable] objects rather than [PSCustomObject] Objects
Function Get-Values($Object) {
if ($Object -is [String]) { $Object } # A string is also Enumerable
elseif ($Object -is [Collections.IDictionary]) { $Object.get_Values() | ForEach-Object { Get-Values $_ } }
elseif ($Object -is [Collections.IEnumerable]) { $Object.GetEnumerator() | ForEach-Object { Get-Values $_ } }
else { $Object }
}
$Object = ConvertFrom-Json -AsHashTable '{
"value": [
{
"ExternalKey": "12345",
"PortfolioId": "ABC",
"InceptionDate": null,
"TerminationDate": null,
"Version": 2,
"SourceRef": "ABC",
"MasterSeries": [
{
"Points": [
{
"Date": "1900-01-01T01:00:00+01:00",
"Name": "XYZ Name",
"Type": "IP",
"Status": "Active",
"Version": 1
}
],
"ExternalKey": "12345",
"Version": 1,
"IsDeleted": false,
"SourceRef": "ABC"
}
]
}
]
}'
Get-Values $Object
12345
False
1
IP
Monday, January 1, 1900 1:00:00 AM
XYZ Name
Active
ABC
1
12345
2
ABC
ABC

Related

Sort JSON objects by value of the key (Index) in PowerShell

I'm trying to sort the following JSON objects by the value of the "Index" key.
JSON = {
"PIPoint": {
"Label": "\"PIPoint\"",
"Visible": "True",
"Index": "2"
},
"Description": {
"Label": "\"Description\"",
"Visible": "True",
"Index": "3"
},
"Analysis": {
"Label": "\"Analysis\"",
"Visible": "True",
"Index": "4"
},
"PIPointExist": {
"Label": "\"PIPointExist\"",
"Visible": "True",
"Index": "5"
},
"Custom Location": {
"Label": "\"Custom Location\"",
"Visible": "True",
"Index": "1"
}
}
I've try the following code but it did not work
$json = Get-Content 'C:\column.json' | ConvertFrom-Json
$json = $json | Sort-Object -Property {$_.Index}
$json | ConvertTo-Json | Out-File 'C:\\column_sorted.json'
One way to do this would be to sort the properties and use Select-Object to reorder the output:
# Return objects based on JSON strings
$jsonObjs = Get-Content 'C:\column.json' | ConvertFrom-Json
# Sorted properties list
$Properties = ($jsonObjs.psobject.properties | Sort-Object {[int]$_.Value.Index}).Name
# Create new JSON strings with new order
$jsonObjs | Select-Object $Properties | ConvertTo-Json
$jsonĀ¹ is a single object (with just 5 properties).
So you need to sort the properties:
$Data = Get-Content 'C:\column.json' | ConvertFrom-Json
$Properties = [ordered]#{}
$Data.PSObject.Properties |Sort-Object { $_.Value.Index } |ForEach-Object { $Properties[$_.Name] = $_.Value }
[pscustomobject]$Properties | ConvertTo-Json
I would call it $Json as it is no longer a Json string but an object

Converting JSON to NDJSON with Powershell

I'm trying to convert a text file that has an array of json objects to an NDJSON formatted file for another team to consume.
I've almost got it, except for one problem. I have an array of objects nested inside the JSON (which then has nested arrays and objects inside of it, the structure gets pretty complex, I'll include a sample below) and for whatever reason, when I use ConvertFrom-JSON it drops this nested array and in my output, I end up with a blank string for that key, instead of the nested array object. I tried using the -Depth flag but when I do that my output file ends up blank, which doesn't make a ton of sense to me. I don't have a whole lot of experience with powershell, so I'm not really sure where I'm going wrong here.
Code:
$JSONSourceFile = Get-Content -Path "input/sample.json" | ConvertFrom-JSON
$NDJSONTargetFile = "output/sample.json"
New-Item $NDJSONTargetFile -ItemType file
for ( $i = 0 ; $i -lt $JSONSourceFile.Length ; $i++) {
$item = $JSONSourceFile.item($i)
$row = ($item | ConvertTo-JSON -Compress)
Add-Content $NDJSONTargetFile $row
}
Input File:
[
{
"id": "1",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak2354",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "Y"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "N"
}
]
}
]
}
]
},
{
"id": "2",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak1234",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "Y"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "Y"
}
]
}
]
}
]
},
{
"id": "3",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak5678",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "N"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "N"
}
]
}
]
}
]
}
]
And then when I convert it to the output, this is what I get:
{"id":"1","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak2354","Preferences":""}]}
{"id":"2","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak1234","Preferences":""}]}
{"id":"3","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak5678","Preferences":""}]}
Thanks to the comment from Doug Maurer I figured it out, I was adding -Depth to my ConvertFrom-Json command when it should have been in my ConvertTo-Json command. This is the final script and what it gives me:
$JSONSourceFile = Get-Content -Path "input/sample.json" | ConvertFrom-JSON
$NDJSONTargetFile = "output/sample.json"
New-Item $NDJSONTargetFile -ItemType file
for ( $i = 0 ; $i -lt $JSONSourceFile.Length ; $i++) {
$item = $JSONSourceFile.item($i)
$row = ($item | ConvertTo-JSON -Compress -Depth 20)
Add-Content $NDJSONTargetFile $row
}
and the output:
{"id":"1","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak2354","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"Y"},{"ChannelID":"2","ChannelName":"Text","Preference":"N"}]}]}]}
{"id":"2","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak1234","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"Y"},{"ChannelID":"2","ChannelName":"Text","Preference":"Y"}]}]}]}
{"id":"3","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak5678","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"N"},{"ChannelID":"2","ChannelName":"Text","Preference":"N"}]}]}]}

Powershell - issue with matching an object within a list with another object within another list of object

yep, i dunno if i can make a longer title ... ;)
well, here is what i got :
some lists of objects (lets call them "objA[0..z]") in "listA[1..z]"
some lists of objects (lets call them "objB[0..z]") in "listB[1..z]"
Here is a Dataset (modified from answer below)
$lista1 = #'
[
{
"computerName": "123",
"userName": "Antoine",
"otherData1": "test1.xls",
"otherData2": "yyy"
},
{
"computerName": "123",
"userName": "Antoine",
"otherData1": "test2.xls",
"otherData2": "yyy"
},
{
"computerName": "456",
"userName": "Benoit",
"otherData1": "file1.txt",
"otherData2": "yyy"
}
]
'#
$lista2 = #'
[
{
"computerName": "789",
"userName": "Cyril",
"otherData1": "file1.pps",
"otherData2": "yyy"
},
{
"computerName": "789",
"userName": "Cyril",
"otherData1": "file2.pps",
"otherData2": "yyy"
},
{
"computerName": "321",
"userName": "Damien",
"otherData1": "doc.docx",
"otherData2": "yyy"
}
]
'#
$listb1 = #'
[
{
"computerName": "789",
"userName": "Cyril",
"otherData1": "file2.pps",
"otherData2": "yyy"
},
{
"computerName": "321",
"userName": "Damien",
"otherData1": "doc.docx",
"otherData2": "bbb"
}
]
'#
$listb2 = #'
[
{
"computerName": "123",
"userName": "Antoine",
"otherData1": "test1.xls",
"otherData2": "yyy"
},
{
"computerName": "123",
"userName": "Antoine",
"otherData1": "test2.xls",
"otherData2": "yyy"
},
]
'#
# expected output with this dataset :
# 1st and 2nd are not in any listb and 3rd.otherData2 do not match with each other
[
{
"computerName": "456",
"userName": "Benoit",
"otherData1": "file1.txt",
"otherData2": "yyy"
},
{
"computerName": "789",
"userName": "Cyril",
"otherData1": "file1.pps",
"otherData2": "yyy"
},
{
"computerName": "321",
"userName": "Damien",
"otherData1": "doc.docx",
"otherData2": "yyy"
}
]
what i'm trying to do :
If (obj'w' in lista'x') match computerName and userName and otherData1 in (obj'y' in listb'z')
AND IF same object NOT match otherData2 -> return said object
ELSE IF (obj'w' in lista'x') NOT match any object in every listb[0..z] -> return said object
here is what i did so far and where i'm stuck.
I more or less managed to make it work with "single file" on both listA & listB
Problem1 is i'm clearly drowing in foreach loop and i fail to find a "clean way".
Problem2 is i fail to properly check if objA is present in listB resulting in not being able to return it. (or returning all objects)
$listAPath = 'D:\A'
$listBPath = 'D:\B'
# multiples files in each directory
$listAFiles = Get-ChildItem $listAPath
$listBFiles = Get-ChildItem $listBPath
$failList = foreach ($listAFile in $listAFiles) {
$listObjA = Get-Content $listAFile.fullname -raw | ConvertFrom-Json
foreach ($objA in $listObjA) {
foreach ($listBFile in $listBFiles) {
$listObjB = Get-Content $listBFile.fullname -raw | ConvertFrom-Json
foreach ($objB in $listObjB) {
if ($objA.computerName -eq $objB.computerName -and $objA.userName -eq $objB.userName -and $objA.otherData1 -eq $objB.otherData1) {
if (<# do something with OtherData2 #>) {
$ObjA
}
} <# else {return $objA} #>
}
}
}
}
$failList
Reworked this script while typing it, hope i did not forget anything while doing it.
I'd be glad if i could get help here
If you need some more information, do not hesitate to ask.
Ty in advance.
EDITED : trying to apply what was asked in comment / answer, did not modify first "version" of script provided even though it might no match anymore with given dataset and reworked informations... will edit if asked.
For thoose who are curious, otherData2 is a datetime data with the "real" script.
If I understand you correctly, if an object in A matches the 3 properties of an object in list b then you will check OtherData2 and return the object depending on that check. If it's not in list b then just return the object. I've created some sample data for this demonstration.
$lista = #'
[
{
"computerName": 123,
"userName": "john",
"otherData1": "data128",
"otherData2": "yyy"
},
{
"computerName": 456,
"userName": "joan",
"otherData1": "data821",
"otherData2": "nnn"
},
{
"computerName": 789,
"userName": "doug",
"otherData1": "data111",
"otherData2": "abc"
}
]
'#
$listb = #'
[
{
"computerName": 123,
"userName": "john",
"otherData1": "data128",
"otherData2": "yyy"
},
{
"computerName": 789,
"userName": "doug",
"otherData1": "data111",
"otherData2": "abc"
}
]
'#
$listObjA = $lista | ConvertFrom-Json
$listObjB = $listb | ConvertFrom-Json
foreach($objA in $listObjA)
{
if($listObjB | where {$_.computername -eq $objA.computername -and
$_.username -eq $objA.username -and
$_.otherData1 -eq $objA.otherdata1})
{
if($objA.otherdata2 -eq 'yyy')
{
$objA
}
}
else
{
$objA
}
}
The output will be 2 objects from list a, one that was in list b AND otherdata2 met the criteria and the other was not in list b.
computerName userName otherData1 otherData2
------------ -------- ---------- ----------
123 john data128 yyy
456 joan data821 nnn
Edit
Ok, first issue you're going to face is combining of the different JSON files that contain arrays. See the following test.
$listb1 = New-TemporaryFile
$listb2 = New-TemporaryFile
#'
[
{
"computerName": "789",
"userName": "Cyril",
"otherData1": "file2.pps",
"otherData2": "yyy"
},
{
"computerName": "321",
"userName": "Damien",
"otherData1": "doc.docx",
"otherData2": "bbb"
}
]
'# | Set-Content $listb1 -Encoding UTF8
#'
[
{
"computerName": "123",
"userName": "Antoine",
"otherData1": "test1.xls",
"otherData2": "yyy"
},
{
"computerName": "123",
"userName": "Antoine",
"otherData1": "test2.xls",
"otherData2": "yyy"
}
]
'# | Set-Content $listb2 -Encoding UTF8
$listObjB = $listb1,$listb2 | foreach {Get-content $_ -Raw | ConvertFrom-Json}
$listObjB.Count
2
You end up with 2 arrays of 2 objects, surely not what you were expecting. However, by simply putting a subexpression around the get-content | convert command, powershell will unroll those into individual items. See this excellent answer by the other commenter Maximillian Burszley for more info.
$listObjB = $listb1,$listb2 | foreach {(Get-content $_ -Raw | ConvertFrom-Json)}
$listObjB.Count
4
That's what we were expecting. We will need to apply the same to the obja list
$lista1 = New-TemporaryFile
$lista2 = New-TemporaryFile
#'
[
{
"computerName": "123",
"userName": "Antoine",
"otherData1": "test1.xls",
"otherData2": "yyy"
},
{
"computerName": "123",
"userName": "Antoine",
"otherData1": "test2.xls",
"otherData2": "yyy"
},
{
"computerName": "456",
"userName": "Benoit",
"otherData1": "file1.txt",
"otherData2": "yyy"
}
]
'# | Set-Content $lista1 -Encoding UTF8
#'
[
{
"computerName": "789",
"userName": "Cyril",
"otherData1": "file1.pps",
"otherData2": "yyy"
},
{
"computerName": "789",
"userName": "Cyril",
"otherData1": "file2.pps",
"otherData2": "yyy"
},
{
"computerName": "321",
"userName": "Damien",
"otherData1": "doc.docx",
"otherData2": "yyy"
}
]
'# | Set-Content $lista2 -Encoding UTF8
$listobjA = $lista1,$lista2 | foreach {(Get-content $_ -Raw | ConvertFrom-Json)}
Then make a slight change to my suggested solution
foreach($objA in $listobjA)
{
if($matchedobj = $listObjB | where {$_.computername -eq $objA.computername -and
$_.username -eq $objA.username -and
$_.otherData1 -eq $objA.otherdata1})
{
if($objA.otherdata2 -ne $matchedobj.otherData2)
{
$objA
}
}
else
{
$objA
}
}
And we will end up with the expected results.
computerName userName otherData1 otherData2
------------ -------- ---------- ----------
456 Benoit file1.txt yyy
789 Cyril file1.pps yyy
321 Damien doc.docx yyy
You could also do the a conversion as part of the loop like this.
foreach($objA in $lista1,$lista2 | foreach {(Get-content $_ -Raw | ConvertFrom-Json)})
{
if($matchedobj = $listObjB | where {$_.computername -eq $objA.computername -and
$_.username -eq $objA.username -and
$_.otherData1 -eq $objA.otherdata1})
{
if($objA.otherdata2 -ne $matchedobj.otherData2)
{
$objA
}
}
else
{
$objA
}
}
But I personally feel the former was easier to read and understand.

Amending Existing JSON Files in Powershell 4

I have seen many questions around this however none of the answers seem to work (please correct me if i am wrong)
I have created a function that queries an API and saves the result as a json file.
However I would like to amend the file that is saved
powershell code:
$search = ""
function searchMtgApi($searchName){
$uriSafeName = [uri]::EscapeDataString($searchName)
$res = Invoke-WebRequest "https://api.magicthegathering.io/v1/cards?name=$uriSafeName" -UseBasicParsing
$resJson = $res.content | ConvertFrom-Json
$resJson.cards | Format-Table
$resJson.cards | Select name, setName, "quantity" | ConvertTo-Json | Out-File "D:\Magic the Gathering\$searchName.json"
}
while ($search -ne "exit"){
Write-Host #'
To exit, type "Exit".
'#
$search = Read-Host "Search Magic the Gathering.IO by card name"
if($search -ne "exit"){
searchMtgApi $search
}
}
Json generated (searching for the card 'Ponder')
[
{
"name": "Ponder",
"setName": "Commander 2018",
"quantity": null
},
{
"name": "Ponder",
"setName": "Lorwyn",
"quantity": null
},
{
"name": "Ponder",
"setName": "Magic 2010",
"quantity": null
},
{
"name": "Ponder",
"setName": "Magic 2012",
"quantity": null
},
{
"name": "Ponder",
"setName": "Magic Player Rewards 2008",
"quantity": null
},
{
"name": "Ponder",
"setName": "Magic Online Promos",
"quantity": null
}
]
What I would like to do is load the file and amend the "quantity" for a particular set.
Can anyone point me in the right direction?
Update -
I will get the json content from a file:
function updateJsonFile($jsonfile){
$resJson = Get-Content "$jsonfile.json" | ConvertFrom-Json
#Edit Quantity here
$resJson | ConvertFrom-Json | Format-Table
}
Focusing just on the core question, with abridged sample input:
The following selectively updates the quantity property for an object with a given setName property value, converting from and to JSON:
$setName = 'Magic 2010' # the set name of interest
$newQuantity = 42 # new quantity
(#'
[
{
"name": "Ponder",
"setName": "Commander 2018",
"quantity": null
},
{
"name": "Ponder",
"setName": "Lorwyn",
"quantity": null
},
{
"name": "Ponder",
"setName": "Magic 2010",
"quantity": null
}
]
'# | ConvertFrom-Json) |
ForEach-Object {
if ($_.setName -eq $setName) {
$_.quantity = $newQuantity
}
$_ # pass the (possibly updated) object through.
} | ConvertTo-Json
Note the need to enclose the ConvertFrom-Json call in (...), which forces enumeration of the individual objects in the array constructed from the JSON input (see this answer for background information).
The above yields (note the updated last quantity value):
[
{
"name": "Ponder",
"setName": "Commander 2018",
"quantity": null
},
{
"name": "Ponder",
"setName": "Lorwyn",
"quantity": null
},
{
"name": "Ponder",
"setName": "Magic 2010",
"quantity": 42
}
]
Applied to your updateJsonFile function:
function updateJsonFile($jsonfile, $setName, $newQuantity){
$resJson = Get-Content "$jsonfile.json"
# Note the (...) around the ConvertFrom-Json command.
($resJson | ConvertFrom-Json) |
ForEach-Object {
if ($_.setName -eq $setName) {
$_.quantity = $newQuantity
}
$_ # pass the (possibly updated) object through.
} | ConvertTo-Json | Set-Content -Encoding Utf8 $jsonfile
}

How to read a JSON file and rearrange the code

I want to read my JSON file and write the same file with a rearranged code.
My JSON file give me something like that:
{
"one": [
{
"name": { "displayName": "Alex" },
"two": [
{
"date": "20072008",
"data": {
"id": "524556",
"fullname": "Alexandre Gagne",
"name": "Alexandre",
"lastName": "Gagne"
}
}
]
}
]
}
But I want something like this
{
"id": "524556",
"fullname": "Alexandre Gagne",
"name": "Alexandre",
"lastName": "Gagne"
}
Here's the code I use:
$jsonfile118 = 'C:/Users/Web_Q/Desktop/json/8472382-2017-2018.json'
$json118 = Get-Content $jsonfile118 | Out-String | ConvertFrom-Json
$json118.stats.splits
$json118 | Add-Member -Type NoteProperty -Name "id" -Value 524556
$json118 | ConvertTo-Json -Depth 10 | Set-Content $jsonfile118
I also use another piece of code that give me what I want but only in the terminal, when I use Set-Content the output looks like this:
"{ \'id'\ 536453 }"
Here's the code:
(New-Object PSObject |
Add-Member -PassThru NoteProperty id $json118.stats.splits.getId
) | ConvertTo-Json
What you need is,
Convert the Json to Object
Select the required member
Convert back to json
$Data = $JsonString | ConvertFrom-Json
$Data.One.Two.Data | ConvertTo-Json
Using this Flatten-Object cmdlet:
#'
{
"one": [
{
"name": { "displayName": "Alex" },
"two": [
{
"date": "20072008",
"data": {
"id": "524556",
"fullname": "Alexandre Gagne",
"name": "Alexandre",
"lastName": "Gagne"
}
}
]
}
]
}
'# | ConvertFrom-Json | Flatten-Object -Depth 9 -Base "" | ConvertTo-Json
Result:
{
"one.name.displayName": "Alex",
"one.two.date": "20072008",
"one.two.data.id": "524556",
"one.two.data.fullname": "Alexandre Gagne",
"one.two.data.name": "Alexandre",
"one.two.data.lastName": "Gagne"
}
It is not exactly what you defined as "Something like this" because it uses the parent names in the property names. This is to make sure that the names are always unique as it is technical possible to have the same child names under different parents.