Currently i'm attempting to create a script to convert my json files to csv files. At the moment I'm receiving most of the data, but the issue i'm having is expanding out the "glyphs" field to add that data into the csv as well.
Below is an example of the json currently. This is similar to what's already out there, but my problem is with the -depth that the convertfrom-json pulls. It's only going to two levels and I need it to go to four.
{
"nodes": [
{
"id": 23,
"type": "Group",
"label": "DOMAIN ADMINS#.COM",
"glyphs": {
"0": {
"position": "top-right",
"font": "\"Font Awesome 5 Free\"",
"content": "",
"fillColor": "black",
"fontScale": 1.5,
"fontStyle": "900"
}
},
"folded": {
"nodes": {},
"edges": {}
},
$user1 = $env:USERNAME
Get-Content C:\Users\$user1\Documents\json_to_convert.json |
convertfrom-json | select -ExpandProperty nodes |
Export-CSV C:\Users\$user1\Documents\jsonTest_$((Get-Date).ToString('MM-dd-yy')).csv -NoTypeInformation
So i'm just hoping to also input the "position", "font", "content", "fillColor", "fontScale", and "fontstyle" from "glyphs into my CSV file as well. Thos are the only fields i'm having trouble with.
If you want to place the nested objects into separate CSV fields then you do have to think about how to represent that in a csv file. The below can be used to create custom properties that are calculated using an expression.
Get-Content C:\Users\$user1\Documents\json_to_convert.json | ConvertFrom-Json | Select -ExpandProperty nodes | `
Select -Property id,type,label, `
#{Name='Glyphs-position';Expression={$_.glyphs.0 | Select -ExpandProperty position}},`
#{Name='Glyphs-font';Expression={$_.glyphs.0 | Select -ExpandProperty font}},`
#{Name='Glyphs-content';Expression={$_.glyphs.0 | Select -ExpandProperty content}},`
#{Name='Glyphs-fillColor';Expression={$_.glyphs.0 | Select -ExpandProperty fillColor}},`
#{Name='Glyphs-fontScale';Expression={$_.glyphs.0 | Select -ExpandProperty fontScale}},`
#{Name='Glyphs-fontStyle';Expression={$_.glyphs.0 | Select -ExpandProperty fontStyle}}`
| ConvertTo-Csv
This could probably be written more efficiently but it demonstrates how to get the values you are after. The resultant PowerShell object produced is
id : 23
type : Group
label : DOMAIN ADMINS#.COM
Glyphs-position : top-right
Glyphs-font : "Font Awesome 5 Free"
Glyphs-content :
Glyphs-fillColor : black
Glyphs-fontScale : 1.5
Glyphs-fontStyle : 900
and once converted to csv
"id","type","label","Glyphs-position","Glyphs-font","Glyphs-content","Glyphs-fillColor","Glyphs-fontScale","Glyphs-fontStyle","folded"
"23","Group","DOMAIN ADMINS#.COM","top-right","""Font Awesome 5 Free""","","black","1.5","900"
Related
I am trying to get the data into a csv from an InvokeWeb request. Currently there are only 2 different values with the individual values in it, but later also times more. It makes the headers of the table, and the correct number of columns, but throws the 2 different values into a cell in both rows. So how can I prevent it from looking like this? I have already read about AddMember or +=, but did not find the right way.
$valstring = iwr -Proxy http://my-proxy -ProxyUseDefaultCredentials -Method GET -Uri https://my-URL -Headers #{'ContentTyp' = 'application/json';"X-Api-Key" = "my-API-Key"} -UseBasicParsing | Select-Object Content | foreach {$_.Content}
$pathToOutputFile = "H:\Wichtiges\json2csvexport.csv"
$deckung = $valstring | ConvertFrom-Json | ForEach-Object {
foreach ($uuid in $_ ) {
[PsCustomObject] #{
name = $_.name | Out-String
version = $_.version | Out-String
active = $_.active | Out-String
}
}
}
$deckung | Export-Csv $pathToOutputFile -NoTypeInformation -UseCulture
The Json-Data looks like that
[
{
"name": "App1",
"version": "1.1.1",
"uuid": "a74a1969-f57d-437b-943f-d4c3dd3bc4bb",
"active": true,
"metrics": {
"high": 617,
"medium": 1250,
"low": 103,
}
},
{
"name": "App2",
"version": "1.3.1",
"uuid": "59eda14a-56a6-4dc6-8238-dd168bd0df3f",
"active": true,
"metrics": {
"high": 6,
"medium": 13,
"low": 0,
}
}
]
But it always comes out like that (second and third together and fourth and fifth row together):
|name | version | active |
|------|-----------|---------|
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
-----------------------------
How can I change it to get one per row that it looks like:
|name | version | active |
|------|-----------|---------|
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
-----------------------------
Thx
Kind regards
Edit (Solution):
That solution works for me, because I can put things from mectrics into the csv and give the Heading an other name:
Select-Object #{Name='Name';Expression={$_.name}},
#{Name='Version';Expression={$_.version}},
#{Name='Active';Expression={$_.Active}},
#{Name='Risk-High';Expression={$_.metrics.high}}
Edit (Another question):
How can I prevent numbers in the csv from being converted to dates?
Thx
Try the following:
$pathToOutputFile = "H:\Wichtiges\json2csvexport.csv"
# Note:
# * the use of Invoke-*RestMethod*, which has ConvertFrom-Json *built in*.
# * the (...) around it, to ensure that the array is *enumerated*.
(Invoke-RestMehod -Proxy http://my-proxy -ProxyUseDefaultCredentials -Method GET -Uri https://my-URL -Headers #{'ContentType' = 'application/json';"X-Api-Key" = "my-API-Key"}) |
Select-Object name, version, active |
Export-Csv $pathToOutputFile -NoTypeInformation -UseCulture
As for what you tried:
Your foreach ($uuid in $_ ) loop mistakenly used $_ - the array to loop over - instead of $uuid - the iteration variable containing the element at hand.
The simpler alternative, to explicit enumeration is to wrap a call to ConvertFrom-Json / Invoke-RestMethod in (...) in order to automatically force enumeration of an array being returned.
Note that this is no longer necessary in PowerShell (Core) 7+, where arrays are enumerated by default (and you need -NoEnumerate in order to send an array as a whole through the pipeline, which is the invariable default behavior in Windows PowerShell).
For simple stringification of a value, Out-String is the wrong tool, primarily because it - unexpectedly - adds a trailing newline to the output - see GitHub issue #14444.
Use a [string] cast instead; however, given that Export-Csv implicitly performs such a stringification for you, there's no need to do it explicitly, allowing you to use a simple Select-Object to pick the properties of interest, as shown above.
I have two JSON files and want to transfer collection of objects from one file to another. Suppose, the from.json file contains property which represents collection of clients:
"Clients":
[
{
"Name": "Name1",
"Age": "12"
},
{
"Name": "Name2",
"Age": "14"
}
]
to.json file contains an empty collection, "Objects: []" ,which must be filled with objects from from.json. Each objects in toJson variable must contain additional property - Id, so eventually, my "to.json" file should look like this:
"Objects":
[
{
"Id": "{new-id}",
"Name": "Name1",
"Age": "12"
},
{
"Id": "{new-id}",
"Name": "Name1",
"Age": "12"
}
]
I've converted two files into variables:
$fromJson = (Get-Content -Raw -Path {fromPath}) | ConvertFrom-Json
$toJson = (Get-Content -Raw -Path {toPath}) | ConvertFrom-Json
I know that objects from fromJson to toJson can be transferred in the following manner:
toJson.Objects += fromJson.Clients, but that's not enough in my case. I think that it could be done by iterating through fromJson.Clients array but have no idea how to create an object and add it into toJson.Objects collection.
Here's a more efficient solution, based on:
Use of a calculated property with Select-Object, which allows you to place the new property first in the output objects.
Instead of building the array one by one with += (which is inefficient, because a new array must technically be created behind the scenes in every iteration), the solution below lets PowerShell collect the output objects of the Select-Object call in an array automatically (the [array] type constraint is needed to ensure that an array is created even if only one object happens to be output.)
# Sample input.
$fromJson = ConvertFrom-Json '{"Clients":[{"Name":"Name1","Age":"12"},{"Name":"Name2","Age":"14"}]}'
$toJson = ConvertFrom-Json '{ "Objects": [] }'
[array] $toJson.Objects =
$fromJson.Clients |
Select-Object #{ Name='Id'; Expression = { [string] (New-Guid) } }, *
$toJson | ConvertTo-Json -Depth 3 # append | Set-Content as needed.
Kind of new to the PowerShell, but after a bit of investigation came up with the following solution:
fromJson.Clients | ForEach-Object {
$_ | Add-Member -MemberType NoteProperty -Name 'Id' -Value ([guid]::NewGuid().Guid.ToString())
$toJson += $_
}
...
$toJson | ConvertTo-Json | Out-File {to.json_path}
Frankly, don't know if that is a 'proper' way to do that, but generally it works for that particular case. For now, see no other solution.
I have searched looking for examples of converting complex json (embedded arrays] into csv file(s) using Powershell. The goal is to accept json data
into a MSAccess database. MSAccess does not provide an intrinsic function to do this. I am new to Powershell and json, but I did discover the ConvertFrom-JSON cmdlet that got my interest.
The best info I have found is the
Flatten-Object function by iRon
in response to this article
PowerShell convert nested JSON array into separate columns in CSV file
While this function works to create a single csv, I am interested in creating multiple csv files if there are embedded arrays in the json. The idea is to create a csv file for data at each level. Level 2 and lower will require a link field (id/name) to be used as a primary key in level1, and as foreign key in level2. A PK field at the level2, would be included as a foreign key at level3 and so on. Since Access can import csv data to a table, my feeling is that getting the data into "normalized" csv files would be a repeatable method to get json data into an MSAccess database.
So with respect to my objective and the Flatten-Object function, I am looking for advice/direction on the following:
Could the function be adjusted/used to identify
the levels in the json file,
to create a csv for each of those levels with
a selectable PK field(s) to relate
the csv data files in a normalized manner for import to MSAccess??
I do realize that some human intervention will be required for each json file. So I'm looking for an approach that simplifies the effort and is repeatable.
I have created a simple script to take a simple json file (no embedded array) and convert it to CSV. I have used the Shell command in vba to execute the PS script.
<#CarsBasic.ps1
.DESCRIPTION
This script takes the cars.json file and reads it into memory
Converts it from Json, then selects id,manufacturer,year from the result
and exports the data to C:\Programs\CarsJack.csv as a csv file with header
#>
(Get-Content C:\Programs\MendipDataSystems\JSONParser\Files\Cars.json -Raw |
ConvertFrom-Json) |Select id,manufacturer,year |
Export-CSV c:\programs\CarsJack.csv -NoTypeInformation
Thanks in advance.
I have adjusted this post based on request/comment by iRon.
Sample json file that has Squad, SquadMember and SquadMemberPower levels. I would like to get a Squad.csv that has Squad info, and a SquadMember.csv that has the Squadname and each of the Member details, and a SquadmemberPower csv that has the SquadName and the Member Name identifying to whom that Power belongs. in effect, these 3 csv files would be loaded into MSAccess as 3 normalized tables. This is my test case, but I'd like a more general, reusable approach--if possible. Here is the MultiSquad.json
[{
"squadName": "Super hero squad Alpha",
"homeTown": "Metro City",
"formed": 2016,
"secretBase": "Large tent in the forest",
"active": "True",
"members": [{
"name": "Molecule Man",
"age": 29,
"secretIdentity": "Dan Jukes",
"powers": ["Radiation resistance",
"Turning tiny",
"Radiation blast"]
},
{
"name": "Madame Uppercut",
"age": 39,
"secretIdentity": "Jane Wilson",
"powers": ["Million tonne punch",
"Damage resistance",
"Superhuman reflexes"]
},
{
"name": "Eternal Flame",
"age": 1000000,
"secretIdentity": "Unknown",
"powers": ["Immortality",
"Heat Immunity",
"Inferno",
"Teleportation",
"Interdimensional travel"]
}]
},
{
"squadName": "Second squad Baker",
"homeTown": "Metro Toronto",
"formed": 2017,
"secretBase": "CN tower",
"active": "True",
"members": [{
"name": "Kathleen Wynne",
"age": 49,
"secretIdentity": "Cyan Arrah",
"powers": ["XRay vision",
"Invisibility",
"Radiation blast"]
},
{
"name": "Madame Butterfly",
"age": 27,
"secretIdentity": "Iman Angel",
"powers": ["Magical hearing",
"Fantastic ideas"]
},
{
"name": "Gassy Misty Cloud",
"age": 1000,
"secretIdentity": "Puff of Smoke",
"powers": ["Immortality",
"Heat and Flame Immunity",
"Impeccable hearing",
"Xray Vision",
"Able to jump tall buildings",
"Teleportation",
"Intergalactic travel"]
}]
}]
Expected Output: 3 csv files
1) Squad.csv with fields
"squadName","homeTown","formed","secretBase","active"
2) SquadMembers.csv with fields
"squadName","name","age","secretIdentity"
3)SquadMemberPowers.csv with fields
"Name","powers"
Specific solution
Assuming that $JSON contains your JSON object:
$Squads = #(); $SquadMembers = #(); $SquadMemberPowers = #()
ForEach ($Squad In $JSON) {
$Squads += New-Object PSObject ($Squad | Select squadName, homeTown, formed, secretBase, active)
ForEach ($member In $Squad.members) {
$SquadMembers += New-Object PSObject ($member | Select #{label = "squadName" ;expression = {$Squad.squadName}}, name, age, secretIdentity)
ForEach ($power In $member.powers) {
$SquadMemberPowers += New-Object PSObject ($member | Select #{label = "name" ;expression = {$member.name}}, #{label = "powers" ;expression = {$power}})
}
}
}
$Squads | Export-CSV ".\Squad.csv" -NoTypeInformation
$SquadMembers | Export-CSV ".\SquadMembers.csv" -NoTypeInformation
$SquadMemberPowers | Export-CSV ".\SquadMemberPowers.csv" -NoTypeInformation
General solution
With regards to a general (reusable) solution, I don't think that your request is general enough for that: at the members level you have an array with hashtables that you want to enumerate, at the powers level you like to transpose the array and than you want to pickup some properties from the parent that are not common (squadname vs name. You might consider here to refer to the first property but hashtables in PowerShell do not always stay in order, see: Powershell Hashtables Key Order).
In other words, for a general solution you will need to supply so many arguments that there will not much of an added value in comparison specific script as purposed above and changing it's adjusting it's functions and variables.
First get the json to an object:
$obj = Get-Content C:/input.json | ConvertFrom-Json
Then you have at least two ways how to select the items you want.
Simple selection:
$obj | select squadName, homeTown, formed, secretBase, active | Convertto-csv > c:\squads.csv
Complex selection:
$members = $obj | foreach {
$squadName = $_.squadName
$_.members | foreach {
[pscustomobject]#{
squadName = $squadName
name = $_.name
age = $_.age
secretIdentity = $_.secretIdentity
}
}
}
$members | ConvertTo-Csv > c:\members.csv
$powers = $obj.members | foreach {
$memberName = $_.name
$_.powers | foreach {
[pscustomobject]#{
name = $memberName
power = $_
}
}
}
$powers | ConvertTo-Csv > c:\powers.csv
The below command can be used for separating the csv data into columns with the delimiter ",".
For Example:
Import-Csv "C:\Result.csv" -delimiter "," | Sort-Object _from -Unique | Export-csv "C:\FINAL_REPORT.csv"
I'm trying to parse a JSON string which has been flattened from a relational dataset from the looks of it... Into a CSV in order to import into an Oracle database. I have name/value pairs similar to this (simplified) example
Device.1.Service.1.Channel.1.someProperty : 1,
Device.1.Service.1.Channel.1.someOtherProperty : "billy",
Device.1.Service.1.Channel.2.someProperty : 8,
Device.1.Service.1.Channel.2.someOtherProperty : "frank",
Device.1.Service.1.Channel.3.someProperty : 12,
Device.1.Service.1.Channel.3.someOtherProperty : "sam",
Device.1.Service.2.Channel.1.someProperty : 3,
Device.1.Service.2.Channel.1.someOtherProperty : "john",
EDIT: A .json file with a similar structure is produced per day per device. So when consolidated (Get-Content .\*.json -Raw) I see the same property names occur many times in console.
As part of the conversion, I'd like parts of the property name to become fields in the database. This will enable us to visualise the data better using dynamic slider filters etc. later.
| Device | Service | Channel | someOtherProperty |
| 1 | 1 | 1 | billy |
| 1 | 1 | 2 | frank |
| 1 | 1 | 3 | sam |
Right now, I'm using the cmdlet ConvertFrom-Json. I'm then selecting the fields (out of nearly 2000 possible fields) using wildcards. The number of channels for example is dynamic.
Get-Content .\*.json -Raw | ConvertFrom-Json |
Select Device.1.Service.1.Channel.?.someOtherProperty
Which returns a PSCustomObject. I'd like to figure out the channel number and use that as a derived field. Pseudo example:
Select #{n="Channel";e={$_.getCharAtWildcard()}},
$_.theValueofTheCurrentObject()
If I pipe the data selected (using wildcard) to Get-Member, the PSCostomObject contains method names as well as my field names (but not the values).
I'm confused about how I access the field name (since 'Name' gives blank rows), then extract the channel number (the character at wildcard position), and then the value to structure my output correctly.
Any pointers? Googled and have a lynda.com sub, but can't seem to find a solution to this specific problem - probably because I'm not using the correct terms?
## ANSGAR's SOLUTION - WORKS FOR SINGLE FILE ##
$dataDir = "C:\ps_json2csv\dummydata"
CD $dataDir
$dict = Get-Content .\*.json -Raw | ConvertFrom-Json | Select -Expand data
$p = 'DvbId'
$re = "frontend\.(\d+)\.logicalchannel\.(\d+)\.service\.(\d+)\..*?\.$p"
## modified regex to match my data, example string: FrontEnd.2.LogicalChannel.3.Service.1.stat.DvbId
$fullset = $dict.PSObject.Properties | Where-Object {
$_.Name -match $re
} | ForEach-Object {
$prop = [ordered]#{
FrontEnd = $matches[1]
LogicalChannel = $matches[2]
Service = $matches[3]
$p = $_.Value
}
New-Object -Type PSObject -Property $prop
}
## inspect $dict - its populated
## inspect $fullset - its empty! :(
Data of which is 2 files contained in C:\ps_json2csv\dummydata :
File1.json
{
"data": {
"Device.1.Service.1.ChannelInfo.Channel.1.Stats.someProperty" : "1",
"Device.1.Service.1.ChannelInfo.Channel.2.Stats.someProperty" : "8",
"Device.1.Service.1.ChannelInfo.Channel.3.Stats.someProperty" : "12",
"FrontEnd.2.LogicalChannel.3.Service.1.stat.DvbId" : "john",
"FrontEnd.2.LogicalChannel.3.Service.2.stat.DvbId" : "billy",
"FrontEnd.2.LogicalChannel.3.Service.3.stat.DvbId" : "frank",
"FrontEnd.2.LogicalChannel.4.Service.1.stat.DvbId" : "sam",
"Device.1.Service.2.ChannelInfo.Channel.1.Stats.someProperty" : "3",
"Some.value.im.not.intersted.in.just.yet": "Sat Jan 1 00:00:00 GMT 0001",
"foo.bar" : "0",
"random.stuff" : "hi there"
}
}
File2.json
{
"data": {
"Device.1.Service.1.ChannelInfo.Channel.1.Stats.someProperty" : "0",
"Device.1.Service.1.ChannelInfo.Channel.2.Stats.someProperty" : "7",
"Device.1.Service.1.ChannelInfo.Channel.3.Stats.someProperty" : "6",
"FrontEnd.2.LogicalChannel.3.Service.1.stat.DvbId" : "john",
"FrontEnd.2.LogicalChannel.3.Service.2.stat.DvbId" : "billy",
"FrontEnd.2.LogicalChannel.3.Service.3.stat.DvbId" : "frank",
"FrontEnd.2.LogicalChannel.4.Service.1.stat.DvbId" : "sam",
"Device.1.Service.2.ChannelInfo.Channel.1.Stats.someProperty" : "4",
"Some.value.im.not.intersted.in.just.yet": "Sun Jan 2 00:00:00 GMT 0001",
"foo.bar" : "0",
"random.stuff" : "hi there"
}
}
I would probably use a regular expression match to filter and extract your data:
$p = 'someOtherProperty'
$re = "device\.(\d+)\.service\.(\d+)\..*?\.channel\.(\d+)\..*?\.$p"
$fullset = $dict.PSObject.Properties | Where-Object {
$_.Name -match $re
} | ForEach-Object {
$prop = [ordered]#{
Device = $matches[1]
Service = $matches[2]
Channel = $matches[3]
$p = $_.Value
}
New-Object -Type PSObject -Property $prop
}
I'd rather convert a string of this syntax into a PSObject with the entire set of your properties. Like this:
$dict=Get-Content .\*.json -Raw | ConvertFrom-Json
$fullset=$dict.psobject.properties | % {
$parse=$_.name.split('.')
$parse+=,$_.value # since JSON values might be non-plain, we need to add the value as single object
$obj=new-object psobject
for ($i=0;$i -lt $parse.length;$i+=2) { # name-value
$v1=$parse[$i]
$v2=$parse[1+$i]
$obj | add-member -type noteproperty -name $v1 -value $v2
}
$obj
}
Then you parse $fullset like you would with a normal list with where {$_.device -eq '1' -and $_.someOtherProperty -ne $null} etc.
I have a DataTable $dt with same data, I would like to pipe the data to JSON using the cmdlet ConvertTo-JSON in Powershell v3
$ds.Tables["mytable"] | ConvertTo-Json
The result is all the properties of the DataTable are returned, but I only need the records in the data table.
I am wondering if there is a way to do this without looking through each column/ row and adding then into a custom object..etc.
This is what I get when I run the above line;
[
{
"RowError": "",
"RowState": 2,
"Table": {
"CaseSensitive": false,
"IsInitialized": true,
"RemotingFormat": 0,
"ChildRelations": "",
"Columns": "Id Name IsActive",
"Constraints": "",
"DataSet": "System.Data.DataSet",
"DefaultView": "System.Data.DataRowView System.Data.DataRowView",
"DisplayExpression": "",
"ExtendedProperties": "System.D.......
Thanks
Yasir
http://www.sqlist.co.uk
After playing with a sample datatable a bit, I ended up with this:
($ds.Tables["mytable"] | select $ds.Tables["mytable"].Columns.ColumnName ) | ConvertTo-Json
You can also exclude properties with Select-Object, removing the noise.
$dt | Select-Object * -ExcludeProperty ItemArray, Table, RowError, RowState, HasErrors | ConvertTo-Json