I have a DataTable $dt with same data, I would like to pipe the data to JSON using the cmdlet ConvertTo-JSON in Powershell v3
$ds.Tables["mytable"] | ConvertTo-Json
The result is all the properties of the DataTable are returned, but I only need the records in the data table.
I am wondering if there is a way to do this without looking through each column/ row and adding then into a custom object..etc.
This is what I get when I run the above line;
[
{
"RowError": "",
"RowState": 2,
"Table": {
"CaseSensitive": false,
"IsInitialized": true,
"RemotingFormat": 0,
"ChildRelations": "",
"Columns": "Id Name IsActive",
"Constraints": "",
"DataSet": "System.Data.DataSet",
"DefaultView": "System.Data.DataRowView System.Data.DataRowView",
"DisplayExpression": "",
"ExtendedProperties": "System.D.......
Thanks
Yasir
http://www.sqlist.co.uk
After playing with a sample datatable a bit, I ended up with this:
($ds.Tables["mytable"] | select $ds.Tables["mytable"].Columns.ColumnName ) | ConvertTo-Json
You can also exclude properties with Select-Object, removing the noise.
$dt | Select-Object * -ExcludeProperty ItemArray, Table, RowError, RowState, HasErrors | ConvertTo-Json
Related
I am trying to get the data into a csv from an InvokeWeb request. Currently there are only 2 different values with the individual values in it, but later also times more. It makes the headers of the table, and the correct number of columns, but throws the 2 different values into a cell in both rows. So how can I prevent it from looking like this? I have already read about AddMember or +=, but did not find the right way.
$valstring = iwr -Proxy http://my-proxy -ProxyUseDefaultCredentials -Method GET -Uri https://my-URL -Headers #{'ContentTyp' = 'application/json';"X-Api-Key" = "my-API-Key"} -UseBasicParsing | Select-Object Content | foreach {$_.Content}
$pathToOutputFile = "H:\Wichtiges\json2csvexport.csv"
$deckung = $valstring | ConvertFrom-Json | ForEach-Object {
foreach ($uuid in $_ ) {
[PsCustomObject] #{
name = $_.name | Out-String
version = $_.version | Out-String
active = $_.active | Out-String
}
}
}
$deckung | Export-Csv $pathToOutputFile -NoTypeInformation -UseCulture
The Json-Data looks like that
[
{
"name": "App1",
"version": "1.1.1",
"uuid": "a74a1969-f57d-437b-943f-d4c3dd3bc4bb",
"active": true,
"metrics": {
"high": 617,
"medium": 1250,
"low": 103,
}
},
{
"name": "App2",
"version": "1.3.1",
"uuid": "59eda14a-56a6-4dc6-8238-dd168bd0df3f",
"active": true,
"metrics": {
"high": 6,
"medium": 13,
"low": 0,
}
}
]
But it always comes out like that (second and third together and fourth and fifth row together):
|name | version | active |
|------|-----------|---------|
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
-----------------------------
How can I change it to get one per row that it looks like:
|name | version | active |
|------|-----------|---------|
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
-----------------------------
Thx
Kind regards
Edit (Solution):
That solution works for me, because I can put things from mectrics into the csv and give the Heading an other name:
Select-Object #{Name='Name';Expression={$_.name}},
#{Name='Version';Expression={$_.version}},
#{Name='Active';Expression={$_.Active}},
#{Name='Risk-High';Expression={$_.metrics.high}}
Edit (Another question):
How can I prevent numbers in the csv from being converted to dates?
Thx
Try the following:
$pathToOutputFile = "H:\Wichtiges\json2csvexport.csv"
# Note:
# * the use of Invoke-*RestMethod*, which has ConvertFrom-Json *built in*.
# * the (...) around it, to ensure that the array is *enumerated*.
(Invoke-RestMehod -Proxy http://my-proxy -ProxyUseDefaultCredentials -Method GET -Uri https://my-URL -Headers #{'ContentType' = 'application/json';"X-Api-Key" = "my-API-Key"}) |
Select-Object name, version, active |
Export-Csv $pathToOutputFile -NoTypeInformation -UseCulture
As for what you tried:
Your foreach ($uuid in $_ ) loop mistakenly used $_ - the array to loop over - instead of $uuid - the iteration variable containing the element at hand.
The simpler alternative, to explicit enumeration is to wrap a call to ConvertFrom-Json / Invoke-RestMethod in (...) in order to automatically force enumeration of an array being returned.
Note that this is no longer necessary in PowerShell (Core) 7+, where arrays are enumerated by default (and you need -NoEnumerate in order to send an array as a whole through the pipeline, which is the invariable default behavior in Windows PowerShell).
For simple stringification of a value, Out-String is the wrong tool, primarily because it - unexpectedly - adds a trailing newline to the output - see GitHub issue #14444.
Use a [string] cast instead; however, given that Export-Csv implicitly performs such a stringification for you, there's no need to do it explicitly, allowing you to use a simple Select-Object to pick the properties of interest, as shown above.
I have two JSON files and want to transfer collection of objects from one file to another. Suppose, the from.json file contains property which represents collection of clients:
"Clients":
[
{
"Name": "Name1",
"Age": "12"
},
{
"Name": "Name2",
"Age": "14"
}
]
to.json file contains an empty collection, "Objects: []" ,which must be filled with objects from from.json. Each objects in toJson variable must contain additional property - Id, so eventually, my "to.json" file should look like this:
"Objects":
[
{
"Id": "{new-id}",
"Name": "Name1",
"Age": "12"
},
{
"Id": "{new-id}",
"Name": "Name1",
"Age": "12"
}
]
I've converted two files into variables:
$fromJson = (Get-Content -Raw -Path {fromPath}) | ConvertFrom-Json
$toJson = (Get-Content -Raw -Path {toPath}) | ConvertFrom-Json
I know that objects from fromJson to toJson can be transferred in the following manner:
toJson.Objects += fromJson.Clients, but that's not enough in my case. I think that it could be done by iterating through fromJson.Clients array but have no idea how to create an object and add it into toJson.Objects collection.
Here's a more efficient solution, based on:
Use of a calculated property with Select-Object, which allows you to place the new property first in the output objects.
Instead of building the array one by one with += (which is inefficient, because a new array must technically be created behind the scenes in every iteration), the solution below lets PowerShell collect the output objects of the Select-Object call in an array automatically (the [array] type constraint is needed to ensure that an array is created even if only one object happens to be output.)
# Sample input.
$fromJson = ConvertFrom-Json '{"Clients":[{"Name":"Name1","Age":"12"},{"Name":"Name2","Age":"14"}]}'
$toJson = ConvertFrom-Json '{ "Objects": [] }'
[array] $toJson.Objects =
$fromJson.Clients |
Select-Object #{ Name='Id'; Expression = { [string] (New-Guid) } }, *
$toJson | ConvertTo-Json -Depth 3 # append | Set-Content as needed.
Kind of new to the PowerShell, but after a bit of investigation came up with the following solution:
fromJson.Clients | ForEach-Object {
$_ | Add-Member -MemberType NoteProperty -Name 'Id' -Value ([guid]::NewGuid().Guid.ToString())
$toJson += $_
}
...
$toJson | ConvertTo-Json | Out-File {to.json_path}
Frankly, don't know if that is a 'proper' way to do that, but generally it works for that particular case. For now, see no other solution.
Currently i'm attempting to create a script to convert my json files to csv files. At the moment I'm receiving most of the data, but the issue i'm having is expanding out the "glyphs" field to add that data into the csv as well.
Below is an example of the json currently. This is similar to what's already out there, but my problem is with the -depth that the convertfrom-json pulls. It's only going to two levels and I need it to go to four.
{
"nodes": [
{
"id": 23,
"type": "Group",
"label": "DOMAIN ADMINS#.COM",
"glyphs": {
"0": {
"position": "top-right",
"font": "\"Font Awesome 5 Free\"",
"content": "",
"fillColor": "black",
"fontScale": 1.5,
"fontStyle": "900"
}
},
"folded": {
"nodes": {},
"edges": {}
},
$user1 = $env:USERNAME
Get-Content C:\Users\$user1\Documents\json_to_convert.json |
convertfrom-json | select -ExpandProperty nodes |
Export-CSV C:\Users\$user1\Documents\jsonTest_$((Get-Date).ToString('MM-dd-yy')).csv -NoTypeInformation
So i'm just hoping to also input the "position", "font", "content", "fillColor", "fontScale", and "fontstyle" from "glyphs into my CSV file as well. Thos are the only fields i'm having trouble with.
If you want to place the nested objects into separate CSV fields then you do have to think about how to represent that in a csv file. The below can be used to create custom properties that are calculated using an expression.
Get-Content C:\Users\$user1\Documents\json_to_convert.json | ConvertFrom-Json | Select -ExpandProperty nodes | `
Select -Property id,type,label, `
#{Name='Glyphs-position';Expression={$_.glyphs.0 | Select -ExpandProperty position}},`
#{Name='Glyphs-font';Expression={$_.glyphs.0 | Select -ExpandProperty font}},`
#{Name='Glyphs-content';Expression={$_.glyphs.0 | Select -ExpandProperty content}},`
#{Name='Glyphs-fillColor';Expression={$_.glyphs.0 | Select -ExpandProperty fillColor}},`
#{Name='Glyphs-fontScale';Expression={$_.glyphs.0 | Select -ExpandProperty fontScale}},`
#{Name='Glyphs-fontStyle';Expression={$_.glyphs.0 | Select -ExpandProperty fontStyle}}`
| ConvertTo-Csv
This could probably be written more efficiently but it demonstrates how to get the values you are after. The resultant PowerShell object produced is
id : 23
type : Group
label : DOMAIN ADMINS#.COM
Glyphs-position : top-right
Glyphs-font : "Font Awesome 5 Free"
Glyphs-content :
Glyphs-fillColor : black
Glyphs-fontScale : 1.5
Glyphs-fontStyle : 900
and once converted to csv
"id","type","label","Glyphs-position","Glyphs-font","Glyphs-content","Glyphs-fillColor","Glyphs-fontScale","Glyphs-fontStyle","folded"
"23","Group","DOMAIN ADMINS#.COM","top-right","""Font Awesome 5 Free""","","black","1.5","900"
I have searched looking for examples of converting complex json (embedded arrays] into csv file(s) using Powershell. The goal is to accept json data
into a MSAccess database. MSAccess does not provide an intrinsic function to do this. I am new to Powershell and json, but I did discover the ConvertFrom-JSON cmdlet that got my interest.
The best info I have found is the
Flatten-Object function by iRon
in response to this article
PowerShell convert nested JSON array into separate columns in CSV file
While this function works to create a single csv, I am interested in creating multiple csv files if there are embedded arrays in the json. The idea is to create a csv file for data at each level. Level 2 and lower will require a link field (id/name) to be used as a primary key in level1, and as foreign key in level2. A PK field at the level2, would be included as a foreign key at level3 and so on. Since Access can import csv data to a table, my feeling is that getting the data into "normalized" csv files would be a repeatable method to get json data into an MSAccess database.
So with respect to my objective and the Flatten-Object function, I am looking for advice/direction on the following:
Could the function be adjusted/used to identify
the levels in the json file,
to create a csv for each of those levels with
a selectable PK field(s) to relate
the csv data files in a normalized manner for import to MSAccess??
I do realize that some human intervention will be required for each json file. So I'm looking for an approach that simplifies the effort and is repeatable.
I have created a simple script to take a simple json file (no embedded array) and convert it to CSV. I have used the Shell command in vba to execute the PS script.
<#CarsBasic.ps1
.DESCRIPTION
This script takes the cars.json file and reads it into memory
Converts it from Json, then selects id,manufacturer,year from the result
and exports the data to C:\Programs\CarsJack.csv as a csv file with header
#>
(Get-Content C:\Programs\MendipDataSystems\JSONParser\Files\Cars.json -Raw |
ConvertFrom-Json) |Select id,manufacturer,year |
Export-CSV c:\programs\CarsJack.csv -NoTypeInformation
Thanks in advance.
I have adjusted this post based on request/comment by iRon.
Sample json file that has Squad, SquadMember and SquadMemberPower levels. I would like to get a Squad.csv that has Squad info, and a SquadMember.csv that has the Squadname and each of the Member details, and a SquadmemberPower csv that has the SquadName and the Member Name identifying to whom that Power belongs. in effect, these 3 csv files would be loaded into MSAccess as 3 normalized tables. This is my test case, but I'd like a more general, reusable approach--if possible. Here is the MultiSquad.json
[{
"squadName": "Super hero squad Alpha",
"homeTown": "Metro City",
"formed": 2016,
"secretBase": "Large tent in the forest",
"active": "True",
"members": [{
"name": "Molecule Man",
"age": 29,
"secretIdentity": "Dan Jukes",
"powers": ["Radiation resistance",
"Turning tiny",
"Radiation blast"]
},
{
"name": "Madame Uppercut",
"age": 39,
"secretIdentity": "Jane Wilson",
"powers": ["Million tonne punch",
"Damage resistance",
"Superhuman reflexes"]
},
{
"name": "Eternal Flame",
"age": 1000000,
"secretIdentity": "Unknown",
"powers": ["Immortality",
"Heat Immunity",
"Inferno",
"Teleportation",
"Interdimensional travel"]
}]
},
{
"squadName": "Second squad Baker",
"homeTown": "Metro Toronto",
"formed": 2017,
"secretBase": "CN tower",
"active": "True",
"members": [{
"name": "Kathleen Wynne",
"age": 49,
"secretIdentity": "Cyan Arrah",
"powers": ["XRay vision",
"Invisibility",
"Radiation blast"]
},
{
"name": "Madame Butterfly",
"age": 27,
"secretIdentity": "Iman Angel",
"powers": ["Magical hearing",
"Fantastic ideas"]
},
{
"name": "Gassy Misty Cloud",
"age": 1000,
"secretIdentity": "Puff of Smoke",
"powers": ["Immortality",
"Heat and Flame Immunity",
"Impeccable hearing",
"Xray Vision",
"Able to jump tall buildings",
"Teleportation",
"Intergalactic travel"]
}]
}]
Expected Output: 3 csv files
1) Squad.csv with fields
"squadName","homeTown","formed","secretBase","active"
2) SquadMembers.csv with fields
"squadName","name","age","secretIdentity"
3)SquadMemberPowers.csv with fields
"Name","powers"
Specific solution
Assuming that $JSON contains your JSON object:
$Squads = #(); $SquadMembers = #(); $SquadMemberPowers = #()
ForEach ($Squad In $JSON) {
$Squads += New-Object PSObject ($Squad | Select squadName, homeTown, formed, secretBase, active)
ForEach ($member In $Squad.members) {
$SquadMembers += New-Object PSObject ($member | Select #{label = "squadName" ;expression = {$Squad.squadName}}, name, age, secretIdentity)
ForEach ($power In $member.powers) {
$SquadMemberPowers += New-Object PSObject ($member | Select #{label = "name" ;expression = {$member.name}}, #{label = "powers" ;expression = {$power}})
}
}
}
$Squads | Export-CSV ".\Squad.csv" -NoTypeInformation
$SquadMembers | Export-CSV ".\SquadMembers.csv" -NoTypeInformation
$SquadMemberPowers | Export-CSV ".\SquadMemberPowers.csv" -NoTypeInformation
General solution
With regards to a general (reusable) solution, I don't think that your request is general enough for that: at the members level you have an array with hashtables that you want to enumerate, at the powers level you like to transpose the array and than you want to pickup some properties from the parent that are not common (squadname vs name. You might consider here to refer to the first property but hashtables in PowerShell do not always stay in order, see: Powershell Hashtables Key Order).
In other words, for a general solution you will need to supply so many arguments that there will not much of an added value in comparison specific script as purposed above and changing it's adjusting it's functions and variables.
First get the json to an object:
$obj = Get-Content C:/input.json | ConvertFrom-Json
Then you have at least two ways how to select the items you want.
Simple selection:
$obj | select squadName, homeTown, formed, secretBase, active | Convertto-csv > c:\squads.csv
Complex selection:
$members = $obj | foreach {
$squadName = $_.squadName
$_.members | foreach {
[pscustomobject]#{
squadName = $squadName
name = $_.name
age = $_.age
secretIdentity = $_.secretIdentity
}
}
}
$members | ConvertTo-Csv > c:\members.csv
$powers = $obj.members | foreach {
$memberName = $_.name
$_.powers | foreach {
[pscustomobject]#{
name = $memberName
power = $_
}
}
}
$powers | ConvertTo-Csv > c:\powers.csv
The below command can be used for separating the csv data into columns with the delimiter ",".
For Example:
Import-Csv "C:\Result.csv" -delimiter "," | Sort-Object _from -Unique | Export-csv "C:\FINAL_REPORT.csv"
Powershell command-let ConvertTo-Json has the following limitations
1) It returns Enum values as Integers instead of their text
2) It doesn't return the date in a readable format
For point #1 see below, Status and VerificationMethod Properties
PS C:\Windows\system32> Get-Msoldomain | ConvertTo-Json
{
"ExtensionData": {
},
"Authentication": 0,
"Capabilities": 5,
"IsDefault": true,
"IsInitial": true,
"Name": "myemail.onmicrosoft.com",
"RootDomain": null,
"Status": 1,
"VerificationMethod": 1
}
To handle this, I changed my command as below
PS C:\Windows\system32> Get-Msoldomain | ConvertTo-Csv | ConvertFrom-Csv | ConvertTo-Json
{
"ExtensionData": "System.Runtime.Serialization.ExtensionDataObject",
"Authentication": "Managed",
"Capabilities": "Email, OfficeCommunicationsOnline",
"IsDefault": "True",
"IsInitial": "True",
"Name": "curtisjmspartnerhotmail.onmicrosoft.com",
"RootDomain": "",
"Status": "Verified",
"VerificationMethod": "DnsRecord"
}
Now you see, that the enums are being returned with their text values above (Status and VerificationMethod) instead of their integer values.
However, There are a few limitations with this approach:
1) ConvertTo-Csv doesn't retain the Arrays or Complex Objects, and
outputs them as their Class Names (Watch the ExtensionData Properties
in both the outputs). In the second output, we tend to lose the data,
and just get the className
System.Runtime.Serialization.ExtensionDataObject as a string
2) Both ConvertTo-Csv and ConvertFrom-Csv are not the script-level
commandlets, but they are command-level commandlets, which means that
we can't use them at the end of the script , but they will have to be
used with the individual commands like I am doing above. WHEREAS,
ConvertTo-Json need not be applied at the commmandLevel, but just
applied once for the script output.
My question is:
1) How do I still use the convertTo-Json, so that all my enum properties are returned with their texts and not integers, and ALSO the Complex Objects or Arrays are not lost? In the approach I have used, the complex objects are getting lost
2) Also, it should be generic enough so that It can be applied at the end of the script, and not at the command level
ConvertTo-Json and ConvertTo-Csv are both forms of serializing objects in some sort of text representation and both are useful in different use cases.
ConvertTo-Csv is perhaps best used for 2-dimensional data that can be expressed in a table such as a spreadsheet. Hence it is awkward to try to convert "complex" objects, i.e. those with properties that contain other structured data, into a simple table. In such cases PowerShell represents such data as the full name of the data type.
ConvertTo-Json is capable of serializing more complicated objects, since the format allows for nested arrays/data structures, e.g. the ExtensionData property in your example. Note that you may need to use the -Depth parameter to ensure that deeply nested data is serialized correctly.
So the problem really comes down to how the ConvertTo-Json cmdlet serializes enums, which can be demonstrated with:
[PS]> (Get-Date).DayOfWeek
Tuesday
[PS]> (Get-Date).DayOfWeek.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True DayOfWeek System.Enum
[PS]> Get-Date | Select DayOfWeek | ConvertTo-Json
{
"DayOfWeek": 2
}
So before you convert to JSON you need to ensure that the DayOfWeek property (in this example) or Status and VerificationMethod properties (from your example) are converted to their string equivalents first.
You can do this using an expression with Select-Object to convert the data as it passes down the pipe. Note that you do need to include all the properties that you want included in the final JSON:
[PS]> Get-Date |
Select DateTime,#{Label="DayOfWeek";Expression={$_.DayOfWeek.ToString()}} |
ConvertTo-Json
{
"DateTime": "13 June 2017 10:33:51",
"DayOfWeek": "Tuesday"
}
So in your case you'd need something like this:
[PS]> Get-Msoldomain |
Select-Object ExtensionData,IsDefault,IsInitial,Name,RootDomain `
,{Label="Authentication";Expression={$_.Authentication.ToString()}} `
,{Label="Capabilities";Expression={$_.Capabilities.ToString()}} `
,{Label="Status";Expression={$_.Status.ToString()}} `
,{Label="VerificationMethod";Expression={$_.VerificationMethod.ToString()}} |
ConvertTo-Json
#puneet, following your comment on my other answer, here is an example of how you might build up a new object, based on an existing one, with the Enum types converted to strings.
The idea is to create a new "empty" object, then loop through all the properties of the original object and add them to the new one, but if any of the original properties are Enums, then those are converted to strings.
$data = [PSCustomObject]#{}
(Get-Date).PSObject.Properties | Select Name,Value | Foreach-Object {
if($_.Value.GetType().BaseType.FullName -eq "System.Enum"){
$data | Add-Member -MemberType NoteProperty -Name $_.Name -Value $_.Value.ToString()
}
else {
$data | Add-Member -MemberType NoteProperty -Name $_.Name -Value $_.Value
}
}
$data | ConvertTo-Json
You may want to finesse this a little for your own application, but hopefully the idea behind it is clear. Definitely check to see that all the properties are being treated correctly in the JSON output.
to keep enum,array and date when converting psObject to json, you can use newtonsoft. a sample here https://github.com/chavers/powershell-newtonsoft using Nerdy Mishka powershell module.
$obj = New-Object pscustomobject -Property #{Enum = (Get-DAte).DayOfWeek; int = 2; string = "du text"; array = #("un", "deux", "trois"); obj= #{enum = (Get-DAte).DayOfWeek; int = 2; string = "du text"; array = #("un", "deux", "trois")}}
Import-Module Fmg-PrettyJson
$settings = Get-NewtonsoftJsonSettings
$enumconv = "Newtonsoft.Json.Converters.StringEnumConverter"
$e = New-Object $enumconv
$settings.Converters.Add($e)
Set-NewtonsoftJsonSettings $settings
$obj | ConvertTo-NewtonsoftJson
return:
{
"array": [
"un",
"deux",
"trois"
],
"enum": "Thursday",
"int": 2,
"obj": {
"enum": "Thursday",
"array": [
"un",
"deux",
"trois"
],
"int": 2,
"string": "du text"
},
"string": "du text"
}