Powershell - Json to csv - values in separate rows - json

I am trying to get the data into a csv from an InvokeWeb request. Currently there are only 2 different values with the individual values in it, but later also times more. It makes the headers of the table, and the correct number of columns, but throws the 2 different values into a cell in both rows. So how can I prevent it from looking like this? I have already read about AddMember or +=, but did not find the right way.
$valstring = iwr -Proxy http://my-proxy -ProxyUseDefaultCredentials -Method GET -Uri https://my-URL -Headers #{'ContentTyp' = 'application/json';"X-Api-Key" = "my-API-Key"} -UseBasicParsing | Select-Object Content | foreach {$_.Content}
$pathToOutputFile = "H:\Wichtiges\json2csvexport.csv"
$deckung = $valstring | ConvertFrom-Json | ForEach-Object {
foreach ($uuid in $_ ) {
[PsCustomObject] #{
name = $_.name | Out-String
version = $_.version | Out-String
active = $_.active | Out-String
}
}
}
$deckung | Export-Csv $pathToOutputFile -NoTypeInformation -UseCulture
The Json-Data looks like that
[
{
"name": "App1",
"version": "1.1.1",
"uuid": "a74a1969-f57d-437b-943f-d4c3dd3bc4bb",
"active": true,
"metrics": {
"high": 617,
"medium": 1250,
"low": 103,
}
},
{
"name": "App2",
"version": "1.3.1",
"uuid": "59eda14a-56a6-4dc6-8238-dd168bd0df3f",
"active": true,
"metrics": {
"high": 6,
"medium": 13,
"low": 0,
}
}
]
But it always comes out like that (second and third together and fourth and fifth row together):
|name | version | active |
|------|-----------|---------|
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
-----------------------------
How can I change it to get one per row that it looks like:
|name | version | active |
|------|-----------|---------|
|App1 | 1.1.1 | true |
|App2 | 1.3.1 | true |
-----------------------------
Thx
Kind regards
Edit (Solution):
That solution works for me, because I can put things from mectrics into the csv and give the Heading an other name:
Select-Object #{Name='Name';Expression={$_.name}},
#{Name='Version';Expression={$_.version}},
#{Name='Active';Expression={$_.Active}},
#{Name='Risk-High';Expression={$_.metrics.high}}
Edit (Another question):
How can I prevent numbers in the csv from being converted to dates?
Thx

Try the following:
$pathToOutputFile = "H:\Wichtiges\json2csvexport.csv"
# Note:
# * the use of Invoke-*RestMethod*, which has ConvertFrom-Json *built in*.
# * the (...) around it, to ensure that the array is *enumerated*.
(Invoke-RestMehod -Proxy http://my-proxy -ProxyUseDefaultCredentials -Method GET -Uri https://my-URL -Headers #{'ContentType' = 'application/json';"X-Api-Key" = "my-API-Key"}) |
Select-Object name, version, active |
Export-Csv $pathToOutputFile -NoTypeInformation -UseCulture
As for what you tried:
Your foreach ($uuid in $_ ) loop mistakenly used $_ - the array to loop over - instead of $uuid - the iteration variable containing the element at hand.
The simpler alternative, to explicit enumeration is to wrap a call to ConvertFrom-Json / Invoke-RestMethod in (...) in order to automatically force enumeration of an array being returned.
Note that this is no longer necessary in PowerShell (Core) 7+, where arrays are enumerated by default (and you need -NoEnumerate in order to send an array as a whole through the pipeline, which is the invariable default behavior in Windows PowerShell).
For simple stringification of a value, Out-String is the wrong tool, primarily because it - unexpectedly - adds a trailing newline to the output - see GitHub issue #14444.
Use a [string] cast instead; however, given that Export-Csv implicitly performs such a stringification for you, there's no need to do it explicitly, allowing you to use a simple Select-Object to pick the properties of interest, as shown above.

Related

How do I extract a JSON value if another value is higher than zero?

My current code:
Powershell -Noprofile "(Get-Content 'allacts.txt' | ConvertFrom-Json).activityid | Out-File -FilePath ids%filenum%.txt"
This queries the JSON file (allacts.txt) and extracts a single value, activityid.
However, I don't want all the activityids, just the ones where the value attendedpeoplecount is higher than 0. How can I work that if logic into this command?
Sample:
[
{
"activityid": 610228,
"attendedpeoplecount": 0
},
{
"activityid": 568168,
"attendedpeoplecount": 36
}
]
Current command on the above json would return
610228
568168
The desired output would be
568168
Using the Where-Object Cmdlet should get you what you need.
Powershell -Noprofile "((Get-Content 'allacts.txt' | ConvertFrom-Json) | Where-Object { $_.attendedpeoplecount -gt 0 }).activityid | Out-File -FilePath ids%filenum%.txt"
Update: Wrapped Get-Content and ConvertFrom-Json in parenthesis

Displaying Data to a certain Depth When Using Convertfrom-json

Currently i'm attempting to create a script to convert my json files to csv files. At the moment I'm receiving most of the data, but the issue i'm having is expanding out the "glyphs" field to add that data into the csv as well.
Below is an example of the json currently. This is similar to what's already out there, but my problem is with the -depth that the convertfrom-json pulls. It's only going to two levels and I need it to go to four.
{
"nodes": [
{
"id": 23,
"type": "Group",
"label": "DOMAIN ADMINS#.COM",
"glyphs": {
"0": {
"position": "top-right",
"font": "\"Font Awesome 5 Free\"",
"content": "",
"fillColor": "black",
"fontScale": 1.5,
"fontStyle": "900"
}
},
"folded": {
"nodes": {},
"edges": {}
},
$user1 = $env:USERNAME
Get-Content C:\Users\$user1\Documents\json_to_convert.json |
convertfrom-json | select -ExpandProperty nodes |
Export-CSV C:\Users\$user1\Documents\jsonTest_$((Get-Date).ToString('MM-dd-yy')).csv -NoTypeInformation
So i'm just hoping to also input the "position", "font", "content", "fillColor", "fontScale", and "fontstyle" from "glyphs into my CSV file as well. Thos are the only fields i'm having trouble with.
If you want to place the nested objects into separate CSV fields then you do have to think about how to represent that in a csv file. The below can be used to create custom properties that are calculated using an expression.
Get-Content C:\Users\$user1\Documents\json_to_convert.json | ConvertFrom-Json | Select -ExpandProperty nodes | `
Select -Property id,type,label, `
#{Name='Glyphs-position';Expression={$_.glyphs.0 | Select -ExpandProperty position}},`
#{Name='Glyphs-font';Expression={$_.glyphs.0 | Select -ExpandProperty font}},`
#{Name='Glyphs-content';Expression={$_.glyphs.0 | Select -ExpandProperty content}},`
#{Name='Glyphs-fillColor';Expression={$_.glyphs.0 | Select -ExpandProperty fillColor}},`
#{Name='Glyphs-fontScale';Expression={$_.glyphs.0 | Select -ExpandProperty fontScale}},`
#{Name='Glyphs-fontStyle';Expression={$_.glyphs.0 | Select -ExpandProperty fontStyle}}`
| ConvertTo-Csv
This could probably be written more efficiently but it demonstrates how to get the values you are after. The resultant PowerShell object produced is
id : 23
type : Group
label : DOMAIN ADMINS#.COM
Glyphs-position : top-right
Glyphs-font : "Font Awesome 5 Free"
Glyphs-content : 
Glyphs-fillColor : black
Glyphs-fontScale : 1.5
Glyphs-fontStyle : 900
and once converted to csv
"id","type","label","Glyphs-position","Glyphs-font","Glyphs-content","Glyphs-fillColor","Glyphs-fontScale","Glyphs-fontStyle","folded"
"23","Group","DOMAIN ADMINS#.COM","top-right","""Font Awesome 5 Free""","","black","1.5","900"

Powershell - Retain Complex objects with ConvertTo-Json

Powershell command-let ConvertTo-Json has the following limitations
1) It returns Enum values as Integers instead of their text
2) It doesn't return the date in a readable format
For point #1 see below, Status and VerificationMethod Properties
PS C:\Windows\system32> Get-Msoldomain | ConvertTo-Json
{
"ExtensionData": {
},
"Authentication": 0,
"Capabilities": 5,
"IsDefault": true,
"IsInitial": true,
"Name": "myemail.onmicrosoft.com",
"RootDomain": null,
"Status": 1,
"VerificationMethod": 1
}
To handle this, I changed my command as below
PS C:\Windows\system32> Get-Msoldomain | ConvertTo-Csv | ConvertFrom-Csv | ConvertTo-Json
{
"ExtensionData": "System.Runtime.Serialization.ExtensionDataObject",
"Authentication": "Managed",
"Capabilities": "Email, OfficeCommunicationsOnline",
"IsDefault": "True",
"IsInitial": "True",
"Name": "curtisjmspartnerhotmail.onmicrosoft.com",
"RootDomain": "",
"Status": "Verified",
"VerificationMethod": "DnsRecord"
}
Now you see, that the enums are being returned with their text values above (Status and VerificationMethod) instead of their integer values.
However, There are a few limitations with this approach:
1) ConvertTo-Csv doesn't retain the Arrays or Complex Objects, and
outputs them as their Class Names (Watch the ExtensionData Properties
in both the outputs). In the second output, we tend to lose the data,
and just get the className
System.Runtime.Serialization.ExtensionDataObject as a string
2) Both ConvertTo-Csv and ConvertFrom-Csv are not the script-level
commandlets, but they are command-level commandlets, which means that
we can't use them at the end of the script , but they will have to be
used with the individual commands like I am doing above. WHEREAS,
ConvertTo-Json need not be applied at the commmandLevel, but just
applied once for the script output.
My question is:
1) How do I still use the convertTo-Json, so that all my enum properties are returned with their texts and not integers, and ALSO the Complex Objects or Arrays are not lost? In the approach I have used, the complex objects are getting lost
2) Also, it should be generic enough so that It can be applied at the end of the script, and not at the command level
ConvertTo-Json and ConvertTo-Csv are both forms of serializing objects in some sort of text representation and both are useful in different use cases.
ConvertTo-Csv is perhaps best used for 2-dimensional data that can be expressed in a table such as a spreadsheet. Hence it is awkward to try to convert "complex" objects, i.e. those with properties that contain other structured data, into a simple table. In such cases PowerShell represents such data as the full name of the data type.
ConvertTo-Json is capable of serializing more complicated objects, since the format allows for nested arrays/data structures, e.g. the ExtensionData property in your example. Note that you may need to use the -Depth parameter to ensure that deeply nested data is serialized correctly.
So the problem really comes down to how the ConvertTo-Json cmdlet serializes enums, which can be demonstrated with:
[PS]> (Get-Date).DayOfWeek
Tuesday
[PS]> (Get-Date).DayOfWeek.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True DayOfWeek System.Enum
[PS]> Get-Date | Select DayOfWeek | ConvertTo-Json
{
"DayOfWeek": 2
}
So before you convert to JSON you need to ensure that the DayOfWeek property (in this example) or Status and VerificationMethod properties (from your example) are converted to their string equivalents first.
You can do this using an expression with Select-Object to convert the data as it passes down the pipe. Note that you do need to include all the properties that you want included in the final JSON:
[PS]> Get-Date |
Select DateTime,#{Label="DayOfWeek";Expression={$_.DayOfWeek.ToString()}} |
ConvertTo-Json
{
"DateTime": "13 June 2017 10:33:51",
"DayOfWeek": "Tuesday"
}
So in your case you'd need something like this:
[PS]> Get-Msoldomain |
Select-Object ExtensionData,IsDefault,IsInitial,Name,RootDomain `
,{Label="Authentication";Expression={$_.Authentication.ToString()}} `
,{Label="Capabilities";Expression={$_.Capabilities.ToString()}} `
,{Label="Status";Expression={$_.Status.ToString()}} `
,{Label="VerificationMethod";Expression={$_.VerificationMethod.ToString()}} |
ConvertTo-Json
#puneet, following your comment on my other answer, here is an example of how you might build up a new object, based on an existing one, with the Enum types converted to strings.
The idea is to create a new "empty" object, then loop through all the properties of the original object and add them to the new one, but if any of the original properties are Enums, then those are converted to strings.
$data = [PSCustomObject]#{}
(Get-Date).PSObject.Properties | Select Name,Value | Foreach-Object {
if($_.Value.GetType().BaseType.FullName -eq "System.Enum"){
$data | Add-Member -MemberType NoteProperty -Name $_.Name -Value $_.Value.ToString()
}
else {
$data | Add-Member -MemberType NoteProperty -Name $_.Name -Value $_.Value
}
}
$data | ConvertTo-Json
You may want to finesse this a little for your own application, but hopefully the idea behind it is clear. Definitely check to see that all the properties are being treated correctly in the JSON output.
to keep enum,array and date when converting psObject to json, you can use newtonsoft. a sample here https://github.com/chavers/powershell-newtonsoft using Nerdy Mishka powershell module.
$obj = New-Object pscustomobject -Property #{Enum = (Get-DAte).DayOfWeek; int = 2; string = "du text"; array = #("un", "deux", "trois"); obj= #{enum = (Get-DAte).DayOfWeek; int = 2; string = "du text"; array = #("un", "deux", "trois")}}
Import-Module Fmg-PrettyJson
$settings = Get-NewtonsoftJsonSettings
$enumconv = "Newtonsoft.Json.Converters.StringEnumConverter"
$e = New-Object $enumconv
$settings.Converters.Add($e)
Set-NewtonsoftJsonSettings $settings
$obj | ConvertTo-NewtonsoftJson
return:
{
"array": [
"un",
"deux",
"trois"
],
"enum": "Thursday",
"int": 2,
"obj": {
"enum": "Thursday",
"array": [
"un",
"deux",
"trois"
],
"int": 2,
"string": "du text"
},
"string": "du text"
}

finding the character at 'wildcard' position

I'm trying to parse a JSON string which has been flattened from a relational dataset from the looks of it... Into a CSV in order to import into an Oracle database. I have name/value pairs similar to this (simplified) example
Device.1.Service.1.Channel.1.someProperty : 1,
Device.1.Service.1.Channel.1.someOtherProperty : "billy",
Device.1.Service.1.Channel.2.someProperty : 8,
Device.1.Service.1.Channel.2.someOtherProperty : "frank",
Device.1.Service.1.Channel.3.someProperty : 12,
Device.1.Service.1.Channel.3.someOtherProperty : "sam",
Device.1.Service.2.Channel.1.someProperty : 3,
Device.1.Service.2.Channel.1.someOtherProperty : "john",
EDIT: A .json file with a similar structure is produced per day per device. So when consolidated (Get-Content .\*.json -Raw) I see the same property names occur many times in console.
As part of the conversion, I'd like parts of the property name to become fields in the database. This will enable us to visualise the data better using dynamic slider filters etc. later.
| Device | Service | Channel | someOtherProperty |
| 1 | 1 | 1 | billy |
| 1 | 1 | 2 | frank |
| 1 | 1 | 3 | sam |
Right now, I'm using the cmdlet ConvertFrom-Json. I'm then selecting the fields (out of nearly 2000 possible fields) using wildcards. The number of channels for example is dynamic.
Get-Content .\*.json -Raw | ConvertFrom-Json |
Select Device.1.Service.1.Channel.?.someOtherProperty
Which returns a PSCustomObject. I'd like to figure out the channel number and use that as a derived field. Pseudo example:
Select #{n="Channel";e={$_.getCharAtWildcard()}},
$_.theValueofTheCurrentObject()
If I pipe the data selected (using wildcard) to Get-Member, the PSCostomObject contains method names as well as my field names (but not the values).
I'm confused about how I access the field name (since 'Name' gives blank rows), then extract the channel number (the character at wildcard position), and then the value to structure my output correctly.
Any pointers? Googled and have a lynda.com sub, but can't seem to find a solution to this specific problem - probably because I'm not using the correct terms?
## ANSGAR's SOLUTION - WORKS FOR SINGLE FILE ##
$dataDir = "C:\ps_json2csv\dummydata"
CD $dataDir
$dict = Get-Content .\*.json -Raw | ConvertFrom-Json | Select -Expand data
$p = 'DvbId'
$re = "frontend\.(\d+)\.logicalchannel\.(\d+)\.service\.(\d+)\..*?\.$p"
## modified regex to match my data, example string: FrontEnd.2.LogicalChannel.3.Service.1.stat.DvbId
$fullset = $dict.PSObject.Properties | Where-Object {
$_.Name -match $re
} | ForEach-Object {
$prop = [ordered]#{
FrontEnd = $matches[1]
LogicalChannel = $matches[2]
Service = $matches[3]
$p = $_.Value
}
New-Object -Type PSObject -Property $prop
}
## inspect $dict - its populated
## inspect $fullset - its empty! :(
Data of which is 2 files contained in C:\ps_json2csv\dummydata :
File1.json
{
"data": {
"Device.1.Service.1.ChannelInfo.Channel.1.Stats.someProperty" : "1",
"Device.1.Service.1.ChannelInfo.Channel.2.Stats.someProperty" : "8",
"Device.1.Service.1.ChannelInfo.Channel.3.Stats.someProperty" : "12",
"FrontEnd.2.LogicalChannel.3.Service.1.stat.DvbId" : "john",
"FrontEnd.2.LogicalChannel.3.Service.2.stat.DvbId" : "billy",
"FrontEnd.2.LogicalChannel.3.Service.3.stat.DvbId" : "frank",
"FrontEnd.2.LogicalChannel.4.Service.1.stat.DvbId" : "sam",
"Device.1.Service.2.ChannelInfo.Channel.1.Stats.someProperty" : "3",
"Some.value.im.not.intersted.in.just.yet": "Sat Jan 1 00:00:00 GMT 0001",
"foo.bar" : "0",
"random.stuff" : "hi there"
}
}
File2.json
{
"data": {
"Device.1.Service.1.ChannelInfo.Channel.1.Stats.someProperty" : "0",
"Device.1.Service.1.ChannelInfo.Channel.2.Stats.someProperty" : "7",
"Device.1.Service.1.ChannelInfo.Channel.3.Stats.someProperty" : "6",
"FrontEnd.2.LogicalChannel.3.Service.1.stat.DvbId" : "john",
"FrontEnd.2.LogicalChannel.3.Service.2.stat.DvbId" : "billy",
"FrontEnd.2.LogicalChannel.3.Service.3.stat.DvbId" : "frank",
"FrontEnd.2.LogicalChannel.4.Service.1.stat.DvbId" : "sam",
"Device.1.Service.2.ChannelInfo.Channel.1.Stats.someProperty" : "4",
"Some.value.im.not.intersted.in.just.yet": "Sun Jan 2 00:00:00 GMT 0001",
"foo.bar" : "0",
"random.stuff" : "hi there"
}
}
I would probably use a regular expression match to filter and extract your data:
$p = 'someOtherProperty'
$re = "device\.(\d+)\.service\.(\d+)\..*?\.channel\.(\d+)\..*?\.$p"
$fullset = $dict.PSObject.Properties | Where-Object {
$_.Name -match $re
} | ForEach-Object {
$prop = [ordered]#{
Device = $matches[1]
Service = $matches[2]
Channel = $matches[3]
$p = $_.Value
}
New-Object -Type PSObject -Property $prop
}
I'd rather convert a string of this syntax into a PSObject with the entire set of your properties. Like this:
$dict=Get-Content .\*.json -Raw | ConvertFrom-Json
$fullset=$dict.psobject.properties | % {
$parse=$_.name.split('.')
$parse+=,$_.value # since JSON values might be non-plain, we need to add the value as single object
$obj=new-object psobject
for ($i=0;$i -lt $parse.length;$i+=2) { # name-value
$v1=$parse[$i]
$v2=$parse[1+$i]
$obj | add-member -type noteproperty -name $v1 -value $v2
}
$obj
}
Then you parse $fullset like you would with a normal list with where {$_.device -eq '1' -and $_.someOtherProperty -ne $null} etc.

Expand Multi-level Powershell nested hash tables from JSON api call

I am having an issue with a JSON API call using Powershell, that returns a multi-level nested array. Using the standard convertfrom-json & convertto-json doesn't give me the full results. I just get a ...
I have done a lot of research on this and found the following link on stack for expanding the nested hash tables, and this is about the best format that I have gotten. For reference, I am trying to insert employee records into SQL Server. I have a JSON function for doing so, and need to get the JSON response in the expanded format. There are some security aspects, that won't allow me to post the actual detail so I am trying to provide an example of some issues that I am having. Keep in mind the response is for 1 employee record out of hundreds overall, and in SQL will be one table with a row per employee.
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | convertfrom-json
$workers = $json1.workers | select -property *
$workers | format-table Name, #{n='Value'; e={
if ($_.Value -is [Hashtable]) {
$ht = $_.Value
$a = $ht.keys | sort | % { '{0}={1}' -f $_, $ht[$_] }
'{{{0}}}' -f ($a -join ', ')
} else {
$_.Value
}
}}
$workers
I have also tried:
$workers | convertto-json
and
$workers = json1.workers | foreach-object {$_}
The data returned comes back like this:
associateOID : XXXXXXXXXXXXXX
workerID : #{idValue=XXXXXXX}
person : #{governmentIDs=System.Object[]; legalName=; birthDate=0000-00-00; legalAddress=; genderCode=; maritalStatusCode=; socialInsurancePrograms=System.Object[]; tobaccoUserIndicator=False; raceCode=; customFieldGroup=}
workerDates : #{originalHireDate=2015-03-09; terminationDate=2016-03-18}
workerStatus : #{statusCode=}
businessCommunication : #{emails=System.object[]}
workAssignments : {#{itemID=xxxxxx; occupationalclassification=System.Object[]; etc}}
I need it to come back with all of the columns on the left side, utilizing the "AssociateOID" as the key identifier for the individual. I have previously gotten the JSON response to come back completely expanded using this format but it wasn't working with the import into SQL Server or looking very nice like the Google Postman response:
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | format-list -property *
You mean you want to get a JSON file that all objects/properties are expanded, right? Use the -Depth option of ConvertTo-Json to write a JSON file properly. Default value of -Depth option is 2.
$workers | ConvertTo-Json -Depth 100
Specify the appropriate depth according to your JSON files.