I have a folder with dozens of json files that I output to csv via powershell. All the json files have the same format. My current powershell script is defining each file one by one by the file name, selecting various values and outputting it to the same csv.
How do I modify the powershell script so that it simply goes through each file in the folder without having to define each one by its file name specifically?
Each json's content is like this:
{
"ItemName": "A",
"ItemID": "I001",
"ItemDate": "2021-03-01",
"ItemValue": "1000",
"ItemTags": [
{
"Name": "Tag 1",
"Value": "medium"
},
{
"Name": "Tag 2",
"Value": "red"
},
{
"Name": "Tag 3",
"Value": null
},
{
"Name": "Tag 4",
"Value": "Yes"
}
]
},
{
"ItemName": "B",
"ItemID": "I002",
"ItemDate": "2021-02-01",
"ItemValue": "3000",
"ItemTags": [
{
"Name": "Tag 1",
"Value": "best"
},
{
"Name": "Tag 2",
"Value": "green"
},
{
"Name": "Tag 3",
"Value": null
},
{
"Name": "Tag 4",
"Value": "No"
}
]
Here's a sample of the powershell. How do I modify it so goes through the C:\temp folder and do the same output for each file without the -Path part needing to be specified with the file name?
$obj1 = Get-Content -Path "C:\Temp\sample1.json" | ConvertFrom-Json
$obj1 | select ItemName, ItemID, ItemDate, ItemValue, #{Name = 'Tag 4'; Expression ={($_.itemtags | where-object Name -eq "Tag 4").Value}} | Export-CSV "C:\Temp\items.csv" -NoTypeInformation -Append
$obj2 = Get-Content -Path "C:\Temp\sample2.json" | ConvertFrom-Json
$obj2 | select ItemName, ItemID, ItemDate, ItemValue, #{Name = 'Tag 4'; Expression ={($_.itemtags | where-object Name -eq "Tag 4").Value}} | Export-CSV "C:\Temp\items.csv" -NoTypeInformation -Append
$obj3 = Get-Content -Path "C:\Temp\sample3.json" | ConvertFrom-Json
$obj3 | select ItemName, ItemID, ItemDate, ItemValue, #{Name = 'Tag 4'; Expression ={($_.itemtags | where-object Name -eq "Tag 4").Value}} | Export-CSV "C:\Temp\items.csv" -NoTypeInformation -Append
Try with this:
$properties = #(
'ItemName', 'ItemID', 'ItemDate'
'ItemValue', #{
Name = 'Tag 4'
Expression = {
($_.itemtags | Where-Object Name -eq "Tag 4").Value
}
}
)
Get-ChildItem "C:\Temp" -File -Filter *.json | ForEach-Object {
Get-Content $_ -Raw | ConvertFrom-Json |
Select-Object $properties
} | Export-CSV "C:\Temp\items.csv" -NoTypeInformation
Edit
I reproduced the JSON from your example, if all JSONs look like this let me know and I'll edit my code.
PS \> $json[0]
ItemName : A
ItemID : I001
ItemDate : 2021-03-01
ItemValue : 1000
ItemTags : {#{Name=Tag 1; Value=medium}, #{Name=Tag 2; Value=red}, #{Name=Tag 3; Value=}, #{Name=Tag 4; Value=Yes}}
ItemName : B
ItemID : I002
ItemDate : 2021-02-01
ItemValue : 3000
ItemTags : {#{Name=Tag 1; Value=best}, #{Name=Tag 2; Value=green}, #{Name=Tag 3; Value=}, #{Name=Tag 4; Value=No}}
PS \> $json[0][0]
ItemName : A
ItemID : I001
ItemDate : 2021-03-01
ItemValue : 1000
ItemTags : {#{Name=Tag 1; Value=medium}, #{Name=Tag 2; Value=red}, #{Name=Tag 3; Value=}, #{Name=Tag 4; Value=Yes}}
Edit 2
$jsonFiles = Get-ChildItem "C:\temp" -File -Filter *.json
$result = foreach($json in $jsonFiles)
{
$content = Get-Content $json -Raw | ConvertFrom-Json
foreach($element in $content)
{
$tag4 = $element.itemtags.where({$_.Name -eq "Tag 4"}).Value
[pscustomobject]#{
ItemName = $element.ItemName
ItemID = $element.ItemID
ItemDate = $element.ItemDate
ItemValue = $element.ItemValue
Tag4 = $tag4
}
}
}
$result | Export-Csv "C:\temp\items.csv" -NoTypeInformation
Related
I have a JSON file with the below given sample structure. How can I convert this into CSV and get the content of CSV as the below given expected output?
{
"code":"A123",
"name":"ABC",
"parties":[
{
"businessTeam":{
"code":"B123",
"name":"Plaza"
},
"TotalNumber":"1000"
},
{
"businessTeam":{
"code":"B125",
"name":"Marina"
},
"TotalNumber":"2000"
},
{
"businessTeam":{
"code":"B130",
"name":"Marriot"
},
"TotalNumber":"2500"
}
]
}
Expected Output:
Code, Name,BusinessPartyCode,BusinessPartyName,Capacity
A123,ABC,B123,Plaza,1000
A123,ABC,B125,Marina,2000
A123,ABC,B130,Marriot,2500
I have tied with the below script and was able to extract the array values as a single delimiter concatenated values.
$deploy = Get-Content 'C:\psscripts\sample.json' -Raw | ConvertFrom-Json
$items = #()
foreach ($server in $deploy) {
foreach ($item in $server) {
$items += New-Object -TypeName PSObject -Property (
[ordered]#{
code = #($item.Code) -replace '"','#' -join '~'
businessparty = #($item.parties.businessteam.code) -join '-'
businesspartyName = #($item.parties.businessteam.name) -join '-'
Capacity = #($item.parties.businessteamtotalnumber) -join '-'
}
)
}
}
$items
-> output A123,ABC,B123-B125-B130,Plaza-Marina-Marriot,1000-2000-2500
Regards,
Sandeep
You're missing an inner loop to expand the values of businessTeam:
Get-Content 'C:\psscripts\sample.json' -Raw | ConvertFrom-Json | ForEach-Object {
foreach($item in $_.parties) {
foreach($team in $item.businessTeam) {
[pscustomobject]#{
Code = $_.code
Name = $_.name
BusinessPartyCode = $team.code
BusinessPartyName = $team.name
Capacity = $item.TotalNumber
}
}
}
} | Format-Table
Using the Json in question, the array of objects generated using this code would be:
Code Name BusinessPartyCode BusinessPartyName Capacity
---- ---- ----------------- ----------------- --------
A123 ABC B123 Plaza 1000
A123 ABC B125 Marina 2000
A123 ABC B130 Marriot 2500
I am new to Powershell so please excuse any errors I may have.
I need to write a Powershell script to convert a JSON variable file to CSV.
I need to iterate through the file and pick out all the variables for Name and Id fields.
Firstly I convert the json file to a String.But I am not sure how to iterate through the json file and pick out any of the Name field items. I then export it to a csv file.
This converts the json file to a String
$data = (Get-Content "C:\Users\QVL6\Downloads\express-ordering-web-
variables.json" | ConvertFrom-Json)
This exports it to a csv file:
$data | Select-Object -Name, Description | Export -Csv -Path .\data.csv
- NoClobber -NoTypeInformation
I need help with a for each loop that will iterate through the file and place any of the Name field values to the csv file under a heading Name.
Below is the first 3 objects of the json file:
{
"Id": "f73bdd3d-0449-036d-c2b6-b5fde280b05f",
"Name": "CIFolderPermissionGroup",
"Description": null,
"Scope": {},
"IsEditable": true,
"Prompt": null,
"Type": "String",
"IsSensitive": false
},
{
"Id": "f138f849-1647-4346-6ac4-cee4bdbd808a",
"Name": "CustomInstallFolder",
"Value": "c:\\inetpub\\wwwroot",
"Description": null,
"Scope": {},
"IsEditable": true,
"Prompt": null,
"Type": "String",
"IsSensitive": false
},
{
"Id": "99d478fb-6ef3-cc21-7997-4a9b12f3ad00",
"Name": "eimasConfiguartion",
"Value": "{\"issuelocal/":true}",
"Description": null,
"Scope": {
"Environment": [
"Environments-63"
]
}
This is my code so far:
$json = (Get-Content "C:\Users\QVL6\Downloads\express-ordering-web-
variables.json" | ConvertFrom-Json)
# helper to turn PSCustomObject into a list of key/value pairs
function Get-ObjectMembers {
[CmdletBinding()]
Param(
[Parameter(Mandatory=$True, ValueFromPipeline=$True)]
[PSCustomObject]$obj
)
$obj | Get-Member -MemberType NoteProperty | ForEach-Object {
$key = $_.Name
[PSCustomObject]#{Key = $key; Value = $obj."$key"}
}
}
#Produce a list of output objects with Name, Type, Value and Description
$ | ConvertFrom-Json | Get-ObjectMembers | foreach {
$_.Value | Get-ObjectMembers | where Key -match "Name" | foreach {
[PSCustomObject]#{
Name = $_.value.data.value | select
Type = $_.Value.data | value | select
Value = $_.Value.data | value | select
Description = $_.Value.data | value | select
}
}
}
$path = C:\Users\QVL6\
$data | Select-ObjectMambers -Property Name, Type, Value, Description |
Export -Csv -Path .\data.csv -NoClobber -NoTypeInformation
My JSON looks like this:
{
"data": [
{
"name": "engagement",
"period": "lifetime",
"values": [
{
"value": 52
}
],
"title": "Engagement",
"description": "Total number of likes and comments on the media object",
"id": "1798601712/insights/engagement/lifetime"
},
{
"name": "impressions",
"period": "lifetime",
"values": [
{
"value": 796
}
],
"title": "Impressions",
"description": "Total number of times the media object has been seen",
"id": "1798601712/insights/impressions/lifetime"
}
]
}
What I managed to achieve at this moment:
"1798601712/insights/engagement/lifetime","engagement","52"
"1798601712/insights/impressions/lifetime","impressions","796"
"1798601712/insights/reach/lifetime","reach","422"
Using the following code:
$Ident = Import-Csv -Path ".\src\Process.txt" -Header $Header |
Select-Object -Skip 2
foreach ($idka in $ident) {
$sid = $idka.id
$request_n = "https://api/"+ $sid +"/data=20190101&file=json"
foreach($dane1 in $request_n) {
Invoke-WebRequest $dane1 |
ConvertFrom-Json |
Select -ExpandProperty data |
Select id, name, #{label = "values";Expression ={$_.values.value}} |
Export-Csv $filename -NoTypeInformation -Append
}
}
I need my csv to look like this:
id engagement impressions reach
1798601712 52 796 422
1786717942 34 428 346
1787997335 29 376 281
1788199840 30 532 439
1788311007 48 1053 867
1788353947 28 609 497
1788403484 43 809 460
After expanding the data array group the nested objects by the ID you extract from the id field. For each group build a hashtable in which you map the values from each nested object to their name property. Create a custom object from the hashtable, then export the result to the output CSV.
...|
Select-Object -Expand data |
Group-Object { $_.id.Split('/')[0] } |
ForEach-Object {
$prop = #{
'id' = $_.Name
}
$_.Group | ForEach-Object {
$prop[$_.name] = $_.values.value
}
New-Object -Type PSObject -Property $prop
} |
Select-Object id, engagement, impressions, reach |
Export-Csv $filename -NoType -Append
Note that with PowerShell v3 or newer you can use an ordered hashtable and the [PSCustomObject] type accelerator instead of New-Object, which would allow you to omit the last Select-Object (whose sole purpose is getting the output fields in the desired order).
I have a JSON file that looks like this:
{
"id": 10011,
"title": "Test procedure",
"slug": "slug",
"url": "http://test.test",
"email": "test#test.com",
"link": "http://test.er",
"subject": "testing",
"level": 1,
"disciplines": [
"discipline_a",
"discipline_b",
"discipline_c"
],
"areas": [
"area_a",
"area_b"
]
},
I was trying to use the following command to convert that into the CSV file:
(Get-Content "PATH_TO\test.json" -Raw | ConvertFrom-Json)| Convertto-CSV -NoTypeInformation | Set-Content "PATH_TO\test.csv"
However, for disciplines and areas I am getting System.Object[] in the resulting CSV file.
Is there a way to put all those nested values as a separate columns in CSV file like area_1, area_2 etc. And the same for disciplines.
2017-11-20, Completely rewrote function to improve performance and add features as -ArrayBase and support for PSStandardMembers and grouped objects.
Flatten-Object
Recursively flattens objects containing arrays, hash tables and (custom) objects. All added properties of the supplied objects will be aligned with the rest of the objects.
Requires PowerShell version 2 or higher.
Cmdlet
Function Flatten-Object { # Version 00.02.12, by iRon
[CmdletBinding()]Param (
[Parameter(ValueFromPipeLine = $True)][Object[]]$Objects,
[String]$Separator = ".", [ValidateSet("", 0, 1)]$Base = 1, [Int]$Depth = 5, [Int]$Uncut = 1,
[String[]]$ToString = ([String], [DateTime], [TimeSpan]), [String[]]$Path = #()
)
$PipeLine = $Input | ForEach {$_}; If ($PipeLine) {$Objects = $PipeLine}
If (#(Get-PSCallStack)[1].Command -eq $MyInvocation.MyCommand.Name -or #(Get-PSCallStack)[1].Command -eq "<position>") {
$Object = #($Objects)[0]; $Iterate = New-Object System.Collections.Specialized.OrderedDictionary
If ($ToString | Where {$Object -is $_}) {$Object = $Object.ToString()}
ElseIf ($Depth) {$Depth--
If ($Object.GetEnumerator.OverloadDefinitions -match "[\W]IDictionaryEnumerator[\W]") {
$Iterate = $Object
} ElseIf ($Object.GetEnumerator.OverloadDefinitions -match "[\W]IEnumerator[\W]") {
$Object.GetEnumerator() | ForEach -Begin {$i = $Base} {$Iterate.($i) = $_; $i += 1}
} Else {
$Names = If ($Uncut) {$Uncut--} Else {$Object.PSStandardMembers.DefaultDisplayPropertySet.ReferencedPropertyNames}
If (!$Names) {$Names = $Object.PSObject.Properties | Where {$_.IsGettable} | Select -Expand Name}
If ($Names) {$Names | ForEach {$Iterate.$_ = $Object.$_}}
}
}
If (#($Iterate.Keys).Count) {
$Iterate.Keys | ForEach {
Flatten-Object #(,$Iterate.$_) $Separator $Base $Depth $Uncut $ToString ($Path + $_)
}
} Else {$Property.(($Path | Where {$_}) -Join $Separator) = $Object}
} ElseIf ($Objects -ne $Null) {
#($Objects) | ForEach -Begin {$Output = #(); $Names = #()} {
New-Variable -Force -Option AllScope -Name Property -Value (New-Object System.Collections.Specialized.OrderedDictionary)
Flatten-Object #(,$_) $Separator $Base $Depth $Uncut $ToString $Path
$Output += New-Object PSObject -Property $Property
$Names += $Output[-1].PSObject.Properties | Select -Expand Name
}
$Output | Select ([String[]]($Names | Select -Unique))
}
}; Set-Alias Flatten Flatten-Object
Syntax
<Object[]> Flatten-Object [-Separator <String>] [-Base "" | 0 | 1] [-Depth <Int>] [-Uncut<Int>] [ToString <Type[]>]
or:
Flatten-Object <Object[]> [[-Separator] <String>] [[-Base] "" | 0 | 1] [[-Depth] <Int>] [[-Uncut] <Int>] [[ToString] <Type[]>]
Parameters
-Object[] <Object[]>
The object (or objects) to be flatten.
-Separator <String> (Default: .)
The separator used between the recursive property names. .
-Depth <Int> (Default: 5)
The maximal depth of flattening a recursive property. Any negative value will result in an unlimited depth and could cause a infinitive loop.
-Uncut <Int> (Default: 1)
The number of object iterations that will left uncut further object properties will be limited to just the DefaultDisplayPropertySet. Any negative value will reveal all properties of all objects.
-Base "" | 0 | 1 (Default: 1)
The first index name of an embedded array:
1, arrays will be 1 based: <Parent>.1, <Parent>.2, <Parent>.3, ...
0, arrays will be 0 based: <Parent>.0, <Parent>.1, <Parent>.2, ...
"", the first item in an array will be unnamed and than followed with 1: <Parent>, <Parent>.1, <Parent>.2, ...
-ToString <Type[]= [String], [DateTime], [TimeSpan]>
A list of value types (default [String], [DateTime], [TimeSpan]) that will be converted to string rather the further flattened. E.g. a [DateTime] could be flattened with additional properties like Date, Day, DayOfWeek etc. but will be converted to a single (String) property instead.
Note:
The parameter -Path is for internal use but could but used to prefix property names.
Examples
Answering the specific question:
(Get-Content "PATH_TO\test.json" -Raw | ConvertFrom-Json) | Flatten-Object | Convertto-CSV -NoTypeInformation | Set-Content "PATH_TO\test.csv"
Result:
{
"url": "http://test.test",
"slug": "slug",
"id": 10011,
"link": "http://test.er",
"level": 1,
"areas.2": "area_b",
"areas.1": "area_a",
"disciplines.3": "discipline_c",
"disciplines.2": "discipline_b",
"disciplines.1": "discipline_a",
"subject": "testing",
"title": "Test procedure",
"email": "test#test.com"
}
Stress testing a more complex custom object:
New-Object PSObject #{
String = [String]"Text"
Char = [Char]65
Byte = [Byte]66
Int = [Int]67
Long = [Long]68
Null = $Null
Booleans = $False, $True
Decimal = [Decimal]69
Single = [Single]70
Double = [Double]71
Array = #("One", "Two", #("Three", "Four"), "Five")
HashTable = #{city="New York"; currency="Dollar"; postalCode=10021; Etc = #("Three", "Four", "Five")}
Object = New-Object PSObject -Property #{Name = "One"; Value = 1; Text = #("First", "1st")}
} | Flatten
Result:
Double : 71
Decimal : 69
Long : 68
Array.1 : One
Array.2 : Two
Array.3.1 : Three
Array.3.2 : Four
Array.4 : Five
Object.Name : One
Object.Value : 1
Object.Text.1 : First
Object.Text.2 : 1st
Int : 67
Byte : 66
HashTable.postalCode : 10021
HashTable.currency : Dollar
HashTable.Etc.1 : Three
HashTable.Etc.2 : Four
HashTable.Etc.3 : Five
HashTable.city : New York
Booleans.1 : False
Booleans.2 : True
String : Text
Char : A
Single : 70
Null :
Flatting grouped objects:
$csv | Group Name | Flatten | Format-Table # https://stackoverflow.com/a/47409634/1701026
Flatting common objects:
(Get-Process)[0] | Flatten-Object
Or a list (array) of objects:
Get-Service | Flatten-Object -Depth 3 | Export-CSV Service.csv
Note that a command as below takes hours to compute:
Get-Process | Flatten-Object | Export-CSV Process.csv
Why? because it results in a table with a few hundred rows and several thousand columns. So if you if would like to use this for flatting process, you beter limit the number of rows (using the Where-Object cmdlet) or the number of columns (using the Select-Object cmdlet).
For the latest Flatten-Object version, see: https://powersnippets.com/flatten-object/
The CSV conversion/export cmdlets have no way of "flattening" an object, and I may be missing something, but I know of no way to do this with a built-in cmdlet or feature.
If you can guarantee that disciplines and areas will always have the same number of elements, you can trivialize it by using Select-Object with derived properties to do this:
$properties=#('id','title','slug','url','email','link','subject','level',
#{Name='discipline_1';Expression={$_.disciplines[0]}}
#{Name='discipline_2';Expression={$_.disciplines[1]}}
#{Name='discipline_3';Expression={$_.disciplines[2]}}
#{Name='area_1';Expression={$_.areas[0]}}
#{Name='area_2';Expression={$_.areas[1]}}
)
(Get-Content 'PATH_TO\test.json' -Raw | ConvertFrom-Json)| Select-Object -Property $properties | Export-CSV -NoTypeInformation -Path 'PATH_TO\test.csv'
However, I am assuming that disciplines and areas will be variable length for each record. In that case, you will have to loop over the input and pull the highest count value for both disciplines and areas, then build the properties array dynamically:
$inputData = Get-Content 'PATH_TO\test.json' -Raw | ConvertFrom-Json
$counts = $inputData | Select-Object -Property #{Name='disciplineCount';Expression={$_.disciplines.Count}},#{Name='areaCount';Expression={$_.areas.count}}
$maxDisciplines = $counts | Measure-Object -Maximum -Property disciplineCount | Select-Object -ExpandProperty Maximum
$maxAreas = $counts | Measure-Object -Maximum -Property areaCount | Select-Object -ExpandProperty Maximum
$properties=#('id','title','slug','url','email','link','subject','level')
1..$maxDisciplines | % {
$properties += #{Name="discipline_$_";Expression=[scriptblock]::create("`$_.disciplines[$($_ - 1)]")}
}
1..$maxAreas | % {
$properties += #{Name="area_$_";Expression=[scriptblock]::create("`$_.areas[$($_ - 1)]")}
}
$inputData | Select-Object -Property $properties | Export-CSV -NoTypeInformation -Path 'PATH_TO\test.csv'
This code hasn't been fully tested, so it may need some tweaking to work 100%, but I believe the ideas are solid =)
I am extracting a JSON from Facebook, this is what it looks like:
pressions_by_paid_non_paid_unique/day
{
"data": [
{
"name": "page_impressions_by_paid_non_paid_unique",
"period": "day",
"values": [
{
"value": {
"total": 549215,
"unpaid": 549215,
"paid": 0
},
"end_time": "2017-06-02T07:00:00+0000"
},
What I would like is to create a CSV that would either looks like this:
total,unpaid,paid,endtime
549215,549215,0,2017-06-02T07:00:00+0000
or
value,num,end_time
total,549215,2017-06-02T07:00:00+0000
unpaid,549215,2017-06-02T07:00:00+0000
paid,0,2017-06-02T07:00:00+0000
What I came up with is:
$file = "D:\BI_LP\001-Solutions_Sources\script\Powershell\curl\facebook21.json"
Get-Content $file -Raw |
ConvertFrom-Json |
Select -Expand data |
Select -Expand values | % {
$end_time = $_.end_time
$value = $_.value
$_.value | select #{n='end_time';e={$end_time}}, #{n='value';e={$value}}
}
Which gives me:
end_time value
-------- -----
2017-05-21T07:00:00+0000 #{total=608837; unpaid=608837; paid=0}
2017-05-22T07:00:00+0000 #{total=682090; unpaid=682090; paid=0}
2017-05-23T07:00:00+0000 #{total=885274; unpaid=885274; paid=0}
2017-05-24T07:00:00+0000 #{total=810845; unpaid=810845; paid=0}
2017-05-25T07:00:00+0000 #{total=755453; unpaid=755453; paid=0}
2017-05-26T07:00:00+0000 #{total=629096; unpaid=629096; paid=0}
Does anyone have any suggestions?
I think you are almost there. You could try creating the output object a bit like this in the foreach loop:
[pscustomobject]#{
total = $_.value.total
unpaid = $_.value.unpaid
paid = $_.value.paid
end_time = $_.end_time
}
Based on your code:
#"
{
"data": [
{
"name": "page_impressions_by_paid_non_paid_unique",
"period": "day",
"values": [
{
"value": {
"total": 549215,
"unpaid": 549215,
"paid": 0
},
"end_time": "2017-06-02T07:00:00+0000"
}
]
}
]
}
"# | ConvertFrom-Json | Select -Expand data | Select -Expand values | % {
$end_time = $_.end_time
$value = $_.value
$_.value | select #{n='endtime'; e={$value.total}}, #{n='unpaid'; e={$value.unpaid}}, #{n='paid'; e={$value.paid}}, #{n='end_time';e={$end_time}}
} | ft -autosize
Gives the following output:
endtime unpaid paid end_time
------- ------ ---- --------
549215 549215 0 2017-06-02T07:00:00+0000
And to export to csv, change the ft -autosize to Export-Csv -Path c:\...
FYI: Your json sample is missing a lot of closing brackets.
Thanks Charlie, you got me in the right direction :)
$file = "D:\BI_LP\001-Solutions_Sources\script\Powershell\curl\facebook21.json"
Get-Content $file -Raw |
ConvertFrom-Json |
Select -Expand data |
Select -Expand values | % {
$end_time = $_.end_time
$total = $_.value.total
$unpaid = $_.value.unpaid
$paid = $_.value.paid
$_.value | select #{n='end_time';e={$end_time}}, #{n='total';e={$total}},#{n='unpaid';e={$unpaid}},#{n='paid';e={$paid}}
}