PowerShell flattening nested JSON and convert it to CSV - json

I have a JSON file below. I am trying to flatten and convert it to CSV.
{
"tempid": "template_86CE6E3BE3AD4EAB95727BCBFAD6A83C",
"auid": "audit_00006F5D7A114CE59AD572E3E878E726",
"created_at": "2017-01-12T08:54:48.835Z",
"Dateat": "2019-04-26T14:24:09.496Z",
"Datefrom": {
"score": 64,
"duration": 1754,
"space": {
"device_id": "88888888888888",
"owner": "John Paul"
},
"header_items": [
{
"item_id": "357085FF-B66A-4C28-B9D",
"children": "66f7893245d45-ea77-0020"
},
{
"parent_id": "357949D",
"item_id": "f3789245d40-ea7o89797a66",
"label": "Audit Title",
"options": "#{is_mandatory=False}",
"responses": "#{text=}"
}
],
"items": [
{
"parent_id": "81C1FFE",
"item_id": "B9CD2607897898-427898",
"label": "TURN LEFT.",
"type": "category"
},
{
"parent_id": "456487k78978578",
"item_id": "687fgfgfd",
"label": "ANY RUBBISH?"
}
]
}
I have tried the code below and I get errors "header_item" cannot be found. I would like it flatten into csv files.
Get-Content C:\can\Test\XY.json -Raw |
ConvertFrom-Json |
Select -Expand header_items |
Select -Expand items |
Export-Csv C:\can\Test\XY.csv -NoTypeInformation

First thing provided json is malformed (missing brackets) so i assume its:
{
"tempid": "template_86CE6E3BE3AD4EAB95727BCBFAD6A83C",
"auid": "audit_00006F5D7A114CE59AD572E3E878E726",
"created_at": "2017-01-12T08:54:48.835Z",
"Dateat": "2019-04-26T14:24:09.496Z",
"Datefrom": {},
"score": 64,
"duration": 1754,
"space": {
"device_id": "88888888888888",
"owner": "John Paul"
},
"header_items": [
{
"item_id": "357085FF-B66A-4C28-B9D",
"children": "66f7893245d45-ea77-0020"
},
{
"parent_id": "357949D",
"item_id": "f3789245d40-ea7o89797a66",
"label": "Audit Title",
"options": "#{is_mandatory=False}",
"responses": "#{text=}"
}
],
"items": [
{
"parent_id": "81C1FFE",
"item_id": "B9CD2607897898-427898",
"label": "TURN LEFT.",
"type": "category"
},
{
"parent_id": "456487k78978578",
"item_id": "687fgfgfd",
"label": "ANY RUBBISH?"
}
]
}
Second thing when using Export-Csv PowerShell uses the first object to determine the CSV headers, so you will have to list list all properties of all objects
$sourceFilePath = "C:\can\Test\XY.json"
$json = Get-Content $sourceFilePath -Raw | ConvertFrom-Json
$all = #( ($json.header_items | Select-Object *, #{Name = 'ItemType'; Expression = { 'HeaderItem' } }) ) + ($json.items | Select-Object *, #{Name = 'ItemType'; Expression = { 'Item' } })
$properties = ($all | ForEach-Object { $_ | Get-Member -MemberType NoteProperty}) | Select-Object -Unique -ExpandProperty Name
$destinationFilePath = "C:\can\Test\XY.jcsv"
$all | Select-Object -Property $properties | Export-Csv -NoTypeInformation -Path $destinationFilePath
Csv file opened and formatted with Excel
Regards,

Related

Adding a substring in powershell

I import a csv file in powershell with this code:
import-csv "input.csv" | ConvertTo-Json | Add-Content -Path "output.json"
My output:
{
"users": [
{
"firstName": "name1",
"lastName": "lastname1",
}
{
"firstName": "name2",
"lastName": "lastname2",
}
}
The csv file look like this:
firstname,lastname
name1,lastname1
name2,lastname2
But I need to ad a extra column called "roles" with sub content (I dont know what its called). It should be after lastname, but still part of the users object and in every entry.
Need this output
{
"users": [
{
"firstName": "name1",
"lastName": "lastname1"
"roles:" ["role1", "role2"]
}
{
"firstName": "name2",
"lastName": "lastname2",
"roles:" ["role1", "role2"]
}
}
New output
"Roles": {
"value": [
"Role1",
"Role2"
],
"Count": 2
}
Add-Member is a somewhat slow method but viable...
$json = ConvertFrom-Json #'
{
"users": [{
"firstName": "name1",
"lastName": "lastname1"
},
{
"firstName": "name2",
"lastName": "lastname2"
}
]
}
'#
$json.users | Add-Member -MemberType NoteProperty -Name Roles -Value 'empty'
$json | ConvertTo-Json
Output
{
"users": [{
"firstName": "name1",
"lastName": "lastname1",
"Roles": "empty"
},
{
"firstName": "name2",
"lastName": "lastname2",
"Roles": "empty"
}
]
}
But we are leaning towards you are looking for something like this instead...
$json | Add-Member -MemberType NoteProperty -Name roles -Value #('role1','role2')
Output
{
"users": [
{
"firstName": "name1",
"lastName": "lastname1"
},
{
"firstName": "name2",
"lastName": "lastname2"
}
],
"roles": [
"role1",
"role2"
]
}
#'
firstname,lastname
name1,lastname1
name2,lastname2
'# |
ConvertFrom-Csv |
Select-Object -Property *,#{Name = 'Roles';Expression= {#('Role1','Role2')}} |
ConvertTo-Json
... produces this output:
[
{
"firstname": "name1",
"lastname": "lastname1",
"Roles": [
"Role1",
"Role2"
]
},
{
"firstname": "name2",
"lastname": "lastname2",
"Roles": [
"Role1",
"Role2"
]
}
]

Compare 2 JSON and retrieve subset from one of them based on condition in Powershell

I have two JSON files abc.json and xyz.json.
Content in abc.json is:
[{"id": "121",
"name": "John",
"location": "europe"
},
{"id": "100",
"name": "Jane",
"location": "asia"
},
{"id": "202",
"name": "Doe",
"location": "america"
}
]
Updated -> Content in xyz.json is:
{
"value": [
{
"id": "111",
"city": "sydney",
"profession": "painter"
},
{
"id": "200",
"city": "istanbul",
"profession": "actor"
},
{
"id": "202",
"city": "seattle",
"profession": "doctor"
}
],
"count": {
"type": "Total",
"value": 3
}
}
I want to get those records of abc.json in when the id in both objects are equal.In this case:
{"id": "202",
"name": "Doe",
"location": "america"
}
I need to do this in Powershell and the version I am using is 5.1.This is what I have tried:
$OutputList = #{}
$abcHash = Get-Content 'path\to\abc.json' | Out-String | ConvertFrom-Json
$xyzHash = Get-Content 'path\to\xyz.json' | Out-String | ConvertFrom-Json
$xyzResp = $xyzHash.value
foreach($item in $xyzResp){
foreach ($record in $abcHash){
if ($item.id -eq $record.id){
$OutputList.Add($record, $null)
}
}
}
Write-Output $OutputList
But on printing the OutputList , I get like this:
Key: #{"id": "202",
"name": "Doe",
"location": "america"
}
Value:
Name:#{"id": "202",
"name": "Doe",
"location": "america"
}
What I require is more of a PSObject like:
id: 202
name:Doe
location:america
I tried using Get-Member cmdlet but could not quite reach there.
Is there any suggestion I could use?
I have corrected your example xyz.json because there was an extra comma in there that should not be there. Also, the example did not have an iten with id 202, so there would be no match at all..
xyz.json
{
"value": [
{
"id": "111",
"city": "sydney",
"profession": "painter"
},
{
"id": "202",
"city": "denver",
"profession": "painter"
},
{
"id": "111",
"city": "sydney",
"profession": "painter"
}
],
"count": {
"type": "Total",
"value": 3
}
}
That said, you can use a simple Where-Object{...} to get the item(s) with matching id's like this:
$abc = Get-Content 'path\to\abc.json' -Raw | ConvertFrom-Json
$xyz = Get-Content 'path\to\xyz.json' -Raw | ConvertFrom-Json
# get the items with matching id's as object(s)
$abc | Where-Object { $xyz.value.id -contains $_.id}
Output:
id name location
-- ---- --------
202 Doe america
Of course you can capture the output first and display as list and/or save to csv, convert back to json and save that.

Remove an attribute from the JSON object in powershell

I have following JSON and I would like to remove streets from the JSON object under Address which is an array. I am trying to do this in powershell
{
"Customer": [
{
"id": "123"
}
],
"Nationality": [
{
"name": "US",
"id": "456"
}
],
"address": [
{
"$type": "Home",
"name": "Houston",
"streets": [
{
"name": "Union",
"postalCode": "10",
}
]
},
{
"$type": "Office",
"name": "Hawai",
"streets": [
{
"name": "Rock",
"postalCode": "11",
}
]
}
],
"address": [
{
"$type": "Home1",
"name": "Houston",
"streets": [
{
"name": "Union1",
"postalCode": "14",
}
]
},
{
"$type": "Office1",
"name": "Hawaii1",
"streets": [
{
"name": "Rock1",
"postalCode": "15",
}
]
}
],
}
I would like to remove streets from the JSON object and here is my powershell script but it is not working! I am trying to convert JSON into object and then loop over properties to remove those.
$FileContent = Get-Content -Path "Test.json" -Raw | ConvertFrom-Json
foreach ($content in $FileContent) {
#Write-Host $content.address
$content.address = $content.address | Select-Object * -ExcludeProperty streets
}
$FileContent | ConvertTo-Json -Depth 100 | Out-File "Test.json" -Force
When you use ConvertFrom-Json it will automatically convert things to objects for you. Once they're objects you can use Select-Object to specify what properties you want to include in the pipeline, with which you can just set $FileContent.address (an array of objects) to equal itself, excluding the streets property from each object in the array.
$FileContent = Get-Content -Path "Test.json" -Raw | ConvertFrom-Json
$FileContent.address = $FileContent.address | Select-Object * -ExcludeProperty streets
$FileContent | ConvertTo-Json

Can't run powershall script

I have a powershell script like that:
Param(
[Parameter(Mandatory=$True)]
[String]
$NEWPARAMVALUE,
[Parameter(Mandatory=$True)]
[String]
$PARAM_NAME
)
$FILEPATH = “arm_template.json”
$JSON = Get-Content $FILEPATH | Out-String | ConvertFrom-Json
$JSON.$PARAM_NAME = $NEWPARAMVALUE
$JSON | ConvertTo-Json -Depth 10 | Set-Content $FILEPATH
When I used this script without $PARAM_NAME variable (I just put a parameter as a plain text, example: $JSON.resources.properties.parameters.daasServiceBaseUrl.defaultValue) - it worked.
I tried different ways:
$JSON.resources.$PARAM_NAME (where $PARAM_NAME = 'properties.parameters.daasServiceBaseUrl.defaultValue')
$JSON"."$PARAM_NAME"
I think that's because of the point between two env variables. Could please someone help with it?
Update:
Error that I get from Azure DevOps:
The property
| 'properties.parameters.parametername.defaultValue' cannot
| be found on this object. Verify that the property exists and
| can be set.
How I tried it manually:
$env:PARAM_NAME="resources.properties.parameters.daasServiceBaseUrl.defaultValue"
./test.ps1 -NEWPARAMVALUE "[parameters('DaaS_Contract_Daily_Trigger_properties_Daas-UI-to-Contract_parameters_daasServiceBaseUrl')]" -PARAM_NAME $env:PARAM_NAME
What I got:
Line |
12 | $JSON.$PARAM_NAME = $NEWPARAMVALUE
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Exception setting "resources.properties.parameters.daasServiceBaseUrl.defaultValue": "The property
| 'resources.properties.parameters.daasServiceBaseUrl.defaultValue' cannot be found on this object. Verify that the property exists and can be
| set."
My JSON:
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"dataFactoryName": {
"type": "string",
"defaultValue": ""
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Data Factory location matches Resource group location"
}
},
"project": {
"type": "string",
"defaultValue": "ct",
"metadata": {
"description": "Azure Tag used to track projects"
}
},
"environment": {
"type": "string",
"defaultValue": "default",
"metadata": {
"description": "Azure Tag used to track environments"
}
},
"sleepTime": {
"type": "int",
"defaultValue": "1"
},
"DaaS_Contract_Daily_Trigger_properties_Daas-UI-to-Contract_parameters_daasServiceBaseUrl": {
"type": "string",
"defaultValue": "http://digital-daas-service-om-sqa.nplabsusk8s.com/daas"
}
},
"resources": [
{
"apiVersion": "2018-06-01",
"name": "[parameters('dataFactoryName')]",
"location": "[parameters('location')]",
"tags": {
"project": "[parameters('project')]",
"environment": "[parameters('environment')]"
},
"type": "Microsoft.DataFactory/factories",
"identity": {
"type": "SystemAssigned"
},
"properties": {}
},
{
"name": "[concat(parameters('dataFactoryName'), '/pipeline1')]",
"type": "Microsoft.DataFactory/factories/pipelines",
"apiVersion": "2018-06-01",
"properties": {
"activities": [
{
"name": "Wait1",
"type": "Wait",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"waitTimeInSeconds": "[parameters('sleepTime')]"
}
},
{
"name": "Copy Delivery Request",
"type": "Copy",
"dependsOn": [
{
"activity": "Delete old Req File",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "RestSource",
"httpRequestTimeout": "00:01:40",
"requestInterval": "00.00:00:00.010",
"requestMethod": "GET"
},
"sink": {
"type": "JsonSink",
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"formatSettings": {
"type": "JsonWriteSettings",
"quoteAllText": true
}
},
"enableStaging": false
},
"inputs": [
{
"referenceName": "Rest_Json_File",
"type": "DatasetReference",
"parameters": {
"relativeURL": {
"value": "#{pipeline().parameters.daasServiceRelURL}/#{variables('createdDate')}",
"type": "Expression"
},
"BaseUrl": {
"value": "#pipeline().parameters.daasServiceBaseUrl",
"type": "Expression"
}
}
}
]
}
],
"parameters": {
"deliveryReqFileName": {
"type": "string",
"defaultValue": "delivery_request.json"
},
"daasServiceBaseUrl": {
"type": "string",
"defaultValue": "http://digital-daas-service-om-sqa.nplabsusk8s.com/daas"
}
},
"annotations": []
},
"dependsOn": []
}
]
}
When you pass a string "a.b.c.d" as the right-hand operand of the . member access operator, PowerShell will attempt to resolve the exact member name a.b.c.d, not "a, then b, then c, then d".
While this could be solved by executing Invoke-Expression against "${JSON}.${PARAM_NAME}", that also opens your script up to arbitrary execution of whatever the user might pass in as $PARAM_NAME.
To avoid that, you'll need to manually:
Split $PARAM_NAME into individual member names, like $names = $PARAM_NAME.Split('.'), then
Resolve each property name except for the last one manually
Assign to $resolvedObject."$last"
Here's how that might look in a generic helper function that takes care of step 2 and 3:
function Set-PropertyChained
{
param(
[Parameter(Mandatory = $true, ValueFromPipeline = $true, Position = 0)]
[ValidateScript({$_ -isnot [ValueType]})]
[object]$RootObject,
[Parameter(Mandatory = $true, Position = 1)]
[string[]]$MemberChain,
[Parameter(Mandatory = $true, Position = 2)]
[object]$Value,
[switch]$PassThru
)
begin {
$resolutionChain = #($MemberChain |Select -SkipLast 1)
$terminalMember = $($MemberChain |Select -Last 1)
}
process {
$obj = $RootObject
# Member resolution up until the parent of the last property
foreach($memberName in $resolutionChain){
Write-Vebose "Resolving $memberName on $obj"
$obj = $obj.$memberName
}
# Set the value of the last property on its parent
Write-Vebose "Setting $memberName on $obj to $Value"
$obj.$terminalMember = $Value
if($PassThru){
# Write modified object back if requested
Write-Output $RootObject -NoEnumerate
}
}
}
Then in your script:
Param(
[Parameter(Mandatory=$True)]
[String]
$NEWPARAMVALUE,
[Parameter(Mandatory=$True)]
[String]
$PARAM_NAME
)
$FILEPATH = “arm_template.json”
$JSON = Get-Content $FILEPATH | Out-String | ConvertFrom-Json
# Step 1, split the resolution chain into individual components
$MemberNames = $PARAM_NAME.Split('.')
# Step 2 + 3, resolve n-2 members in chain, then assign to n-1
$JSON | Set-PropertyChained -MemberChain $MemberNames -Value $NEWPARAMVALUE
$JSON | ConvertTo-Json -Depth 10 | Set-Content $FILEPATH

Nested Json to CSV powershell

I tried to solve it on my own but my knowlegde of PS and Json is just not so good that I can adapt other Solutions to my Problem.
The Tasks seems quite simple, but my json Structure which I get through a Rest API is kinda stupid for me.
Example JSON (which has more columns like (test-id,test-Name) but excatly the same logic over all columns in Fields):
{
"entities": [
{
"Fields": [
{
"Name": "test-id",
"values": [
{
"value": "1851"
}
]
},
{
"Name": "test-name",
"values": [
{
"value": "01_DUMMY"
}
]
}
],
"Type": "run",
"children-count": 0
},
{
"Fields": [
{
"Name": "test-id",
"values": [
{
"value": "1852"
}
]
},
{
"Name": "test-name",
"values": [
{
"value": "02_DUMMY"
}
]
}
],
"Type": "run",
"children-count": 0
}
],
"TotalResults": 2
}
I tried to following PS Script:
Get-Content $file -raw |
convertfrom-json | select -ExpandProperty entities |
select -ExpandProperty Fields |
select -ExpandProperty Values |
Export-CSV $OutputFile -NoTypeInformation
But my CSV Result Looks like this:
"value"
"1851"
"01_DUMMY"
"N"
I would like to receive following result:
test-id,test-Name,run,children-count
1851,01_DUMMY,run,0
1852,02_DUMMY,run,0