Converting JSON to NDJSON with Powershell - json

I'm trying to convert a text file that has an array of json objects to an NDJSON formatted file for another team to consume.
I've almost got it, except for one problem. I have an array of objects nested inside the JSON (which then has nested arrays and objects inside of it, the structure gets pretty complex, I'll include a sample below) and for whatever reason, when I use ConvertFrom-JSON it drops this nested array and in my output, I end up with a blank string for that key, instead of the nested array object. I tried using the -Depth flag but when I do that my output file ends up blank, which doesn't make a ton of sense to me. I don't have a whole lot of experience with powershell, so I'm not really sure where I'm going wrong here.
Code:
$JSONSourceFile = Get-Content -Path "input/sample.json" | ConvertFrom-JSON
$NDJSONTargetFile = "output/sample.json"
New-Item $NDJSONTargetFile -ItemType file
for ( $i = 0 ; $i -lt $JSONSourceFile.Length ; $i++) {
$item = $JSONSourceFile.item($i)
$row = ($item | ConvertTo-JSON -Compress)
Add-Content $NDJSONTargetFile $row
}
Input File:
[
{
"id": "1",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak2354",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "Y"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "N"
}
]
}
]
}
]
},
{
"id": "2",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak1234",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "Y"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "Y"
}
]
}
]
}
]
},
{
"id": "3",
"TransactionDttm": "2021-02-22T15:45:00:00.000-05:00",
"Array1": [
{
"UserID": "ak5678",
"Preferences": [
{
"CagegoryID": "01",
"CategoryName": "Reminder",
"Channels": [
{
"ChannelID": "1",
"ChannelName": "Email",
"Preference": "N"
},
{
"ChannelID": "2",
"ChannelName": "Text",
"Preference": "N"
}
]
}
]
}
]
}
]
And then when I convert it to the output, this is what I get:
{"id":"1","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak2354","Preferences":""}]}
{"id":"2","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak1234","Preferences":""}]}
{"id":"3","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak5678","Preferences":""}]}

Thanks to the comment from Doug Maurer I figured it out, I was adding -Depth to my ConvertFrom-Json command when it should have been in my ConvertTo-Json command. This is the final script and what it gives me:
$JSONSourceFile = Get-Content -Path "input/sample.json" | ConvertFrom-JSON
$NDJSONTargetFile = "output/sample.json"
New-Item $NDJSONTargetFile -ItemType file
for ( $i = 0 ; $i -lt $JSONSourceFile.Length ; $i++) {
$item = $JSONSourceFile.item($i)
$row = ($item | ConvertTo-JSON -Compress -Depth 20)
Add-Content $NDJSONTargetFile $row
}
and the output:
{"id":"1","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak2354","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"Y"},{"ChannelID":"2","ChannelName":"Text","Preference":"N"}]}]}]}
{"id":"2","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak1234","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"Y"},{"ChannelID":"2","ChannelName":"Text","Preference":"Y"}]}]}]}
{"id":"3","TransactionDttm":"2021-02-22T15:45:00:00.000-05:00","Array1":[{"UserID":"ak5678","Preferences":[{"CagegoryID":"01","CategoryName":"Reminder","Channels":[{"ChannelID":"1","ChannelName":"Email","Preference":"N"},{"ChannelID":"2","ChannelName":"Text","Preference":"N"}]}]}]}

Related

Iterate through a text file and add entries to the JSON using powershell

I have a text file with below data which is Dynamically generated, so the contents of the repo.txt keep changing
repo.txt -> Content as below
repo1
repo2
repo_3
And a JSON file
template.json -> Content as below
{
"name": "new-report-test-1",
"resources": {
"repositories": [
{
"name": "repoa"
},
{
"name": "repo_b"
}
]
},
"filters": {
"severities": [
"High"
]
}
}
now i want to read the repo list from the repo.txt file and update the template.json to below
{
"name": "new-report-test-1",
"resources": {
"repositories": [
{
"name": "repoa"
},
{
"name": "repo_b"
},
{
"name": "repo1"
},
{
"name": "repo2"
},
{
"name": "repo_3"
}
]
},
"filters": {
"severities": [
"High",
"Medium",
"Critical"
]
}
}
I have this bash script that does the job thiugh I'm unsure how to translate it to PoweShell. Any inputs would be highly appreciated
cat ./tmpgen_reponames.txt | while read repoEntry do echo " Repository: $repoEntry" cat <<< $(jq --arg REPOID "$repoEntry" '[ .[] , {"name":$REPOID} ]' tmpgen_repoarray.json) > tmpgen_repoarray.json done
You can accomplish this in PowerShell using ConvertFrom-Json to convert your Json into objects then following an object-oriented approach adding new custom objects to the .resources.repositories property and adding the new values Medium and Critical to the .filters.severities property of your Json, lastly, converting it back to a Json string with ConvertTo-Json:
$json = Get-Content path\to\template.json -Raw | ConvertFrom-Json
$json.resources.repositories = #(
$json.resources.repositories
(Get-Content path\to\repo.txt).Trim() | ForEach-Object {
[pscustomobject]#{ Name = $_ }
}
)
$json.filters.severities = #(
$json.filters.severities
'Medium'
'Critical'
)
$json | ConvertTo-Json -Depth 99 | Set-Content path\to\newJson.Json

Powershell replace values in target.json with values from source.json

I have a big target json file (parameters_general.json) where all common settings for a deployment are set.
For each tier I have another json file (ex: parameters_dev.json, parameters_test.json, ....) Settings set in one needs tobe added to the general.json, or overwrite it when already in general.
ex: parameters_general.json
{
"general": {
"db_Policy": {
"type": "Periodic",
"databaseAccountOfferType": "Standard",
"periodicModeProperties": {
"backupIntervalInMinutes": 240,
"backupRetentionIntervalInHours": 8,
"backupStorageRedundancy": "Local"
}
},
"databases": [
{
"name": "CtrlWps",
"Containers": [
{
"name": "ControllerAuthentication",
"partitionKey": "id"
}
],
"ContainersTTL": []
},
{
"name": "CpoOcpi",
"Containers": [
{
"name": "Cpos",
"partitionKey": "cpoId"
},
{
"name": "OcpiCdrLastRecoveries",
"partitionKey": "id"
},
{
"name": "Routes",
"partitionKey": "ocpiCpoId"
}
],
"ContainersTTL": [
{
"name": "OcpiCdrs",
"partitionKey": "pk",
"ttl": 172800
},
{
"name": "OcppTransactionIds",
"partitionKey": "pk",
"ttl": 172800
},
{
"name": "Sessions",
"partitionKey": "pk",
"ttl": 172800
}
]
}
],
"system_engineers": [
],
}
}
If I want to updatethis with.
ex: parameters_test.json
{
"general": {
"system_engineers": [
{
"name": "hans",
"AppPrincipalId": "<id>",
"permissions": [
"get",
"list"
]
},
{
"name": "John do",
"AppPrincipalId": "<pid>",
"permissions": [
"all"
]
}
]
}
}
This works, the users are added to the empty "sytem_engineers" node in the parameters_general.json.
However, If I just want to change a setting on a lower node example:
ex: parameters_dev.json
{
"general": {
"databases": [
{
"name": "CtrlWps",
"Containers": [
{
"name": "ControllerAuthentication",
"partitionKey": "pk"
}
]
}
]
}
}
for replacing the partitionKey in one of the databases it replaces the whole "databases" part so I lose all other database configurations in the target.
The code I use is the following.
function ExtendJSON($base, $ext)
{
$propNames = $($ext | Get-Member -MemberType *Property).Name
foreach ($propName in $propNames) {
if ($base.PSObject.Properties.Match($propName).Count) {
if ($base.$propName.GetType().Name -eq "PSCustomObject")
{
$base.$propName = ExtendJSON $base.$propName $ext.$propName
}
else
{
$base.$propName = $ext.$propName
}
}
else
{
$base | Add-Member -MemberType NoteProperty -Name $propName -Value $ext.$propName
}
}
return $base
}
$tier = 'dev'
$parametersJsonGeneral = Get-Content -Path "./parameters/parameters_general.json" | ConvertFrom-Json
#Write-Output "#####################"
$parametersJsonTier = Get-Content -Path "./parameters/parameters_$($tier).json" | ConvertFrom-Json # overwrites existing values in $parametersJsonGeneral
ExtendJSON $parametersJsonGeneral $parametersJsonTier
Is there a way to loop over the settings from the lowest level up to the higher, and replace only these?
The proposed answer does only work one 1st level THIS WORKS
function merger ($target, $source) {
$source.psobject.Properties | ForEach-Object {
if ($_.TypeNameOfValue -eq 'System.Management.Automation.PSCustomObject' -and $target."$($_.Name)" ) {
merger $target."$($_.Name)" $_.Value
}
else {
$target | Add-Member -MemberType $_.MemberType -Name $_.Name -Value $_.Value -Force
}
}
}
$Json1 ='{
"a": {
"b":"asda"
},
"c": "asdasd"
}
' | ConvertFrom-Json
$Json2 = '{
"a": {
"b":"d"
}
}
' | ConvertFrom-Json
merger $Json1 $Json2
However with this I loose data in $Json1
$Json1 ='{
"a": {
"b": [
{
"name": "admin",
"appconfig": {
"test1": true,
"test2": false
}
}
]
},
"c": "asdasd"
}
' | ConvertFrom-Json
$Json2 = '{
"a": {
"b": [
{
"name": "admin",
"appconfig": {
"test1": false
}
}
]
}
}
' | ConvertFrom-Json
merger $Json1 $Json2
{
"a": {
"b": [
{
"name": "admin",
"appconfig": {
"test1": false
}
}
]
},
"c": "asdasd"
}
test2 is gone!
I don't have enough rep to post general comments yet, so I'll have to post it as an answer:
I tested the code in this answer to another question, and it did what you asked. The advantage of this answer is that it doesn't depend on importing a separate module.
There's another answer in the same question that provides link to a more general module that provides for different types of merges (Left Join, Inner Join, etc.) here, which links to here. However, one commenter said that it didn't work with powershell 7.0.

I need to remove an empty array with PowerShell

This my first post but have used this forum for years for some great advice so thank you to everyone that contributes. I have trawled this site and it feels like all of Google but still can't find a solution to my problem.
I have a task at work to supply a third party with some data in JSON format, this is taken from MS SQL Server and saved as a csv. I then use PowerShell to format it the way they want it in JSON.
After days of work its all working as it should apart from one bit, I need to remove empty arrays that will happen in my data if a customer doesn't have a device. Here is my PowerShell script;
$jsonBase = #{}
$requestor = 'exportedData'
$filename = (get-date).ToUniversalTime().ToString("yyyyMMddHHmmss")
$csvData = Import-Csv -Path example.csv
Function Remove-Null {
[CmdletBinding()]
Param(
# Object from which to remove the null values.
[Parameter(ValueFromPipeline,Mandatory)]
$InputObject,
# Instead of also removing values that are empty strings, include them
# in the output.
[Switch]$LeaveEmptyStrings,
# Additional entries to remove, which are either present in the
# properties list as an object or as a string representation of the
# object.
# I.e. $item.ToString().
[Object[]]$AlsoRemove = #()
)
Process {
# Iterate InputObject in case input was passed as an array
ForEach ($obj in $InputObject) {
$obj | Select-Object -Property (
$obj.PSObject.Properties.Name | Where-Object {
-not (
# If prop is null, remove it
$null -eq $obj.$_ -or
# If -LeaveEmptyStrings is not specified and the property
# is an empty string, remove it
(-not $LeaveEmptyStrings.IsPresent -and
[string]::IsNullOrEmpty($obj.$_)) -or
# If AlsoRemove contains the property, remove it
$AlsoRemove.Contains($obj.$_) -or
# If AlsoRemove contains the string representation of
# the property, remove it
$AlsoRemove.Contains($obj.$_.ToString()) -OR
# remove if it contains a string of 'NULL'
'NULL' -eq $obj.$_
)
}
)
}
}
}
# get an array of PSObjects
# we use 'Group-Object customerID' here to allow extra orders and PIDs
$allcustomers = $csvData | Group-Object customerID | ForEach-Object {
$orders = $_.Group | Select-Object -Unique orderType,orderStart,orderEnd,supplier
$devices = $_.Group | Select-Object -Unique deviceID,#{Name='deviceName';Expression={$_.customerID}},allocationStart,allocationEnd | Where-Object {$_.deviceID -ne 'NULL'}
$customer = $_.Group[0] | Select-Object -Unique * -ExcludeProperty orderType,orderStart,orderEnd,supplier,deviceID,allocationStart,allocationEnd
$newdevices = $devices | Remove-Null
$customer | Add-Member -MemberType NoteProperty -Name 'orders' -Value #($orders)
$customer | Add-Member -MemberType NoteProperty -Name 'devices' -Value #($newdevices)
# output the customer object
$customer
}
# gathered above into a new object and convert that to JSON
$json1 = [PsCustomObject]#{
applicationID = "1"
customers = #($allcustomers)
}
$jsonBase.Add("exportedData",$json1)
$jsonBase | ConvertTo-Json -Depth 5 | Set-Content -Encoding UTF8 -Path "${filename}.json"
{
"exportedData": {
"applicationID": "1",
"customers": [
{
"customerID": "1339306",
"customerName": "Fake Customer 1",
"customerDateOfBirth": "19870525",
"customerAddressText": "Fake Address 1",
"Manager": "Manager 1",
"xID": "UCYWIU",
"xRef": "AHZVXP",
"orders": [
{
"orderType": "Online",
"orderStart": "20210411230000",
"orderEnd": "20220410230000",
"supplier": "Supplier A"
}
],
"devices": [
{
"deviceID": "637148",
"deviceName": "1339306",
"allocationStart": "20210412192100"
}
]
},
{
"customerID": "1339321",
"customerName": "Fake Customer 2",
"customerDateOfBirth": "19960102",
"customerAddressText": "Fake Address 2",
"Manager": "Manager 2",
"xID": "ULYKHN",
"xRef": "STKRFZ",
"orders": [
{
"orderType": "Online",
"orderStart": "20210413230000",
"orderEnd": "20210720230000",
"supplier": "Supplier B"
}
],
"devices": [
{
"deviceID": "641358",
"deviceName": "1339321",
"allocationStart": "20210417183200",
"allocationEnd": "20210508181500"
},
{
"deviceID": "641358",
"deviceName": "1339321",
"allocationStart": "20210508181500",
"allocationEnd": "20210612190500"
},
{
"deviceID": "641358",
"deviceName": "1339321",
"allocationStart": "20210612190500",
"allocationEnd": "20210721193400"
}
]
},
{
"customerID": "1339325",
"customerName": "Fake Customer 3",
"customerDateOfBirth": "19750405",
"customerAddressText": "Fake Address 3",
"Manager": "Manager 3",
"xID": "BCQPQJ",
"xRef": "MTTJBJ",
"orders": [
{
"orderType": "Email",
"orderStart": "20210418230000",
"orderEnd": "20220418230000",
"supplier": "Supplier C"
}
],
"devices": [
{
"deviceID": "641360",
"deviceName": "1339325",
"allocationStart": "20210419205600",
"allocationEnd": "20210602211100"
},
{
"deviceID": "646142",
"deviceName": "1339325",
"allocationStart": "20210602211100",
"allocationEnd": "20210628194500"
},
{
"deviceID": "641831",
"deviceName": "1339325",
"allocationStart": "20210628194600"
}
]
},
{
"customerID": "1355689",
"customerName": "Fake Customer 4",
"customerDateOfBirth": "19891005",
"customerAddressText": "Fake Address 4",
"Manager": "Manager 4",
"xID": "OYNLQL",
"xRef": "DMSUAP",
"orders": [
{
"orderType": "Email",
"orderStart": "20210728230000",
"orderEnd": "20220728230000",
"supplier": "Supplier B"
}
],
"devices": [
]
},
{
"customerID": "1355826",
"customerName": "Fake Customer 5",
"customerDateOfBirth": "19891218",
"customerAddressText": "Fake Address 5",
"Manager": "Manager 5",
"xID": "XSWTNH",
"xRef": "FCIIYO",
"orders": [
{
"orderType": "Online",
"orderStart": "20210728230000",
"orderEnd": "20220728230000",
"supplier": "Supplier B"
}
],
"devices": [
]
}
]
}
}
What I want to do is remove any empty arrays, so the ones that don't have devices I want to hide/remove the array completely so that the "devices" : [] doesn't appear.
I have used a function to remove elements from an array if the value is a string 'NULL'.
I have looked at ForEach and ForEach-Object but cant seem to iterate over the arrays themselves to remove them. As a last attempt I was going to read the JSON back in and use -replace to change the string but couldnt get it to work with the carriage returns and spaces.
This is what the CSV looks like.
The CSV that is imported and transformed in PowerShell
If anyone could help that would be so great.
This requires an recursive function.
To make things easier, I recommend to use the ConvertFrom-Json -AsHashTable parameter. This way, you only need to deal with arrays and hashtables:
$Data = $Json |ConvertFrom-Json
function Remove-EmptyArrays ($Object) {
if ($Object -is [Array]) {
foreach ($Item in $Object) { Remove-EmptyArrays $Item }
}
elseif ($Object -is [HashTable]) {
foreach ($Key in #($Object.get_Keys())) {
if ($Object[$Key] -is [Array] -and $Object[$Key].get_Count() -eq 0) {
$Object.Remove($Key)
}
else { Remove-EmptyArrays $Object[$Key] }
}
}
elseif ($Object -is [PSCustomObject]) {
foreach ($Name in #($Object.psobject.properties.Name)) {
if ($Object.$Name -is [Array] -and $Object.$Name.get_Count() -eq 0) {
$Object.PSObject.Properties.Remove($Name)
}
else { Remove-EmptyArrays $Object.$Name }
}
}
}
Remove-EmptyArrays $Data
$Data |ConvertTo-Json -Depth 9
Update 2021-07-30
I have updated the script to also support [PSCustomObject] types to support Windows PowerShell where ConvertFrom-Json doesn't have a -AsHashTable parameter.

Create a json document based on existing document with powershell 5.1

I'm trying to build a json document which takes it's structure, column names and some values from an already existing base json document, and sets other values to variables assigned via powershell 5.1 in a foreach-object loop.
I tried editing a copy of the base json and creating a new one entirely and I can get close to what I want but not exactly.
A cut down example json document is:
[
{
"name": "Application1",
"tags": ["monitor"],
"description": "Get information about application 1.",
"query":
[
{
"target": {
"version": ["v1","v2","v3","v4","v5"],
"platform": ["Windows","Linux"]
},
"implementation": {
"action": "do something"
}
}
]
},
{
"name": "Application2",
"tags": ["monitor"],
"description": "Get information about application 2",
"query":
[
{
"target": {
"version": ["v1","v2","v3"],
"platform": ["Windows"]
},
"implementation": {
"action": "do something for v1, v2 and v3"
}
},
{
"target": {
"version": ["v4","v5"],
"platform": ["Windows"]
},
"implementation": {
"action": "do something for v4 and v5"
}
}
]
}
]
My script currently looks like:
$BaseJson = (Get-Content "$SourcePath\probes.json" | ConvertFrom-Json)
$ClientConfig = #"
{
"targetHost": "$DigitalSymphonyHost",
"targetPort": "$DigitalSymphonyPort",
"source": "$TargetSqlInstance",
"sensor": {}
}
"#
$ClientJson = ConvertFrom-Json -InputObject $ClientConfig
$BaseJson | Where-Object {$_.tags -contains "monitor"} | ForEach-Object {
# For each sensor in the base-sensors json document
$SensorName = $_.name
$Severity = 2
$Version = $_.query.target.version
$Platform = $_.query.target.platform
$Query = $_.query.implementation.action
$Sensor =#"
{
"severity": "$Severity",
"sensorId": "$SensorId",
"type": "$Type",
"target": {
"version": "$Version",
"platform": "$Platform",
"engineEdition": "$Edition"
},
"implementation": {
"query": "$Query"
}
}
"#
$ClientJson.sensor | Add-Member -Name $SensorName -Value (ConvertFrom-Json $Sensor) -MemberType NoteProperty
}
$ClientJson | ConvertTo-Json | Out-File "$DestinationPath\client-sensors.json"
My desired result is to add the $SensorId as per:
[
{
"targetHost": "servername",
"targetPort": "port",
"source": "servername",
"sensor": [{
"name": "Application1",
"sensorId": "<value_from_$SensorId>",
"tags": [],
"description": "Get information about application 1.",
"query":
[
{
"target": {
"version": ["v1","v2","v3","v4","v5"],
"platform": ["Windows","Linux"]
},
"implementation": {
"action": "do something"
}
}
]
},
{
"name": "Application2",
"sensorId": "<value_from_$SensorId>",
"tags": [],
"description": "Get information about application 2",
"query":
[
{
"target": {
"version": ["v1","v2","v3"],
"platform": ["Windows"]
},
"implementation": {
"action": "do something for v1, v2 and v3"
}
},
{
"target": {
"version": ["v4","v5"],
"platform": ["Windows"]
},
"implementation": {
"action": "do something for v4 and v5"
}
}
]
}]
}
]
My current result is
{
"targetHost": "servername",
"targetPort": "port",
"source": "server",
"sensor": {
"applicationname1": {
"tags": ["monitor],
"description": "Get information about application 2",
"sensorId": "420",
"target": "#{version=v1,v2,v3,v4,v5; platform=Windows Windows;}",
"action": "do something for v4 and v5"
}
}
}
I'm not sure where you're getting sensorID, but I'd do this more like the below:
param(
$SourcePath = $PSScriptRoot,
$DestinationPath = $PSScriptRoot,
$DigitalSymphonyHost = 'SomeHost',
$DigitalSymphonyPort = '8765',
$TargetSqlInstance
)
$BaseObj = (Get-Content "$SourcePath\probes.json" | ConvertFrom-Json)
$Client = [PSCustomObject] #{
targetHost = $DigitalSymphonyHost
targetPort = $DigitalSymphonyPort
source = $TargetSqlInstance
sensor = #()
}
$BaseObj | Where-Object {$_.tags -contains "monitor"} | ForEach-Object {
$sensor = $_
# TODO: How are you getting sensorId?
$sensor | Add-Member -Name 'sensorId' -Value 777 -MemberType NoteProperty
$Client.sensor += $sensor
}
ConvertTo-Json #($Client) -Depth 10 | Out-File "$DestinationPath\client-sensors.json" -Force
That will get you the JSON you're looking for.

Need to convert nested Json response to csv in powershell

I have below sample nested json response and I need to convert this response with specific values into CSV file. Below is the sample nested Json response:
{
"transaction": {
"id": "TestTransID",
"testCode": "NEW",
"TestStatus": "SUCCESS",
"client": {
"TestNumber": "112112111"
},
"subject": {
"individual": {
"additionalAttributes": {
"extraid": "787877878"
},
"addressList": [
{
"city": "New York",
"country": {
"name": "United States"
},
"postalCode": "123456789",
"stateOrProvince": {
"codeValue": "NY"
}
}
],
"gender": "F",
"identificationDocumentList": [
{
"number": "1214558520",
"type": "TestId"
}
],
"name": {
"firstName": "Qusay TestFull",
"lastName": "TestLast",
"middleName": "Middle 3"
}
}
},
"PROCESSConfiguration": {
"id": 1
},
"testProductList": [
{
"product": {
"id": 00,
"name": "Test PROCESS",
"productCode": "EFG",
"disclaimer": "TestDisclaimer"
},
"testSourceResponseList": [
{
"testSource": {
"id": 1,
"name": "TEST"
},
"testSourceRecordList": [
{
"type": "TestRecord",
"alertReasonCode": "TESTS",
"alertReasonDescription": "ACTION LIST HIT - TEST",
"testSource": "TEST",
"varListNameFull": "TEST FULL NAME",
"varListNameShort": "TEST SHORT",
"varProgList": [
"SHORT"
],
"varListId": "3421",
"subject": {
"individual": {
"TestScore": {
"TestScore": 100,
"triggeredRule": "TestRule"
},
"aNameList": [
{
"fullName": " TestNameA",
"lastName": "TestNameA"
},
{
"firstName": "TestFirst",
"fullName": "TestFirst HUSAYN",
"lastName": "TestLast"
},
{
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
},
{
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
}
],
"birthList": [
{
"dateOfBirth": "12 Apr 1910",
"dateOfBirthVerified": "true"
}
],
"name": {
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast"
},
"varNationality": [
{
"verified": "true"
}
],
"remarks": "remark1"
}
}
},
{
"testSource": "TEST",
"varListNameFull": "TEST FULL",
"varListNameShort": "TEST SHORT",
"varProgList": [
"XYZ"
],
"varListId": "1234",
"subject": {
"individual": {
"overallScore": {
"TestScore": 100,
"triggeredRule": "Testing"
},
"birthList": [
{
"dateOfBirth": "1965",
},
{
"dateOfBirth": "1966",
}
],
"name": {
"firstName": "TestFirst",
"fullName": "TestFull",
"lastName": "TestLast",
},
"varNationality": [
{
"verified": "true"
}
],
"remarks": "REMARK2"
}
}
}
]
}
]
}
],
}
}
I need to take response from ""PROCESSConfiguration": {
"id": 1"
from row # 40. If u'll take above code in notepad ++.
Also I need the response with respect to var value like first name, last name full name, DOB etc.
I am still unsure what is being asked for. Let me assume that you want a fully qualified path for each of the elements in the JSON file.
I started by making some small adjustments to the JSON based on http://jsonlint.com/.
Based on that, I did a proof of concept that works for ONE OBJECT like the JSON you posted. It works for THIS CASE. I wrapped the logic to deal with multiple objects making assumptions about when one object started and the next began. Also, logic would should be added to deal with nested series/array containing multiple properties (i.e. Threads in Get-Process) to handle a generic case. A single element with an array of values (like .transaction.varProgList in this case) is handled by putting a ‘;’ between them.
CSV files assume that all the objects are symmetric (have the same properties). There is no check to see that the properties for each object align with the properties of the other objects. Note the handling of nested series is related to this. You may see an example of this by uncommenting the [System.Collections.ICollection] section and trying something like (Get-Process r*) | Select-Object Name,Threads | Expand-NestedProperty | Out-File .\t.csv -Width 100000
The repro goes as follows where $a is the adjusted JSON content and the function is saved as Expand-NestedProperty.ps1.
# Load the function
. .\Expand-NestedProperty.ps1
# Create PowerShell object(s) based on the JSON
$b = $a | ConvertFrom-Json
# Create a file with the CSV contents.
$b | Expand-NestedProperty | Out-File -FilePath .\my.csv -Width 100000
Save this as Expand-NestedProperty.ps1
function Expand-NestedProperty {
[CmdletBinding()]
param (
[Parameter( Position=0,
Mandatory=$true,
ValueFromPipeline=$true,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='Object required...' )]
$InputObject
)
begin {
function ExpandNestedProperty {
[CmdletBinding()]
param (
[Parameter( Position=0,
Mandatory=$true,
ValueFromPipeline=$true,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='Object required...' )]
$InputObject,
[Parameter( Position=1,
Mandatory=$false,
ValueFromPipeline=$false,
ValueFromPipelineByPropertyName=$true,
ValueFromRemainingArguments=$false,
HelpMessage='String required...' )]
[string]
$FullyQualifiedName = ""
)
begin {
$localResults =#()
$FQN = $FullyQualifiedName
$nestedProperties = $null
}
process {
foreach ($obj in $InputObject.psobject.Properties) {
if ($(try {$obj.Value[0] -is [PSCustomObject]} catch {$false})) { # Catch 'Cannot index into a null array' for null values
# Nested properties
$FQN = "$($FullyQualifiedName).$($obj.Name)"
$nestedProperties = $obj.value | ExpandNestedProperty -FullyQualifiedName $FQN
}
elseif ($obj.Value -is [array]) {
# Array property
$FQN = "$($FullyQualifiedName).$($obj.Name)"
[psobject]$nestedProperties = #{
$FQN = ($obj.Value -join ';')
}
}
# Example of how to deal with generic case.
# This needed for the Get-Process values ([System.Collections.ReadOnlyCollectionBase] and [System.Diagnostics.FileVersionInfo]) that are not [array] collection type.
<#
elseif ($obj.Value -is [System.Collections.ICollection]) {
# Nested properties
$FQN = "$($FullyQualifiedName).$($obj.Name)"
$nestedProperties = $obj.value | ExpandNestedProperty -FullyQualifiedName $FQN
}
#>
else { # ($obj -is [PSNoteProperty]) for this case, but could be any type
$FQN = "$($FullyQualifiedName).$($obj.Name)"
[psobject]$nestedProperties = #{
$FQN = $obj.Value
}
}
$localResults += $nestedProperties
} #foreach $obj
}
end {
[pscustomobject]$localResults
}
} # function ExpandNestedProperty
$objectNumber = 0
$firstObject = #()
$otherObjects = #()
}
process {
if ($objectNumber -eq 0) {
$objectNumber++
$firstObject = $InputObject[0] | ExpandNestedProperty
}
else {
if ($InputObject -is [array]) {
foreach ($nextInputObject in $InputObject[1..-1]) {
$objectNumber++
$otherObjects += ,($nextInputObject | ExpandNestedProperty)
}
}
else {
$objectNumber++
$otherObjects += ,($InputObject | ExpandNestedProperty)
}
}
}
end {
# Output CSV header and a line for each object which was the specific requirement here.
# Could create an array of objects using $firstObject + $otherObjects that is then piped to Export-CSV if we want a generic case.
Write-Output "`"$($firstObject.Keys -join '","')`""
Write-Output "`"$($firstObject.Values -join '","')`""
foreach ($otherObject in $otherObjects) {
Write-Output "`"$($otherObject.Values -join '","')`""
}
}
} # function Expand-NestedProperty