I need to pull data with a particular heading from a json file and output it to a csv file
$data = (Get-Content "C:\Users\QVL6\Downloads\express-ordering-web-
variables.json" | ConvertFrom-Json)
get data
[PSCustomObject[]]$data = #(
[PSCustomObject]#{
Name = 'Name'
Type = 'Type'
Value = 'Value'
Description = 'Description'
}
)
$path = C:\Users\QVL6\
$data | Select-Object -Property Name, Type, Value, Description | Export -Csv
-Path .\data.csv -NoClobber -NoTypeInformation
Json file:
{
"Id": "variableset-Projects-174",
"OwnerId": "Projects-174",
"Version": 23,
"Variables": [
{
"Id": "dfd06d9f-5ab5-0b40-bfed-d11cd0d90e62",
"Name": "apiConfig:orderCommandUrl",
"Value": "http://dev.order-service.local",
"Description": null,
"Scope": {
"Environment": [
"Environments-63"
]
},
"IsEditable": true,
"Prompt": null,
"Type": "String",
"IsSensitive": false
},
{
"Id": "252a19a0-4650-4920-7e66-39a80c1c49ec",
"Name": "apiConfig:orderCommandUrl",
"Value": "http://qa.order-service.local",
"Description": null,
"Scope": {
"Environment": [
"Environments-63",
"Environments-64"
]
},
"IsEditable": true,
"Prompt": null,
"Type": "String",
"IsSensitive": false
},
I want to pull out all the values in Name field
Get-Content already returns a string so you could convert the output directly to json.
$data = Get-Content "C:\Users\myFile\Downloads\express-ordering-web-variables.json" | ConvertFrom-Json
The variable $data is already an object so you don't have to convert it again to a csv. You can directly select the needed headers and export them to a csv.
$data = Get-Content "C:\Users\myFile\Downloads\express-ordering-web-
variables.json" | ConvertFrom-Json
#get some random data
[PSCustomObject[]]$data = #(
[PSCustomObject]#{
H1 = 'Test'
H2 = 'Test2'
},
[PSCustomObject]#{
H1 = 'Test'
H2 = 'Test2'
}
)
$data | Select-Object -Property H1, H2 | Export-Csv -Path $Path -NoClobber -NoTypeInformation
Related
This my first post but have used this forum for years for some great advice so thank you to everyone that contributes. I have trawled this site and it feels like all of Google but still can't find a solution to my problem.
I have a task at work to supply a third party with some data in JSON format, this is taken from MS SQL Server and saved as a csv. I then use PowerShell to format it the way they want it in JSON.
After days of work its all working as it should apart from one bit, I need to remove empty arrays that will happen in my data if a customer doesn't have a device. Here is my PowerShell script;
$jsonBase = #{}
$requestor = 'exportedData'
$filename = (get-date).ToUniversalTime().ToString("yyyyMMddHHmmss")
$csvData = Import-Csv -Path example.csv
Function Remove-Null {
[CmdletBinding()]
Param(
# Object from which to remove the null values.
[Parameter(ValueFromPipeline,Mandatory)]
$InputObject,
# Instead of also removing values that are empty strings, include them
# in the output.
[Switch]$LeaveEmptyStrings,
# Additional entries to remove, which are either present in the
# properties list as an object or as a string representation of the
# object.
# I.e. $item.ToString().
[Object[]]$AlsoRemove = #()
)
Process {
# Iterate InputObject in case input was passed as an array
ForEach ($obj in $InputObject) {
$obj | Select-Object -Property (
$obj.PSObject.Properties.Name | Where-Object {
-not (
# If prop is null, remove it
$null -eq $obj.$_ -or
# If -LeaveEmptyStrings is not specified and the property
# is an empty string, remove it
(-not $LeaveEmptyStrings.IsPresent -and
[string]::IsNullOrEmpty($obj.$_)) -or
# If AlsoRemove contains the property, remove it
$AlsoRemove.Contains($obj.$_) -or
# If AlsoRemove contains the string representation of
# the property, remove it
$AlsoRemove.Contains($obj.$_.ToString()) -OR
# remove if it contains a string of 'NULL'
'NULL' -eq $obj.$_
)
}
)
}
}
}
# get an array of PSObjects
# we use 'Group-Object customerID' here to allow extra orders and PIDs
$allcustomers = $csvData | Group-Object customerID | ForEach-Object {
$orders = $_.Group | Select-Object -Unique orderType,orderStart,orderEnd,supplier
$devices = $_.Group | Select-Object -Unique deviceID,#{Name='deviceName';Expression={$_.customerID}},allocationStart,allocationEnd | Where-Object {$_.deviceID -ne 'NULL'}
$customer = $_.Group[0] | Select-Object -Unique * -ExcludeProperty orderType,orderStart,orderEnd,supplier,deviceID,allocationStart,allocationEnd
$newdevices = $devices | Remove-Null
$customer | Add-Member -MemberType NoteProperty -Name 'orders' -Value #($orders)
$customer | Add-Member -MemberType NoteProperty -Name 'devices' -Value #($newdevices)
# output the customer object
$customer
}
# gathered above into a new object and convert that to JSON
$json1 = [PsCustomObject]#{
applicationID = "1"
customers = #($allcustomers)
}
$jsonBase.Add("exportedData",$json1)
$jsonBase | ConvertTo-Json -Depth 5 | Set-Content -Encoding UTF8 -Path "${filename}.json"
{
"exportedData": {
"applicationID": "1",
"customers": [
{
"customerID": "1339306",
"customerName": "Fake Customer 1",
"customerDateOfBirth": "19870525",
"customerAddressText": "Fake Address 1",
"Manager": "Manager 1",
"xID": "UCYWIU",
"xRef": "AHZVXP",
"orders": [
{
"orderType": "Online",
"orderStart": "20210411230000",
"orderEnd": "20220410230000",
"supplier": "Supplier A"
}
],
"devices": [
{
"deviceID": "637148",
"deviceName": "1339306",
"allocationStart": "20210412192100"
}
]
},
{
"customerID": "1339321",
"customerName": "Fake Customer 2",
"customerDateOfBirth": "19960102",
"customerAddressText": "Fake Address 2",
"Manager": "Manager 2",
"xID": "ULYKHN",
"xRef": "STKRFZ",
"orders": [
{
"orderType": "Online",
"orderStart": "20210413230000",
"orderEnd": "20210720230000",
"supplier": "Supplier B"
}
],
"devices": [
{
"deviceID": "641358",
"deviceName": "1339321",
"allocationStart": "20210417183200",
"allocationEnd": "20210508181500"
},
{
"deviceID": "641358",
"deviceName": "1339321",
"allocationStart": "20210508181500",
"allocationEnd": "20210612190500"
},
{
"deviceID": "641358",
"deviceName": "1339321",
"allocationStart": "20210612190500",
"allocationEnd": "20210721193400"
}
]
},
{
"customerID": "1339325",
"customerName": "Fake Customer 3",
"customerDateOfBirth": "19750405",
"customerAddressText": "Fake Address 3",
"Manager": "Manager 3",
"xID": "BCQPQJ",
"xRef": "MTTJBJ",
"orders": [
{
"orderType": "Email",
"orderStart": "20210418230000",
"orderEnd": "20220418230000",
"supplier": "Supplier C"
}
],
"devices": [
{
"deviceID": "641360",
"deviceName": "1339325",
"allocationStart": "20210419205600",
"allocationEnd": "20210602211100"
},
{
"deviceID": "646142",
"deviceName": "1339325",
"allocationStart": "20210602211100",
"allocationEnd": "20210628194500"
},
{
"deviceID": "641831",
"deviceName": "1339325",
"allocationStart": "20210628194600"
}
]
},
{
"customerID": "1355689",
"customerName": "Fake Customer 4",
"customerDateOfBirth": "19891005",
"customerAddressText": "Fake Address 4",
"Manager": "Manager 4",
"xID": "OYNLQL",
"xRef": "DMSUAP",
"orders": [
{
"orderType": "Email",
"orderStart": "20210728230000",
"orderEnd": "20220728230000",
"supplier": "Supplier B"
}
],
"devices": [
]
},
{
"customerID": "1355826",
"customerName": "Fake Customer 5",
"customerDateOfBirth": "19891218",
"customerAddressText": "Fake Address 5",
"Manager": "Manager 5",
"xID": "XSWTNH",
"xRef": "FCIIYO",
"orders": [
{
"orderType": "Online",
"orderStart": "20210728230000",
"orderEnd": "20220728230000",
"supplier": "Supplier B"
}
],
"devices": [
]
}
]
}
}
What I want to do is remove any empty arrays, so the ones that don't have devices I want to hide/remove the array completely so that the "devices" : [] doesn't appear.
I have used a function to remove elements from an array if the value is a string 'NULL'.
I have looked at ForEach and ForEach-Object but cant seem to iterate over the arrays themselves to remove them. As a last attempt I was going to read the JSON back in and use -replace to change the string but couldnt get it to work with the carriage returns and spaces.
This is what the CSV looks like.
The CSV that is imported and transformed in PowerShell
If anyone could help that would be so great.
This requires an recursive function.
To make things easier, I recommend to use the ConvertFrom-Json -AsHashTable parameter. This way, you only need to deal with arrays and hashtables:
$Data = $Json |ConvertFrom-Json
function Remove-EmptyArrays ($Object) {
if ($Object -is [Array]) {
foreach ($Item in $Object) { Remove-EmptyArrays $Item }
}
elseif ($Object -is [HashTable]) {
foreach ($Key in #($Object.get_Keys())) {
if ($Object[$Key] -is [Array] -and $Object[$Key].get_Count() -eq 0) {
$Object.Remove($Key)
}
else { Remove-EmptyArrays $Object[$Key] }
}
}
elseif ($Object -is [PSCustomObject]) {
foreach ($Name in #($Object.psobject.properties.Name)) {
if ($Object.$Name -is [Array] -and $Object.$Name.get_Count() -eq 0) {
$Object.PSObject.Properties.Remove($Name)
}
else { Remove-EmptyArrays $Object.$Name }
}
}
}
Remove-EmptyArrays $Data
$Data |ConvertTo-Json -Depth 9
Update 2021-07-30
I have updated the script to also support [PSCustomObject] types to support Windows PowerShell where ConvertFrom-Json doesn't have a -AsHashTable parameter.
I'm trying to sort the following JSON objects by the value of the "Index" key.
JSON = {
"PIPoint": {
"Label": "\"PIPoint\"",
"Visible": "True",
"Index": "2"
},
"Description": {
"Label": "\"Description\"",
"Visible": "True",
"Index": "3"
},
"Analysis": {
"Label": "\"Analysis\"",
"Visible": "True",
"Index": "4"
},
"PIPointExist": {
"Label": "\"PIPointExist\"",
"Visible": "True",
"Index": "5"
},
"Custom Location": {
"Label": "\"Custom Location\"",
"Visible": "True",
"Index": "1"
}
}
I've try the following code but it did not work
$json = Get-Content 'C:\column.json' | ConvertFrom-Json
$json = $json | Sort-Object -Property {$_.Index}
$json | ConvertTo-Json | Out-File 'C:\\column_sorted.json'
One way to do this would be to sort the properties and use Select-Object to reorder the output:
# Return objects based on JSON strings
$jsonObjs = Get-Content 'C:\column.json' | ConvertFrom-Json
# Sorted properties list
$Properties = ($jsonObjs.psobject.properties | Sort-Object {[int]$_.Value.Index}).Name
# Create new JSON strings with new order
$jsonObjs | Select-Object $Properties | ConvertTo-Json
$jsonĀ¹ is a single object (with just 5 properties).
So you need to sort the properties:
$Data = Get-Content 'C:\column.json' | ConvertFrom-Json
$Properties = [ordered]#{}
$Data.PSObject.Properties |Sort-Object { $_.Value.Index } |ForEach-Object { $Properties[$_.Name] = $_.Value }
[pscustomobject]$Properties | ConvertTo-Json
I would call it $Json as it is no longer a Json string but an object
{
"value": [{
"ExternalKey": "12345",
"PortfolioId": "ABC",
"InceptionDate": null,
"TerminationDate": null,
"Version": 2,
"SourceRef": "ABC",
"MasterSeries": [{
"Points": [{
"Date": "1900-01-01T00:00:00+00:00",
"Name": "XYZ Name",
"Type": "IP",
"Status": "Active",
"Version": 1
}],
"ExternalKey": "12345",
"Version": 1,
"IsDeleted": false,
"SourceRef": "ABC"
}]
}]
}
I tried to loop through each level and manually selecting the keys
ForEach($d in $json.value) { $row=$d.column1+","+ $d.column2 ..
$rootProps = $d| Get-Member | Where-Object { $_.MemberType -match "Property"}
$rootProps | ForEach-Object {
if($_.Name -eq "MasterSeries") {
$d | select -Expand MasterSeries | select-object * -ExcludeProperty Points | ForEach { $row = $_.column1 + "," $_.column2 ..
}
}
and so on..... but when I export it to Out-File "Export.csv" and try importing it gives me a UTF error. Is there a better way to achieve?
If this helps any, here's the typical way to loop through json properties in powershell 5 using the hidden psobject.properties. You might search for "flatten json".
$a = get-content file.json | convertfrom-json
$a.value[0].psobject.properties | select name,value
Name Value
---- -----
ExternalKey 12345
PortfolioId ABC
InceptionDate
TerminationDate
Version 2
SourceRef ABC
MasterSeries
For this you will need to create a recursive function.
To make things easier, I recommend you to use the ConvertFrom-Json -AsHashTable parameter which will return [HashTable] objects rather than [PSCustomObject] Objects
Function Get-Values($Object) {
if ($Object -is [String]) { $Object } # A string is also Enumerable
elseif ($Object -is [Collections.IDictionary]) { $Object.get_Values() | ForEach-Object { Get-Values $_ } }
elseif ($Object -is [Collections.IEnumerable]) { $Object.GetEnumerator() | ForEach-Object { Get-Values $_ } }
else { $Object }
}
$Object = ConvertFrom-Json -AsHashTable '{
"value": [
{
"ExternalKey": "12345",
"PortfolioId": "ABC",
"InceptionDate": null,
"TerminationDate": null,
"Version": 2,
"SourceRef": "ABC",
"MasterSeries": [
{
"Points": [
{
"Date": "1900-01-01T01:00:00+01:00",
"Name": "XYZ Name",
"Type": "IP",
"Status": "Active",
"Version": 1
}
],
"ExternalKey": "12345",
"Version": 1,
"IsDeleted": false,
"SourceRef": "ABC"
}
]
}
]
}'
Get-Values $Object
12345
False
1
IP
Monday, January 1, 1900 1:00:00 AM
XYZ Name
Active
ABC
1
12345
2
ABC
ABC
I would like to add an additional key with value into my existing JSON file. Unfortunately I'm not able. Here an short overview:
My JSON-File before powershell script is run:
[
{
"id": "1",
"description": [
{
"country": "Brazil"
},
{
"country": "Mexico"
}
]
},
{
"id": "2",
"description": [
{
"country": "Argentina"
}
]
}
]
My wish, how the JSON-File should look like, after my powershell script is run:
[
{
"id": "1",
"description": [
{
"country": "Brazil",
"city": "Rio de Janeiro"
},
{
"country": "Mexico",
"city": "Mexico City"
}
]
},
{
"id": "2",
"description": [
{
"country": "Argentina",
"city": "Buenos Aires"
}
]
}
]
My powershell script:
function GetCity($country) {
$x = "not available"
If ( $country -eq "Brazil" ) { $x = "Rio de Janeiro" }
If ( $country -eq "Mexico" ) { $x = "Mexico City" }
If ( $country -eq "Argentina" ) { $x = "Buenos Aires" }
return $x
}
# Source the JSON content
$jsonFile = 'C:\Temp\test.json'
$jsonContent = Get-Content -Path $jsonFile
# Convert JSON to PSObjects
$jsonAsPsObjects = $jsonContent | ConvertFrom-Json
foreach ($info in $jsonAsPsObjects) {
$result = GetCity($info.description.country)
jsonContent | Add-Member -Type NoteProperty -Name "City" -Value $result
}
# Save JSON back to file
$json | ConvertTo-Json | Set-Content $jsonFile
Error:
jsonContent : The term 'jsonContent' is not recognized as the name of
a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path
is correct and try again.
How can I solve this issue?
There at two problems:
jsonContent should be $jsonContent in statement jsonContent | Add-Member ...
You're neglecting to loop over the array elements of the description property, to each of which a city property is to be added.
I suggest streamlining your code as follows:
function Get-City {
param([string] $country)
# Use a `switch` statement:
switch ($country) {
'Brazil' { return 'Rio de Janeiro' }
'Mexico' { return 'Mexico City' }
'Argentina' { return 'Buenos Aires' }
default { return 'not available' }
}
}
$jsonFile = 'C:\Temp\test.json'
(Get-Content -Raw $jsonFile | ConvertFrom-Json) | ForEach-Object {
# Add a 'city' property to each object in the 'description' property.
$_.description.ForEach({
Add-Member -InputObject $_ city (Get-City $_.country)
})
$_ # output the modified object
} | ConvertTo-Json -Depth 3 # | Set-Content $jsonFile
I have a json file with below contents
{
"createOrReplace": {
"object": {
"database": "DB_NAME"
},
"database": {
"name": "DB_NAME",
"compatibilityLevel": 1400,
"model": {
"culture": "en-IN",
"dataSources": [{
"name": "somename",
"connectionString": "somevalue",
"impersonationMode": "impersonateServiceAccount",
"annotations": [{
"name": "ConnectionEditUISource",
"value": "SqlServer"
}]
}]
}
}
}
}
and I am using following a PowerShell script to update the name under dataSources.
$JsonFilePath = "path-to-json-file"
$JsonData = Get-Content $JsonFilePath -raw | ConvertFrom-Json
$JsonData.createOrReplace.database.model.dataSources[0].name = "ssasservername"
$JsonData | ConvertTo-Json | set-content $JsonFilePath
Can anyone help me to set a value of an object inside the array?
$JsonData.createOrReplace.database.model.dataSources[0].name = "ssasservername"
Also note: Unexpected ConvertTo-Json results? Answer: it has a default -Depth of 2:
$JsonData | ConvertTo-Json -Depth 9 | set-content $JsonFilePath