I have a JSON file that I would need a certain value to be incremented. Here is the JSON file structure I have.
{
"Body": {
"Content": {
},
"Lines": [{
"LineNumber": "1",
"DateOfService": "10/20/2017"
},
{
"LineNumber": "1",
"DateOfService": "10/20/2017"
},
{
"LineNumber": "1",
"DateOfService": "10/20/2017"
}]
}
}
I want the "LineNumber" values to be set from 1 and incremented by 1 subsequently. I have close to a 1000 lines.
Thanks for your assistance.
$InputFile = "path\name.json.txt"
$OutPutFile = "path\name.new.json"
$objJsonFile = Get-Content -Raw -Path $InputFile | ConvertFrom-Json
$intLine = 1
foreach ($itmLine in $objJsonFile.Claim.Lines)
{
$itmLine.LineNumber = [string]$intLine++
}
$objJsonFile | ConvertTo-Json -Depth 3 | Out-File $OutPutFile
You could try to do so using the following code.
$fileContent = Get-Content PATH\TO\FILE.json;
$json = $fileContent | ConvertFrom-Json;
$counter = 1;
ForEach($line in $json.Body.Lines) {
$line.LineNumber = $counter;
$counter++;
}
# write json back to file!?
Related
I have the following Json file:
{
"Id":1,
"Name":"john",
"Addresses":[
{
"Id":1,
"Street":"1st Street",
"City":"Riyadh"
},
{
"Id":2,
"Street":"2nd Street",
"City":"Dammam"
}
]
}
I want to remove the second address in the array using powershell.
I tried the following:
$filePath = 'C:\temp\Settings.json'
$settings = $filePath | ConvertFrom-Json
foreach($item in $settings.Addresses)
{
if($item.Id -eq 2)
{
$settings.Addresses.Remove($item)
}
}
Any ideas?
The following commented code snippet could help:
$filePath = 'C:\temp\Settings.json'
$settings = Get-Content -Path $filePath | ConvertFrom-Json
# $settings.Addresses # is an array of fixed size
# $settings.Addresses.IsFixedSize # returns True
# $settings.Addresses.Remove($item) # isn't possible; hence, let's build new array:
$NewAddresses = [System.Collections.ArrayList]::new()
foreach($item in $settings.Addresses)
{
if ($item.Id -ne 2)
{
[void]$NewAddresses.Add( $item )
}
}
# and replace old one:
$settings.Addresses = $NewAddresses
$settings | ConvertTo-Json ### | Out-File -FilePath $filePath -Force -Encoding utf8
Output: .\SO\66855002.ps1
{
"Id": 1,
"Name": "john",
"Addresses": [
{
"Id": 1,
"Street": "1st Street",
"City": "Riyadh"
}
]
}
I would like to add an additional key with value into my existing JSON file. Unfortunately I'm not able. Here an short overview:
My JSON-File before powershell script is run:
[
{
"id": "1",
"description": [
{
"country": "Brazil"
},
{
"country": "Mexico"
}
]
},
{
"id": "2",
"description": [
{
"country": "Argentina"
}
]
}
]
My wish, how the JSON-File should look like, after my powershell script is run:
[
{
"id": "1",
"description": [
{
"country": "Brazil",
"city": "Rio de Janeiro"
},
{
"country": "Mexico",
"city": "Mexico City"
}
]
},
{
"id": "2",
"description": [
{
"country": "Argentina",
"city": "Buenos Aires"
}
]
}
]
My powershell script:
function GetCity($country) {
$x = "not available"
If ( $country -eq "Brazil" ) { $x = "Rio de Janeiro" }
If ( $country -eq "Mexico" ) { $x = "Mexico City" }
If ( $country -eq "Argentina" ) { $x = "Buenos Aires" }
return $x
}
# Source the JSON content
$jsonFile = 'C:\Temp\test.json'
$jsonContent = Get-Content -Path $jsonFile
# Convert JSON to PSObjects
$jsonAsPsObjects = $jsonContent | ConvertFrom-Json
foreach ($info in $jsonAsPsObjects) {
$result = GetCity($info.description.country)
jsonContent | Add-Member -Type NoteProperty -Name "City" -Value $result
}
# Save JSON back to file
$json | ConvertTo-Json | Set-Content $jsonFile
Error:
jsonContent : The term 'jsonContent' is not recognized as the name of
a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path
is correct and try again.
How can I solve this issue?
There at two problems:
jsonContent should be $jsonContent in statement jsonContent | Add-Member ...
You're neglecting to loop over the array elements of the description property, to each of which a city property is to be added.
I suggest streamlining your code as follows:
function Get-City {
param([string] $country)
# Use a `switch` statement:
switch ($country) {
'Brazil' { return 'Rio de Janeiro' }
'Mexico' { return 'Mexico City' }
'Argentina' { return 'Buenos Aires' }
default { return 'not available' }
}
}
$jsonFile = 'C:\Temp\test.json'
(Get-Content -Raw $jsonFile | ConvertFrom-Json) | ForEach-Object {
# Add a 'city' property to each object in the 'description' property.
$_.description.ForEach({
Add-Member -InputObject $_ city (Get-City $_.country)
})
$_ # output the modified object
} | ConvertTo-Json -Depth 3 # | Set-Content $jsonFile
I have a json file with below contents
{
"createOrReplace": {
"object": {
"database": "DB_NAME"
},
"database": {
"name": "DB_NAME",
"compatibilityLevel": 1400,
"model": {
"culture": "en-IN",
"dataSources": [{
"name": "somename",
"connectionString": "somevalue",
"impersonationMode": "impersonateServiceAccount",
"annotations": [{
"name": "ConnectionEditUISource",
"value": "SqlServer"
}]
}]
}
}
}
}
and I am using following a PowerShell script to update the name under dataSources.
$JsonFilePath = "path-to-json-file"
$JsonData = Get-Content $JsonFilePath -raw | ConvertFrom-Json
$JsonData.createOrReplace.database.model.dataSources[0].name = "ssasservername"
$JsonData | ConvertTo-Json | set-content $JsonFilePath
Can anyone help me to set a value of an object inside the array?
$JsonData.createOrReplace.database.model.dataSources[0].name = "ssasservername"
Also note: Unexpected ConvertTo-Json results? Answer: it has a default -Depth of 2:
$JsonData | ConvertTo-Json -Depth 9 | set-content $JsonFilePath
I have a very large JSON Lines File with 4.000.000 Rows, and I need to convert several events from every row. The resulted CSV File contains 15.000.000 rows. How can I optimize this script?
I'm using Powershell core 7 and it takes around 50 hours to complete the conversion.
My Powershell script:
$stopwatch = [system.diagnostics.stopwatch]::StartNew()
$totalrows = 4000000
$encoding = [System.Text.Encoding]::UTF8
$i = 0
$ig = 0
$output = #()
$Importfile = "C:\file.jsonl"
$Exportfile = "C:\file.csv"
if (test-path $Exportfile) {
Remove-Item -path $Exportfile
}
foreach ($line in [System.IO.File]::ReadLines($Importfile, $encoding)) {
$json = $line | ConvertFrom-Json
foreach ($item in $json.events.items) {
$CSVLine = [pscustomobject]#{
Key = $json.Register.Key
CompanyID = $json.id
Eventtype = $item.type
Eventdate = $item.date
Eventdescription = $item.description
}
$output += $CSVLine
}
$i++
$ig++
if ($i -ge 30000) {
$output | Export-Csv -Path $Exportfile -NoTypeInformation -Delimiter ";" -Encoding UTF8 -Append
$i = 0
$output = #()
$minutes = $stopwatch.elapsed.TotalMinutes
$percentage = $ig / $totalrows * 100
$totalestimatedtime = $minutes * (100/$percentage)
$timeremaining = $totalestimatedtime - $minutes
Write-Host "Events: Total minutes passed: $minutes. Total minutes remaining: $timeremaining. Percentage: $percentage"
}
}
$output | Export-Csv -Path $Exportfile -NoTypeInformation -Delimiter ";" -Encoding UTF8 -Append
Write-Output $ig
$stopwatch.Stop()
Here is the structure of the JSON.
{
"id": "111111111",
"name": {
"name": "Test Company GmbH",
"legalForm": "GmbH"
},
"address": {
"street": "Berlinstr.",
"postalCode": "11111",
"city": "Berlin"
},
"status": "liquidation",
"events": {
"items": [{
"type": "Liquidation",
"date": "2001-01-01",
"description": "Liquidation"
}, {
"type": "NewCompany",
"date": "2000-01-01",
"description": "Neueintragung"
}, {
"type": "ControlChange",
"date": "2002-01-01",
"description": "Tested Company GmbH"
}]
},
"relatedCompanies": {
"items": [{
"company": {
"id": "2222222",
"name": {
"name": "Test GmbH",
"legalForm": "GmbH"
},
"address": {
"city": "Berlin",
"country": "DE",
"formattedValue": "Berlin, Deutschland"
},
"status": "active"
},
"roles": [{
"date": "2002-01-01",
"name": "Komplementär",
"type": "Komplementaer",
"demotion": true,
"group": "Control",
"dir": "Source"
}, {
"date": "2001-01-01",
"name": "Komplementär",
"type": "Komplementaer",
"group": "Control",
"dir": "Source"
}]
}, {
"company": {
"id": "33333",
"name": {
"name": "Test2 GmbH",
"legalForm": "GmbH"
},
"address": {
"city": "Berlin",
"country": "DE",
"formattedValue": "Berlin, Deutschland"
},
"status": "active"
},
"roles": [{
"date": "2002-01-01",
"name": "Komplementär",
"type": "Komplementaer",
"demotion": true,
"group": "Control",
"dir": "Source"
}, {
"date": "2001-01-01",
"name": "Komplementär",
"type": "Komplementaer",
"group": "Control",
"dir": "Source"
}]
}]
}
}
as per comment: Try to avoid using the increase assignment operator (+=) to create a collection.
Use the PowerShell pipeline instead, e.g.:
$stopwatch = [system.diagnostics.stopwatch]::StartNew()
$totalrows = 4000000
$encoding = [System.Text.Encoding]::UTF8
$i = 0
$ig = 0
$Importfile = "C:\file.jsonl"
$Exportfile = "C:\file.csv"
if (test-path $Exportfile) {
Remove-Item -path $Exportfile
}
Get-Content $Importfile -Encoding $encoding | Foreach-Object {
$json = $_ | ConvertFrom-Json
$json | ConvertFrom-Json | Foreach-Object {
[pscustomobject]#{
Key = $json.Register.Key
CompanyID = $json.id
Eventtype = $_.type
Eventdate = $_.date
Eventdescription = $_.description
}
}
$i++
$ig++
if ($i -ge 30000) {
$i = 0
$minutes = $stopwatch.elapsed.TotalMinutes
$percentage = $ig / $totalrows * 100
$totalestimatedtime = $minutes * (100/$percentage)
$timeremaining = $totalestimatedtime - $minutes
Write-Host "Events: Total minutes passed: $minutes. Total minutes remaining: $timeremaining. Percentage: $percentage"
}
} | Export-Csv -Path $Exportfile -NoTypeInformation -Delimiter ";" -Encoding UTF8 -Append
Write-Output $ig
$stopwatch.Stop()
Update 2020-05-07
Based on the comments and extra info the question, I have written a small reusable cmdlet that uses the PowerShell Pipeline to read through the .jsonl (Json Lines) file. It collects each line till it find a closing '}' character then it checks for a valid json string (using Test-Json as there might embedded objects. If it is valid it intermediately release the extract object in the pipeline and start collecting lines again:
Function ConvertFrom-JsonLines {
[CmdletBinding()][OutputType([Object[]])]Param (
[Parameter(ValueFromPipeLine = $True, Mandatory = $True)][String]$Line
)
Begin { $JsonLines = [System.Collections.Generic.List[String]]#() }
Process {
$JsonLines.Add($Line)
If ( $Line.Trim().EndsWith('}') ) {
$Json = $JsonLines -Join [Environment]::NewLine
If ( Test-Json $Json -ErrorAction SilentlyContinue ) {
$Json | ConvertFrom-Json
$JsonLines.Clear()
}
}
}
}
You can use it like this:
Get-Content .\file.jsonl | ConvertFrom-JsonLines | ForEach-Object { $_.events.items } |
Export-Csv -Path $Exportfile -NoTypeInformation -Encoding UTF8
I am able to make it ~40 % faster by making two small changes: 1. use Get-Content -ReadCount and unpack the buffered lines and 2. change the pipeline to 'flow' more by avoiding the $json=+foreach part.
$stopwatch = [system.diagnostics.stopwatch]::StartNew()
$totalrows = 4000000
$encoding = [System.Text.Encoding]::UTF8
$i = 0
$ig = 0
$Importfile = "$psscriptroot\input2.jsonl"
$Exportfile = "$psscriptroot\output.csv"
if (Test-Path $Exportfile) {
Remove-Item -Path $Exportfile
}
# Changed the next few lines
Get-Content $Importfile -Encoding $encoding -ReadCount 10000 |
ForEach-Object {
$_
} | ConvertFrom-Json | ForEach-Object {
$json = $_
$json.events.items | ForEach-Object {
[pscustomobject]#{
Key = $json.Register.Key
CompanyID = $json.id
Eventtype = $_.type
Eventdate = $_.date
Eventdescription = $_.description
}
}
$i++
$ig++
if ($i -ge 10000) {
$i = 0
$minutes = $stopwatch.elapsed.TotalMinutes
$percentage = $ig / $totalrows * 100
$totalestimatedtime = $minutes * (100 / $percentage)
$timeremaining = $totalestimatedtime - $minutes
Write-Host "Events: Total minutes passed: $minutes. Total minutes remaining: $timeremaining. Percentage: $percentage"
}
} | Export-Csv -Path $Exportfile -NoTypeInformation -Delimiter ';' -Encoding UTF8 -Append
Write-Output $ig
$stopwatch.Stop()
Very beginner question. I'm trying to get certain values from JSON with Powershell.. Specifically, I want to list the Services: TEST00000 and FAKE only.
When I run the script below, I get this:
TEST00000 FAKE
--------- ----
#{Enabled=True; Processed=2; Sent=3; Failed=4; Downloaded=5} #{Enabled=True}
How can I get a list of only the services?
More importantly, how do I list only the services which has the key/value Enabled=True present inside of them?
Here is the code:
$JSON = '{
"Envs": {
"DEV": {
"Services": {
"TEST00000": {
"Enabled": true,
"Processed": 2,
"Sent": 3,
"Failed": 4,
"Downloaded": 5
},
"FAKE": {
"Enabled": true
}
}
}
},
"Component": {
"Digger": {
"Envs": {
"DEV": {
"DownloadE": 4
}
}
}
}
}'
$jsonobj = ConvertFrom-Json -inputObject $JSON
$jsonobj.Envs.DEV.Services
To get the name of each Services property. You can use Get-Member like user2734259 did or you can use the psobject property which stores useful information about an object.
$ServiceNames = $jsonobj.Envs.DEV.Services.psobject.properties.name
Once you have the names you can loop over them and filter on the sub-property Enabled
$jsonobj.Envs.DEV.Services.psobject.properties.name | ForEach-Object {
$_ | Where-Object {$jsonobj.Envs.DEV.Services.$_.Enabled -eq $True}
}
This should allow you to get the names dynamically, regardless of the service name in your JSON:
$JSON = '{
"Envs": {
"DEV": {
"Services": {
"TEST00000": {
"Enabled": true,
"Processed": 2,
"Sent": 3,
"Failed": 4,
"Downloaded": 5
},
"FAKE": {
"Enabled": true
}
}
}
},
"Component": {
"Digger": {
"Envs": {
"DEV": {
"DownloadE": 4
}
}
}
}
}'
$jsonobj = ConvertFrom-Json -inputObject $JSON
$enabledServices = $NULL
$disabledServices = $NULL
# Since the service names are arbitrary
# create an object that contains only
# objects whose MemberType is NoteProperty
$strServiceNames = #($($jsonobj.Envs.DEV.Services | Get-Member | Where { $_.MemberType -eq "NoteProperty" } | Select Name).Name)
$pscoServiceNames = [PSCustomObject]#{Names=$strServiceNames}
foreach($serviceName in $pscoserviceNames.Names)
{
# dynamically access the service name
# from $jsonobj.Envs.DEV.Services
$serviceStatus = $jsonobj.Envs.DEV.Services.$serviceName.Enabled
# parse results based on value of .Enabled
if($serviceStatus.ToString() -eq "True")
{
$enabledServices = [Array]$enabledServices + [PSCustomObject]#{Name = $serviceName;Enabled = $serviceStatus}
}
else
{
$disabledServices = [Array]$disabledServices + [PSCustomObject]#{Name = $serviceName;Enabled = $serviceStatus}
}
}
# Show the results
Write-Output "`nEnabled Services`n"
$enabledServices | Format-List
Write-Output "`nDisabled Services`n"
$disabledServices | Format-List
If anything is unclear let me know and I can explain in more detail
Hope this helps you out.
Happy 'shelling!