JSON Powershell memory issue - json

I use a command to read a JSON file, this all works perfectly, until the file becomes large.
I currently have a JSON file of about 1.5GB. I read the file using Powershell using the following command:
get-content -Path C:\TEMP\largefile.json | out-string | ConvertFrom-Json
It returns the following error:
out-string : Exception of type 'System.OutOfMemoryException' was thrown.
+ ... oices = get-content -Path C:\TEMP\largefile.json | out-string | Conve ...
+ ~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Out-String], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.OutStringCommand
I've increased the memory as shown here:
get-item wsman:localhost\Shell\MaxMemoryPerShellMB
WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Shell
Type Name SourceOfValue Value
---- ---- ------------- -----
System.String MaxMemoryPerShellMB 8096
Any ideas on how to process this?
Edit (additions based on comments):
When I remove the out-string I get this error:
ConvertFrom-Json : Exception of type 'System.OutOfMemoryException' was thrown.
+ ... oices = get-content -Path C:\TEMP\largefile.json | ConvertFrom-Json ...
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Out-String], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.OutStringCommand
The Powershell version that I have is: 5.1.17763.1490
The file contains multiple columns regarding PDF files. These files are exported via an API into a JSON so it contains the file metadata such as owner and when it was created but also the actual PDF file in the column Body which later will be decoded to an actual PDF file. The structure is as followed:
[{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
]

Thank for the details.
For this issue I would try to convert each line separately and stream that through your process:
Get-Content C:\TEMP\largefile.json | ForEach-Object {
$_ = $_.Trim().TrimStart('[').TrimEnd(']')
if ($_) { $_ | ConvertFrom-Json }
}
As already suggested, I wouldn't be surprised if these memory issues wouldn't appear in PowerShell core. if possible, I recommend you to also give that a try.

Related

Azure extension error while calling PowerShell script

Using PowerShell, I am trying to encode text file to a string and encoded string is provided as input to get cook book name and recipe details. When I check the output cookbook name and Json file, they are string and valid json file. But when I use this code via Azure extension, I get error -
message:
ConvertFrom-Json : Unterminated string passed in.
FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand
#Encode the Input file
[string]$sStringToEncode=Get-Content "c:\temp\json.txt"
$sEncodedString=[Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($sStringToEncode))
write-host -ForegroundColor Green "Encoded String:`n" $sEncodedString
#Decode text
$EncodedText = $sEncodedString.replace("\n","")
write-host "String after replacing \n:" $EncodedText
$DecodedText = [System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String($EncodedText))
#Convert to Json file
$DecodedText | ConvertFrom-Json | ConvertTo-Json | Out-File c:\temp\DNAFiledata.json
#Taking out cookbook Name
$CookbookName = (Get-Content c:\temp\DNAFiledata.json | Out-String | ConvertFrom-Json).cookbook_name
$CookbookName | Out-File "c:\temp\3_CookBookName.txt"
#Taking out recipe details
$Datainput = (Get-Content -Raw -Path c:\temp\DNAFiledata.json | ConvertFrom-Json).chef_attributes | ConvertTo-Json
$Datainput | Out-File "c:\temp\4_Datainput.json"
$Datainput | Set-Content "c:\temp\$CookbookName.json"
Any suggestions on this please.

PowerShell failing to convert JSON to CSV

I have spent a bit of time researching how to convert Json to a CSV file using powershell but am failing to have it complete properly. Below is the syntax I have created:
$pathToOutputFile = "C:\OrderLinQ\TESTING\IN\CatalogsRetrieved\test.txt"
$pathToJsonFile = "C:\OrderLinQ\TESTING\IN\CatalogsRetrieved\test.json"
Get-Content -Path $pathToJsonFile |
ConvertFrom-Json |
ConvertTo-Csv -NoTypeInformation |
Set-Content $pathToOutputFile
However, when I attempt to execute this, I get an argument exception error:
ConvertFrom-Json : Invalid array passed in, ']' expected. (1): [
At line:1 char:69
+ Get-Content "C:\OrderLinQ\TESTING\IN\CatalogsRetrieved\test.json" | ConvertFrom- ...
+ ~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand
Any thoughts as to what I'm doing wrong? I've verified my file using JSONLint and it shows verified. Example file can be found at https://api.myjson.com/bins/mo59w. Any help is appreciated!
Try this. I've also attached the output
$pathToOutputFile = "D:\output.csv"
$pathToJsonFile = "D:\test.json"
$json = Get-Content -Path $pathToJsonFile
function ConvertFrom-JsonToCsv {
param(
[Parameter(ValueFromPipeline)]
$json
)
Process {
($json | ConvertFrom-Json) | ConvertTo-Csv -NoTypeInformation
}
}
ConvertFrom-JsonToCsv $json | Set-Content $pathToOutputFile
Output

merging two Jsons in powershell

I have the following two Objects , which have been got from 2 Json file using:
$Env = ConvertFrom-Json "$(get-content "C:\chef\environments.json")"
$Roles = ConvertFrom-Json "$(get-content "C:\chef\roles.json")"
Heres the out put after conversion :
PS C:\chef> $Env
run_list
--------
{recipe[djin_chef-max_any::default]}
PS C:\chef> $Roles
7-zip : #{home=%SYSTEMDRIVE%\7-zip}
cookbook_versions :
default : #{env=development}
modmon : #{env=dev}
paypal : #{artifact=%5BINTEGRATION%5D}
seven_zip : #{url=https://djcm-zip-local/djcm/chef}
task_sched : #{credentials=XN$q}
windows : #{password=K1N5}
I need to merge these two Json objects in powershell and I tried the following:
PS C:\chef> $Roles+$Env
Method invocation failed because [System.Management.Automation.PSObject] does not contain a method named 'op_Addition'.
At line:1 char:1
+ $Roles+$Env
+ ~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (op_Addition:String) [], RuntimeException
+ FullyQualifiedErrorId : MethodNotFound
Is there another elegant way of doing it if I am doing it wrong or why am I getting this error ?
$Env only has one property, so you could add a new member to $Roles:
$Roles | Add-Member -NotepropertyName run_list -NotePropertyValue $Env.run_list
This syntax work in PowerShell v3, but you listed v2 and v2 in your tags.. so for v2:
$Roles | Add-Member -MemberType NoteProperty -Name run_list -Value $Env.run_list

Issue with get shortcut function not loading in separate script

So I have the following script I am trying to run which keeps erroring out
. .\stshortcut.ps1 |
Get-Shortcut . |
Where Target -eq "cmd.exe" |
%{$myPath, $myNewName = $null;
Write-Warning "Processing $($_.Link)";
If (-Not (Test-Path .\BadShortcuts -PathType Container)) {New-Item -
WhatIf -ItemType Directory BadShortcuts | Out-Null};
[string]$myPath = $_.Arguments.Split()[-1] -replace '"';
[string]$myNewName = $_.Link -replace "\.lnk$";
Rename-Item -WhatIf -Force -Path $myPath -NewName $myNewName;
(Get-Item -Force $myNewName).Attributes = '';
Move-Item -WhatIf $_.LinkPath .\BadShortcuts;}`
the error I get is as follows
The term 'Get-Shortcut' is not recognized as the name of a cmdlet,
function, script file, or operable program. Check the spelling of the
name, or if a path was included, verify that the path is correct and try
again.
At C:\Shared\APPS\FixShortcutX2.ps1:1 Char 13
+ Get Shortcut <<<<< . |
+ CategoryInfor : ObjectNotFound: (Get-Shortcut:String)[],
CommandNotFoundException
+ FullyQualifiedErrorID : CommandNotFoundException
the stshortcut.ps1 script has the get-shortcut and set-shortcut functions and are called to do such - I got this script from
https://www.reddit.com/r/PowerShell/comments/4su2jg/zeroday_malware_renamed_folders_on_a_shared_drive/
which is an answer script to fix a macro virus from a word doc attachment - sent from a spoofed email address -
Any assistance is GREATLY appreciated
EDITx2 after some further helpful advice and editing I now am receiving the following
Where-Object : Cannot bind parameter 'FilterScript' . Cannot convert the "Target" value of type "System.String" to type "System.Management.Automation.ScriptBlock".
At C:\Shared\Apps\FixShortcutX2.ps1:3 char:6
+ Where <<<<< Target -eq "cmd.exe" |
+CategoryInfo : InvalidArgument: (:) [Where-Object], ParameterBindingException
+ FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.WhereObjectCommand
You need to dot-source stshortcut.ps1 before you can use Get-Shortcut.
Change the first line:
.\stshortcut.ps1 |
To
. .\stshortcut.ps1

"Invalid array passed" when parsing JSON

I have this file which I want to read with PowerShell:
var myMap =
[
{
"name": "JSON Example",
"attr": "Another attribute"
}
]
My PowerShell v3 Code:
$str = Get-Content $file | Select -Skip 1;
$str | ConvertFrom-Json;
But I'm always getting this error:
ConvertFrom-Json : Invalid array passed in, ']' expected. (1): [
At S:\ome\Path\script.ps1:60 char:8
+ $str | ConvertFrom-Json;
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand
If I copy and paste the JSON code manually into the code, everything is working fine:
'[
{
"name": "JSON Example",
"attr": "Another attribute"
}
]' | ConvertFrom-Json;
Try to pipe to Out-String before piping to ConvertFrom-Json:
Get-Content $file | Select -Skip 1 | Out-String | ConvertFrom-Json
In your working example the JSON code is a string while the non-working example returns a collection of lines. Piping to Out-String converts the collection to a single string, which is what the InputObject parameter accept.
Alternatively you can use Get-Content -Raw which will retrieve the JSON as a single string.
See this post for more info: http://blogs.technet.com/b/heyscriptingguy/archive/2014/04/23/json-is-the-new-xml.aspx
The var myMap = isn't json, it's javascript. Delete the first line and it will be fine.
EDIT:
Oh, you are skipping the first line. It may be that there's a carriage return missing in the last line of the file, and Powershell 3 is more sensitive to it. It works fine in Powershell 5.1 even without a carriage return at the end.
another answer that also works: use GC -raw <FILE> which will pass in as string