I have this file which I want to read with PowerShell:
var myMap =
[
{
"name": "JSON Example",
"attr": "Another attribute"
}
]
My PowerShell v3 Code:
$str = Get-Content $file | Select -Skip 1;
$str | ConvertFrom-Json;
But I'm always getting this error:
ConvertFrom-Json : Invalid array passed in, ']' expected. (1): [
At S:\ome\Path\script.ps1:60 char:8
+ $str | ConvertFrom-Json;
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand
If I copy and paste the JSON code manually into the code, everything is working fine:
'[
{
"name": "JSON Example",
"attr": "Another attribute"
}
]' | ConvertFrom-Json;
Try to pipe to Out-String before piping to ConvertFrom-Json:
Get-Content $file | Select -Skip 1 | Out-String | ConvertFrom-Json
In your working example the JSON code is a string while the non-working example returns a collection of lines. Piping to Out-String converts the collection to a single string, which is what the InputObject parameter accept.
Alternatively you can use Get-Content -Raw which will retrieve the JSON as a single string.
See this post for more info: http://blogs.technet.com/b/heyscriptingguy/archive/2014/04/23/json-is-the-new-xml.aspx
The var myMap = isn't json, it's javascript. Delete the first line and it will be fine.
EDIT:
Oh, you are skipping the first line. It may be that there's a carriage return missing in the last line of the file, and Powershell 3 is more sensitive to it. It works fine in Powershell 5.1 even without a carriage return at the end.
another answer that also works: use GC -raw <FILE> which will pass in as string
Related
i want to send an event to aws via a cli command in a powershell script. Here is the Json i need to send to the eventbridge:
[
{
"Sensor":"{\"id\":\"880/2021-04-13\",\"attributes\":\"green\",\"Name\":\"SensorGreen\",\"state\":\"SUCCEEDED\"}",
"Source":"google.com"
}
]
Thats what a tried in powershell:
$json='[{"Sensor":"{\"id\":\"880/2021-04-13\",\"attributes\":\"green\",\"Name\":\"SensorGreen\",\"state\":\"SUCCEEDED\"}","Source":"google.com"}]'|ConvertTo-Json -Compress
aws events put-events --entries $json --region "eu-central-1"
That does not work. I even tried to write it to a json file and read it and send it from the file but it doesnt work. It somehow leads to too many "\" slashes or no slashes for my "Sensor" where it is necessary. I even tried to create a object and just convert the Sensor object to Json and then the whole object again to have the escaping, but it is not working.
EDIT:
i tried also this as mentioned in the answer:
$RequestObject = [pscustomobject] #(#{
Sensor = [pscustomobject] #{
id = "880/2021-04-13"
attribute = "green"
Name = "SensorGreen"
state = "SUCCEEDED"
}
Source = "google.com"
})
$RequestObject.Get(0).Sensor=$RequestObject.Get(0).Sensor | ConvertTo-Json -Compress
$Json = $RequestObject | ConvertTo-Json -Compress
aws events put-events --entries $Json --region "eu-central-1"
i included the #() to have an array and converted the sensor object twice to json to include the slashes. but the result is:
Error parsing parameter '--entries': Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
if i use the command line interface only with the aws command and put the object directly into the command, then it works.. but the powershell does not.
You're passing a JSON string to an external program (aws)
Irrespective of how you constructed that string - directly as a string, or from an object / hashtable via ConvertTo-Json - as of PowerShell 7.1 - manual escaping of embedded " as \" is required, which, in turn requires escaping the preexisting \ preceding the embedded " as \\.
Note: None of this should be necessary, but due to a long-standing bug in PowerShell is - see this answer for details.
PowerShell Core 7.2.0-preview.5 now includes experimental feature PSNativeCommandArgumentPassing with an attempted fix, but, unfortunately, it looks like it will lack important accommodations for high-profile CLIs on Windows - see this summary from GitHub issue #15143. However, it would fix calls to aws.exe, the Amazon Web Services CLI.
# The JSON string to be passed as-is.
$json = #'
[
{
"Sensor":"{\"id\":\"880/2021-04-13\",\"attributes\":\"green\",\"Name\":\"SensorGreen\",\"state\":\"SUCCEEDED\"}",
"Source":"google.com"
}
]
'#
# Up to at least PowerShell 7.1:
# Manually perform the required escaping, to work around PowerShell's
# broken argument-passing to external programs.
# Note:
# -replace '\\', '\\' *looks* like a no-op, but replaces each '\' with '\\'
$jsonEscaped = $json -replace '\\', '\\' -replace '"', '\"'
# Now pass the *escaped* JSON string to the aws CLI:
aws events put-events --entries $jsonEscaped --region "eu-central-1"
ConvertTo-Json is for converting objects in PowerShell, not a string that you have tried to already write in Json. Your $Json variable produces this.
"[{\"Sensor\":\"{\\\"id\\\":\\\"880/2021-04-13\\\",\\\"attributes\\\":\\\"green\\\",\\\"Name\\\":\\\"SensorGreen\\\",\\\"state\\\":\\\"SUCCEEDED\\\"}\",\"Source\":\"google.com\"}]"
If you want to create the object in PowerShell and convert it to Json, then you can do this.
$RequestObject = [pscustomobject] #{
Sensor = [pscustomobject] #{
id = "880/2021-04-13"
attribute = "green"
Name = "SensorGreen"
state = "SUCCEEDED"
}
Source = "google.com"
}
$Json = $RequestObject | ConvertTo-Json -Compress
aws events put-events --entries $Json --region "eu-central-1"
Your Json will look like this if you print your variable out.
{"Sensor":{"id":"880/2021-04-13","attribute":"green","Name":"SensorGreen","state":"SUCCEEDED"},"Source":"google.com"}
Which I think is like the Json that the command is expecting. Not entirely sure why you need the strings escaping or the array. Here it is uncompressed.
{
"Sensor": {
"id": "880/2021-04-13",
"attribute": "green",
"Name": "SensorGreen",
"state": "SUCCEEDED"
},
"Source": "google.com"
}
Just noticed the powershell-2.0 tag. If you are using it, then you should do this instead to create your Json.
$Sensor = New-Object psobject -Property #{
id = "880/2021-04-13"
attribute = "green"
Name = "SensorGreen"
state = "SUCCEEDED"
}
$RequestObject = New-Object psobject -Property #{
Sensor = $Sensor
Source = "google.com"
}
$Json = $RequestObject | ConvertTo-Json -Compress
EDIT
If you absolutely must escape the strings in that way and have a single item array, then you should just pass the Json that you have written in your answer without any further conversion.
$json='[{"Sensor":"{\"id\":\"880/2021-04-13\",\"attributes\":\"green\",\"Name\":\"SensorGreen\",\"state\":\"SUCCEEDED\"}","Source":"google.com"}]'
If you want to make PowerShell do that for you then you would need to perform some string replacement on the Sensor object first.
PowerShell 2.0
$Sensor = New-Object psobject -Property #{
id = "880/2021-04-13"
attribute = "green"
Name = "SensorGreen"
state = "SUCCEEDED"
}
$SensorJson = $Sensor | ConvertTo-Json -Compress
$SensorJson.Replace("`"","\`"")
$RequestObject = New-Object psobject -Property #{
Sensor = $SensorJson
Source = "google.com"
}
$Json = $RequestObject | ConvertTo-Json -Compress
PowerShell 3.0+
$Sensor = [pscustomobject] #{
id = "880/2021-04-13"
attribute = "green"
Name = "SensorGreen"
state = "SUCCEEDED"
}
$SensorJson = $Sensor | ConvertTo-Json -Compress
$SensorJson.Replace("`"","\`"")
$RequestObject = [pscustomobject] #{
Sensor = $SensorJson
Source = "google.com"
}
$Json = $RequestObject | ConvertTo-Json -Compress
Then your AWS command
# Add the array around the compressed Json string.
aws events put-events --entries "[$Json]" --region "eu-central-1"
"[$Json]" prints
[{"Sensor":"{\"id\":\"880/2021-04-13\",\"attribute\":\"green\",\"Name\":\"SensorGreen\",\"state\":\"SUCCEEDED\"}","Source":"google.com"}]
I use a command to read a JSON file, this all works perfectly, until the file becomes large.
I currently have a JSON file of about 1.5GB. I read the file using Powershell using the following command:
get-content -Path C:\TEMP\largefile.json | out-string | ConvertFrom-Json
It returns the following error:
out-string : Exception of type 'System.OutOfMemoryException' was thrown.
+ ... oices = get-content -Path C:\TEMP\largefile.json | out-string | Conve ...
+ ~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Out-String], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.OutStringCommand
I've increased the memory as shown here:
get-item wsman:localhost\Shell\MaxMemoryPerShellMB
WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Shell
Type Name SourceOfValue Value
---- ---- ------------- -----
System.String MaxMemoryPerShellMB 8096
Any ideas on how to process this?
Edit (additions based on comments):
When I remove the out-string I get this error:
ConvertFrom-Json : Exception of type 'System.OutOfMemoryException' was thrown.
+ ... oices = get-content -Path C:\TEMP\largefile.json | ConvertFrom-Json ...
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Out-String], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.OutStringCommand
The Powershell version that I have is: 5.1.17763.1490
The file contains multiple columns regarding PDF files. These files are exported via an API into a JSON so it contains the file metadata such as owner and when it was created but also the actual PDF file in the column Body which later will be decoded to an actual PDF file. The structure is as followed:
[{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
{"Id":"ID","ParentId":"parent","Name":"filename","OwnerId":"owner","CreatedDate":"date","Body":"*******"}
]
Thank for the details.
For this issue I would try to convert each line separately and stream that through your process:
Get-Content C:\TEMP\largefile.json | ForEach-Object {
$_ = $_.Trim().TrimStart('[').TrimEnd(']')
if ($_) { $_ | ConvertFrom-Json }
}
As already suggested, I wouldn't be surprised if these memory issues wouldn't appear in PowerShell core. if possible, I recommend you to also give that a try.
How do I modify a Key Value Pair in a JSON File with powershell?
We are trying to modify Database Connection, sometimes it can be two levels nested deep, sometimes it can be three levels deep.
Trying to utilize this answer,
Currently we are switching servers in multiple json files, so we can test in different server environments.
Add new key value pair to JSON file in powershell.
"JWTToken": {
"SecretKey": "Security Key For Generate Token",
"Issuer": "ABC Company"
},
"AllowedHosts": "*",
"ModulesConfiguration": {
"AppModules": [ "ABC Modules" ]
},
"ConnectionStrings": {
"DatabaseConnection": "Server=testserver,1433;Database=TestDatabase;User Id=code-developer;password=xyz;Trusted_Connection=False;MultipleActiveResultSets=true;",
"TableStorageConnection": "etc",
"BlobStorageConnection": "etc"
},
Once you convert JSON string to an object with PowerShell, it's not really a problem to then change the properties. The main issue you are going to face here is that your string is currently invalid JSON for .Net or at least it won't be expecting it in the current format. We can fix that though.
Here is your current JSON.
"JWTToken": {
"SecretKey": "Security Key For Generate Token",
"Issuer": "ABC Company"
},
"AllowedHosts": "*",
"ModulesConfiguration": {
"AppModules": [ "ABC Modules" ]
},
"ConnectionStrings": {
"DatabaseConnection": "Server=testserver,1433;Database=TestDatabase;User Id=code-developer;password=xyz;Trusted_Connection=False;MultipleActiveResultSets=true;",
"TableStorageConnection": "etc",
"BlobStorageConnection": "etc"
},
There may be other issues, for PowerShell JSON, in your application.config file, but these two are immediately noticeable to me.
Unnecessary trailing commas
No definitive opening { and closing }
How Can We Fix This?
We can use simple string concatenation to add { and } where necessary.
$RawText = Get-Content -Path .\path_to\application.config -Raw
$RawText = "{ " + $RawText + " }"
To remove any unnecessary parsing issues with trailing commas when parsing the JSON with ConvertFrom-Json we need to remove them via regex. My proposed approach would be to identify them by whether the current array } or ] closes after them, it might be that these closing brackets have a number of spaces or \s before they appear. So we would have a regex that looks like this:
"\,(?=\s*?[\}\]])".
We could then use that with -replace in PowerShell. Of course we will replace them with an empty string.
$FormattedText = $RawText -replace "\,(?=\s*?[\}\]])",""
From here we convert to JSON.
$JsonObj = $FormattedText | ConvertFrom-Json
We can now change your database string by setting a property.
$JsonObj.ConnectionStrings.DatabaseConnection = "your new string"
We use ConvertTo-Json to convert the array back to a Json string.
$JsonString = $JsonObj | ConvertTo-Json
It's not important to return the trailing commas, they aren't valid JSON, but your file needs the first { and last } removing before we commit it back to file with Set-Content.
# Remove the first { and trim white space. Second TrimStart() clears the space.
$JsonString = $JsonString.TrimStart("{").TrimStart()
# Repeat this but for the final } and use TrimEnd().
$JsonString = $JsonString.TrimEnd("}").TrimEnd()
# Write back to file.
$JsonString | Set-Content -Path .\path_to\application.config -Force
Your config file should be written back more or less as you found it. I will try and think of a regex to fix the appearance of the formatting, it shouldn't error, it just doesn't look great. Hope that helps.
EDIT
Here is a function to fix the unsightly appearance of the text in the file.
function Restore-Formatting {
Param (
[parameter(Mandatory=$true,ValueFromPipeline=$true)][string]$InputObject
)
$JsonArray = $InputObject -split "\n"
$Tab = 0
$Output = #()
foreach ($Line in $JsonArray) {
if ($Line -match "{" -or $Line -match "\[") {
$Output += (" " * $Tab) + $Line.TrimStart()
$Tab += 4
}
elseif ($Line -match "^\s+}" -or $Line -match "^\s+\]") {
$Tab -= 4
$Output += (" " * $Tab) + $Line.TrimStart()
}
else {
$Output += (" " * $Tab) + $Line.TrimStart()
}
}
$Output
}
TL;DR Script:
$RawText = Get-Content -Path .\path_to\application.config -Raw
$RawText = "{ " + $RawText + " }"
$FormattedText = $RawText -replace "\,(?=\s*?[\}\]])",""
$JsonObj = $FormattedText | ConvertFrom-Json
$JsonObj.ConnectionStrings.DatabaseConnection = "your new string"
$JsonString = $JsonObj | ConvertTo-Json
$JsonString = $JsonString.TrimStart("{").TrimStart()
$JsonString = $JsonString.TrimEnd("}").TrimEnd()
$JsonString | Restore-Formatting | Set-Content -Path .\path_to\application.config -NoNewLine -Force
I have spent a bit of time researching how to convert Json to a CSV file using powershell but am failing to have it complete properly. Below is the syntax I have created:
$pathToOutputFile = "C:\OrderLinQ\TESTING\IN\CatalogsRetrieved\test.txt"
$pathToJsonFile = "C:\OrderLinQ\TESTING\IN\CatalogsRetrieved\test.json"
Get-Content -Path $pathToJsonFile |
ConvertFrom-Json |
ConvertTo-Csv -NoTypeInformation |
Set-Content $pathToOutputFile
However, when I attempt to execute this, I get an argument exception error:
ConvertFrom-Json : Invalid array passed in, ']' expected. (1): [
At line:1 char:69
+ Get-Content "C:\OrderLinQ\TESTING\IN\CatalogsRetrieved\test.json" | ConvertFrom- ...
+ ~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand
Any thoughts as to what I'm doing wrong? I've verified my file using JSONLint and it shows verified. Example file can be found at https://api.myjson.com/bins/mo59w. Any help is appreciated!
Try this. I've also attached the output
$pathToOutputFile = "D:\output.csv"
$pathToJsonFile = "D:\test.json"
$json = Get-Content -Path $pathToJsonFile
function ConvertFrom-JsonToCsv {
param(
[Parameter(ValueFromPipeline)]
$json
)
Process {
($json | ConvertFrom-Json) | ConvertTo-Csv -NoTypeInformation
}
}
ConvertFrom-JsonToCsv $json | Set-Content $pathToOutputFile
Output
I've in the middle of writing a translation of a working Slack script from Bash to Powershell, and I'm stumbled on what is literally the meat of the problem; how to use Invoke-WebRequest to replace curl.
This is the curl command that works successfully in Bash on *nix systems:
curl \
-X POST \
-H "Content-type: application/json" \
--data "{
\"attachments\":
[
{
\"mrkdwn_in\": [\"text\"],
\"color\": \"$COLOUR\",
\"text\": \"$MESSAGE\",
}
]
}" \
https://hooks.slack.com/services/tokenpart1/tokenpart2
Note the $COLOUR and $MESSAGE variables are derived from elsewhere in the script (not the section I'm having trouble with).
I can't get it to work in PowerShell. My translation so far is:
$Body = #{
"attachments" = "[
{
"mrkdwn_in": ["text"],
"color": "$Colour",
"text": "$Message",
]"
}
$BodyJSON = $Body |convertto-JSON
Invoke-WebRequest -Headers -Method Post -Body "$BodyJSON" -Uri "https://hooks.slack.com/services/tokenpart1/tokenpart2" -ContentType application/json
This results in the following error:
At C:\path-to-file-etc.ps1:53 char:11
+ "mrkdwn_in": ["text"],
+ ~~~~~~~~~~~~~~~~~~~~~
Unexpected token 'mrkdwn_in": ["text"],
"color": "$COLOUR",
"text": "$MESSAGE",
}
]"' in expression or statement.
At C:\path-to-file-etc.ps1:53 char:11
+ "mrkdwn_in": ["text"],
+ ~
The hash literal was incomplete.At
C:\path-to-file-etc.ps1:53 char:11
+ "mrkdwn_in": ["text"],
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordEx
ception
+ FullyQualifiedErrorId : UnexpectedToken
Process exited with code 1
Process exited with code 1
Step Notify (PowerShell) failed
I have pretty much zero experience in Powershell. Also because this script has to be able to dropped onto a whole variety of boxes,I won't use any libraries or custom cmdlets or anything like that. Out of box methods or die.
Since you're using ConvertTo-Json anyway, it's simpler to construct the entire input as a (nested) custom object / hashtable, and let ConvertTo-Json handle all JSON formatting:
$Body = [pscustomobject] #{
attachments = , [pscustomobject] #{
mrkdwn_in = , 'text'
color = $Colour
text = $Message
}
}
$BodyJson = $Body | ConvertTo-Json -Depth 3
Note: You could substitute [ordered] for [pscustomobject] to use a hashtable with ordered keys instead; even omitting either would work, though seeing the entries in the resulting JSON in different order may be confusing.
Note the use of ,, the array-construction operator, to ensure that the attachments and mrkdwn_in entries are treated as arrays.
Additionally, since ConvertTo-Json only fully serializes to a default depth of 2 levels, -Depth 3 must be used to ensure that the value of entry mrkdwn_in is serialized as an array.
As for what you tried:
Your immediate problem (in addition to the missing } that Jeff Zeitlin points out in a comment on the question): You've neglected to escape the embedded " chars. in your multi-line string.
Therefore, as documentation topic Get-Help about_Quoting_Rules discusses, you can either use `" to embed double quotes inside "..." or use a here-string (#"<nl>....<n>"#).
Even with the syntax problems fixed, your code wouldn't work as intended, however, because ConvertTo-Json wouldn't preserve the pre-formatted JSON in the attachments entry and instead treat the string value as a string literal that needs escaping; here's a simple example:
#{
foo = "[
1,
2
]"
} | ConvertTo-Json
The above yields:
{
"foo": "[\n 1, \n 2 \n ]"
}