PowerShell: ConvertTo-Json problem containing special characters - json

I am writing a script to make changes to a JSON file but when the file is converted back to JSON it expands special characters.
For example the JSON File contain passwords with "&". A quick way to replicate the problem is using following command:
PS> "Password&123" | Convertto-Json
output is:"Password\u0026123"
##Here is how I import the JSON FILE:
$jsonfile = (Get-Content .\example.json -Encoding Ascii) -join "`n" | ConvertFrom-Json
##Exporting JSON FILE without modifying it.
$jsonfile | ConvertTo-Json |Out-File "new.json"
--here is an example of simplified JSON FILE
{
"Server1":
{
"username":"root",
"password":"Password&dfdf"
},
"Server2":
{
"username":"admin",
"password":"Password&1234"
}
}

Try the Unescape() method:
$jsonfile | ConvertTo-Json | % { [System.Text.RegularExpressions.Regex]::Unescape($_) } | Out-File "new.json"

This is caused by the automatic character escape feature of Convertto-Json and it affects several symbols such as <>\'&
ConvertFrom-Json will read the escaped characters properly. Using your example:
PS C:\> {"Password\u0026123"} | ConvertFrom-Json
Password&123
And your example code results in a file that has escaped characters, but ConvertFrom-Json can read it back to the original passwords. See below:
PS C:\> (Get-Content .\example.json -Encoding Ascii) -join "`n" | ConvertFrom-Json
Server1 Server2
------- -------
#{username=root; password=Password&dfdf} #{username=admin; password=Password&1234}
PS C:\> (Get-Content .\new.json -Encoding Ascii) -join "`n" | ConvertFrom-Json
Server1 Server2
------- -------
#{username=root; password=Password&dfdf} #{username=admin; password=Password&1234}
If you need the passwords to be stored unescaped, some fancier work may be needed. See this thread about Converting Unicode strings to escaped ascii strings
Alternatively, avoid affected characters if possible.

Tested with PowerShell 7.2 and appears unicode and other special characters are converted successfully. Not to mention indentation also appears improved.

Related

Adjust json file using ConvertFrom-Json

I have a json file with the following content
{
"name":"myApp",
"version":"1"
}
I would like to use Powershell to read this json file, adjust the version number, then output the new json to the same file.
I currently have this, but it doesn't work and only outputs an empty file
Get-Content file.json | ConvertFrom-Json | ForEach{$_.version = "2"} | ConvertTo-Json | Out-File file.json -Encoding ascii
I am getting stuck on how to update values on the Json object created by ConvertFrom-Json and pipe this back into ConvertTo-Json
I would like to do this all in one line if possible.
Once you assign the JSON to an object, you should be able to change it from there then write it to a file (if desired).
Create JSON object from file contents:
$jsonObject = Get-Content file.json -Raw | ConvertFrom-Json
Change the current object. In your example the "version" property:
#update object
$jsonObject| %{$_.version = "2"}
Write object to file:
$jsonObject | ConvertTo-Json | Out-File "file.json" -Encoding ascii

How to convert cyrillic into utf16

tl;dr Is there a way to convert cyrillic stored in hashtable into UTF-16?
Like кириллица into \u043a\u0438\u0440\u0438\u043b\u043b\u0438\u0446\u0430
I need to import file, parse it into id and value then convert it into .json and now im struggling to find a way to convert value into utf codes.
And yes, it is needed that way
cyrillic.txt:
1 кириллица
PH:
clear-host
foreach ($line in (Get-Content C:\Users\users\Downloads\cyrillic.txt)){
$nline = $line.Split(' ', 2)
$properties = #{
'id'= $nline[0] #stores "1" from file
'value'=$nline[1] #stores "кириллица" from file
}
$temp+=New-Object PSObject -Property $properties
}
$temp | ConvertTo-Json | Out-File "C:\Users\user\Downloads\data.json"
Output:
[
{
"id": "1",
"value": "кириллица"
},
]
Needed:
[
{
"id": "1",
"value": "\u043a\u0438\u0440\u0438\u043b\u043b\u0438\u0446\u0430"
},
]
At this point as a newcomer to PH i have no idea even how to search for it properly
Building on Jeroen Mostert's helpful comment, the following works robustly, assuming that the input file contains no NUL characters (which is usually a safe assumption for text files):
# Sample value pair; loop over file lines omitted for brevity.
$nline = '1 кириллица'.Split(' ', 2)
$properties = [ordered] #{
id = $nline[0]
# Insert aux. NUL characters before the 4-digit hex representations of each
# code unit, to be removed later.
value = -join ([uint16[]] [char[]] $nline[1]).ForEach({ "`0{0:x4}" -f $_ })
}
# Convert to JSON, then remove the escaped representations of the aux. NUL chars.,
# resulting in proper JSON escape sequences.
# Note: ... | Out-File ... omitted.
(ConvertTo-Json #($properties)) -replace '\\u0000', '\u'
Output (pipe to ConvertFrom-Json to verify that it works):
[
{
"id": "1",
"value": "\u043a\u0438\u0440\u0438\u043b\u043b\u0438\u0446\u0430"
}
]
Explanation:
[uint16[]] [char[]] $nline[1] converts the [char] instances of the strings stored in $nline[1] into the underlying UTF-16 code units (a .NET [char] is an unsigned 16-bit integer encoding a Unicode code point).
Note that this works even with Unicode characters that have code points above 0xFFFF, i.e. that are too large to fit into a [uint16]. Such characters outside the so-called BMP (Basic Multilingual Plane), e.g. 👍, are simply represented as pairs of UTF-16 code units, so-called surrogate pairs, which a JSON processor should recognize (ConvertFrom-Json does).
However, on Windows such chars. may not render correctly, depending on your console window's font. The safest option is to use Windows Terminal, available in the Microsoft Store
The call to the .ForEach() array method processes each resulting code unit:
"`0{0:x4}" -f $_ uses an expandable string to create a string that starts with a NUL character ("`0"), followed by a 4-digit hex. representation (x4) of the code unit at hand, created via -f, the format operator.
This trick of replacing what should ultimately be a verbatim \u prefix temporarily with a NUL character is needed, because a verbatim \ embedded in a string value would invariably be doubled in its JSON representation, given that \ acts the escape character in JSON.
The result is something like "<NUL>043a", which ConvertTo-Json transforms as follows, given that it must escape each NUL character as \u0000:
"\u0000043a"
The result from ConvertTo-Json can then be transformed into the desired escape sequences simply by replacing \u0000 (escaped as \\u0000 for use with the regex-based -replace oeprator) with \u, e.g.:
"\u0000043a" -replace '\\u0000', '\u' # -> "\u043a", i.e. к
Here's a way simply saving it to a utf16be file and then reading out the bytes, and formatting it, skipping the first 2 bytes, which is the bom (\ufeff). $_ didn't work by itself. Note that there's two utf16 encodings that have different byte orders, big endian and little endian. The range of cyrillic is U+0400..U+04FF. Added -nonewline.
'кириллица' | set-content utf16be.txt -encoding BigEndianUnicode -nonewline
$list = get-content utf16be.txt -Encoding Byte -readcount 2 |
% { '\u{0:x2}{1:x2}' -f $_[0],$_[1] } | select -skip 1
-join $list
\u043a\u0438\u0440\u0438\u043b\u043b\u0438\u0446\u0430
There must be a simpler way of doing this, but this could work for you:
$temp = foreach ($line in (Get-Content -Path 'C:\Users\users\Downloads\cyrillic.txt')){
$nline = $line.Split(' ', 2)
# output an object straight away so it gets collected in variable $temp
[PsCustomObject]#{
id = $nline[0] #stores "1" from file
value = (([system.Text.Encoding]::BigEndianUnicode.GetBytes($nline[1]) |
ForEach-Object {'{0:x2}' -f $_ }) -join '' -split '(.{4})' -ne '' |
ForEach-Object { '\u{0}' -f $_ }) -join ''
}
}
($temp | ConvertTo-Json) -replace '\\\\u', '\u' | Out-File 'C:\Users\user\Downloads\data.json'
Simpler using .ToCharArray():
$temp = foreach ($line in (Get-Content -Path 'C:\Users\users\Downloads\cyrillic.txt')){
$nline = $line.Split(' ', 2)
# output an object straight away so it gets collected in variable $temp
[PsCustomObject]#{
id = $nline[0] #stores "1" from file
value = ($nline[1].ToCharArray() | ForEach-Object {'\u{0:x4}' -f [uint16]$_ }) -join ''
}
}
($temp | ConvertTo-Json) -replace '\\\\u', '\u' | Out-File 'C:\Users\user\Downloads\data.json'
Value "кириллица" will be converted to \u043a\u0438\u0440\u0438\u043b\u043b\u0438\u0446\u0430

How to replace JSON key value?

{"enable"=false,
"datatype"="DB",
"report"=false}
I need to change the enable value to true.
As now I am using the following code:
(Get-Content 'C:\Shibi\Basic.json') -replace 'false', 'true' | Set-Content 'C:\Shibi\Basic.json'
If you are going to work with a text file rather than a true JSON structure, you can use the *-Content commands with the -replace operator.
(Get-Content 'C:\Shibi\Basic.json') -replace '"enable"=false','"enable"=true' | Set-Content
Note: Set-Content will update the Basic.json file with the replaced data.
You can add the -Raw switch to Get-Content to preserve white space or newline characters of the original file.
If you put the content in JSON format like other people suggested, this solution worked for me:
$jsonstr = Get-Content thecontent.json | ConvertFrom-Json
$jsonstr.enable = "true"
$json = $jsonstr | ConvertTo-Json
echo $json > thecontent.json
Get-Content thecontent.json

JSON formatting, either from file or variable

I have a PS script, which get JSON in variable ant then saves it in file.
Unfortunately, it get value in one string, like this:
{ "persistentdataapi": "https://somevalue.azurewebsites.net/", "collectioncountapi": "https://anothervalue.azurewebsites.net/", "eventserviceapi": "https://thirdvalue.azurewebsites.net/", "securityserviceapi": "https://fourthvalue.azurewebsites.net/" }
Is there any way, to process this value through some (preferably PS) JSON formatting, to get this one:
{
"persistentdataapi": "https://somevalue.azurewebsites.net/",
"collectioncountapi": "https://anothervalue.azurewebsites.net/",
"eventserviceapi": "https://thirdvalue.azurewebsites.net/",
"securityserviceapi": "https://fourthvalue.azurewebsites.net/",
}
Code to get value in Jenkins:
Import-Module "C:\Program Files\WindowsPowerShell\Modules\Octopus-Cmdlets\0.4.4\Octopus-Cmdlets.psd1"
connect-octoserver http://internal-Octopus.azure.com:8082 API-123456789012345678
$raw = (Get-OctoVariable var.Portal.Web DataAPIJson | Where-Object { $_.Environment -eq "QA" } )
$raw.Value | Out-File "$env:WORKSPACE\portal\var.Portal.Web\dataapi.json"
Powershell by default pretty-prints any JSON it produces.
So the correct way to do pretty-printing is to parse the JSON string into an object, and immediately convert it back to a JSON string.
$json = '{ "persistentdataapi": "https://somevalue.azurewebsites.net/", "collectioncountapi": "https://anothervalue.azurewebsites.net/", "eventserviceapi": "https://thirdvalue.azurewebsites.net/", "securityserviceapi": "https://fourthvalue.azurewebsites.net/" }'
$json | ConvertFrom-Json | ConvertTo-Json
produces
{
"persistentdataapi": "https://somevalue.azurewebsites.net/",
"collectioncountapi": "https://anothervalue.azurewebsites.net/",
"eventserviceapi": "https://thirdvalue.azurewebsites.net/",
"securityserviceapi": "https://fourthvalue.azurewebsites.net/"
}
or in your case
$file = "$env:WORKSPACE\portal\var.Portal.Web\dataapi.json"
$raw.Value | ConvertFrom-Json | ConvertTo-Json | Out-File $file -Encoding UTF8
As a side-effect this also makes sure that the JSON in the file is valid, because otherwise ConvertFrom-Json will throw an error.
Please always explicitly specify UTF8 encoding when reading and writing JSON files.
$data = Get-Content $file -Encoding UTF8 | ConvertFrom-Json
$data | ConvertTo-Json | Set-Content $file -Encoding UTF8
The reason for that is
By widely-accepted convention, JSON files ought to be UTF8.
Unless specified otherwise, Get-Content and Set-Content will use the system's default encoding to read/write text files.
The system default is very seldom UTF-8, most of the time it will be a legacy single-byte encoding like Windows-1252.
This creates the risk of
mangling Unicode characters, which are legal in JSON, when reading a JSON file.
creating JSON files that are not UTF-8, making them hard to consume by others.
In fact, always specify an encoding explicitly when working with text files, not only in the case of JSON.

Replace only first occurrence

I have a PowerShell scripts which replaces
"version" : "xxx"
with
"version" : "myBuildNumber"
Now I encountered that I have multiple of these in my file.
I only want to replace the first occurrence.
I tried already Powershell - Replace first occurrences of String but it does not work with my regex.
Here's my script:
(Get-Content myFile.txt) -replace '(?<pre>"version"[\s]*:[\s]*)(?<V>"[^\"]*")', "`$1`"$Env:BUILD_VERSION`"" | Out-File myFile.txt
Since you are patching a JSON file, regex isn't the way to go. Instead you should parse the JSON, access and change the property you want and write it back:
$filePath = 'your_Path_To_project.json'
$json = (Get-Content $filePath -raw | ConvertFrom-Json)
$json.version = $Env:BUILD_VERSION
$json | ConvertTo-Json -Depth 10 | Set-Content $filePath