Why does ConvertFrom-Json produce an Object which turns into a PSCustomObject after assignment? And how do I shortcut that? - json

I have a simple json file as follows:
[
{"ClientName": "Test Site 1", "ClientID": "000001"},
{"ClientName": "Test Site 2", "ClientID": "000002"},
{"ClientName": "Test Site 3", "ClientID": "000003"}
]
When I use the following PowerShell command:
ConvertFrom-Json (Get-Content TestSites.json -Raw)
I get back a System.Object[]. This doesn't allow me to pipe the output to another function I have which accepts "ClientName" and "ClientID" parameters.
However, when I assign that object to another variable, like this:
$myobj = ConvertFrom-Json (Get-Content TestSites.json -Raw)
$myobj is actually a System.Management.Automation.PSCustomObject which is capable of being passed to my function.
How can I just pipe the results of the original command without having to assign it to another variable first?
I hope that makes sense.

Your JSON is an array correct? PowerShell will unroll arrays in the pipeline unless you explicity change that behavior. Assuming your JSON is stored in the variable $json as a single string consider the following examples.
ConvertFrom-Json $json | ForEach-Object{$_.gettype().fullname}
System.Object[]
(convertFrom-Json $json) | ForEach-Object{$_.gettype().fullname}
System.Management.Automation.PSCustomObject
System.Management.Automation.PSCustomObject
System.Management.Automation.PSCustomObject
You should be able to wrap the expression in brackets to change the outcome. In the second example it should be sending the 3 objects individually down the pipe. In the first it is being sent as a single object array.
My explanation needs work but I am sure of the cause is how PowerShell deals with arrays and the pipeline. Unrolling being a common word used to describe it.
So depending on your use case you might just be able to wrap the expression in brackets so it gets processed before the pipe to ForEach in my example.

If you have an array such as System.Object[] you could try piping via foreach:
ConvertFrom-Json (Get-Content TestSites.json -Raw) | %{ $_ | your-function }
If you want to pass the whole array down the pipe as-is, you can try adding a comma (aka a unary comma before the variable:
,$myobj | whatever
You can probably see how the latter works by comparing the following:
$myobj | Get-Member # Shows the type of the elements of the array
Get-Member -InputObject $myobj # Shows the type of the array
,$myobj | Get-Member # Shows the type of the array

Related

Select-Object Data Failure From Succesful Invoke-RestMethod

I have a working call to a rest service using Invoke-RestMethod -Uri https://.... The call's result, in JSON, can be piped the data to Format-Table -Property ... and data is shown.
But when Select-Object -Property ... is used after the invoke with the same parameters, the PSObject has the columns but no data for them. If I use a different webservice the call will work.
What could be causing the PSObject to not show any values?
Working Example on public rest web services
Invoke-RestMethod -Uri https://jsonplaceholder.typicode.com/todos/1 |
Select-Object -Property title
Result
#{title=delectus aut autem}
New Failure Different API
Invoke-RestMethod -Uri https://cat-fact.herokuapp.com/facts | Select-Object -Property text
You've stumbled upon an unholy combination of two PowerShell oddities when converting JSON arrays:
Invoke-RestMethod and ConvertFrom-Json send converted-from-JSON arrays as a whole through the pipeline, instead of element by element, as usual:
Note: In PowerShell (Core) 7.0, ComvertFrom-Json's behavior was changed to align with the usual enumeration-of-elements behavior, and a -NoEnumerate switch was added as an opt-in to the old behavior. For the discussion that led to this change, see GitHub issue #3424.
However, as of this writing (PowerShell (Core 7.2) Invoke-RestMethod still exhibits this unexpected behavior, which is discussed in GitHub issue #15272.
Select-Object does not perform member-access enumeration, so it looks for the specified property (e.g., text) directly on the array, where it doesn't exist.
To demonstrate the problem with a simple example:
# Windows PowerShell only:
# Because ConvertFrom-Json sends an *array* (of 2 custom objects) through
# the pipeline, Select-Object looks for property .text on the *array* -
# and can't find it.
# The same applies to Invoke-RestMethod (also still in
# PowerShell (Core) as of v7.2)
PS> ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]' | Select-Object text
text
----
# NO VALUES
A simple workaround is to enclose the ConvertFrom-Json / Invoke-RestMethod call in (...), which forces enumeration of the array, causing Select-Object to work as expected.:
# (...) forces enumeration
PS> (ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]') | Select-Object text
text
----
a
b
Note that a command such as Select-Object -Property text (without -ExpandProperty) still output custom objects that have a .text property, not the .text property values.
If all you're interested in is property values, the solution is simpler, because you can use the above-mentioned member-access enumeration directly on the array:
# Using .<propName> on an array (collection) implicitly returns the
# property values from the *elements* of that collection (member-access enumeration).
PS> (ConvertFrom-Json '[{ "text": "a" }, { "text": "b" }]').text
a
b
Note how the output now has no text header, because it is mere string values that are being output, not custom objects.
your problem in the 2nd example is that there is NO prop named text. [grin]
the only prop is all and that contains an array of objects that DO contain a prop named text. so you need something that can get that deeper prop. one way is to use two Select-Object calls. something like this ...
$Url = 'https://cat-fact.herokuapp.com/facts'
$RawIRM = Invoke-RestMethod -Uri $Url
$SO_IRM = $RawIRM |
Select-Object -ExpandProperty all |
Select-Object -Property text
the $SO_IRM var now has an array of 178 strings about cats. [grin]

Using Powershell to convert a file's contents into a string that can be transferred using JSON

How does one convert a text file's contents into a string and then insert this string into a JSON file?
For example, if a file contains:
this
is
a
sample
file
The script would generate:
"this\r\nis\r\na\r\nsample\r\nfile"
To insert into a JSON template:
"something":"<insertPoint>"
To produce:
"something":"this\r\nis\r\na\r\nsample\r\nfile"
I'm using Powershell 5 and have managed to load the file, generate some JSON and insert it by running:
# get contents and convert to JSON
$contentToInsert = Get-Content $sourceFilePath -raw | ConvertTo-Json
# write in output file
(Get-Content $outputFile -Raw).replace('<insertPoint>', $contentToInsert) | Set-Content $outputFile
However, a lot of other, unwanted fields are also added.
"something":"{
"value": "this\r\nis\r\na\r\nsample\r\nfile"
"PSPath": "C:\\src\\intro.md",
"PSParentPath": "C:\\src",
"PSChildName": "intro.md",
etc...
Ultimately, I'm trying to send small rich text segments to a web page via JSON but want to edit and store them locally using Markdown. If this doesn't make sense and there's a better way of sending these then please let me know also.
iRon's answer helpfully suggests not using string manipulation to create JSON in PowerShell, but to use hashtables (or custom objects) to construct the data and then convert it to JSON.
However, that alone does not solve your problem:
PS> #{ something = Get-Content -Raw $sourceFilePath } | ConvertTo-Json
{
"something": {
"value": "this\nis\na\nsample\nfile\n",
"PSPath": "/Users/mklement/Desktop/pg/lines.txt",
# ... !! unwanted properties are still there
}
The root cause of the problem is that Get-Content decorates the strings it outputs with metadata in the form of NoteProperty properties, and ConvertTo-Json currently invariably includes these.
A proposal to allow opting out of this decoration when calling Get-Content can be found in GitHub issue #7537.
Complementarily, GitHub issue #5797 suggests that ConvertTo-Json should ignore the extra properties for primitive .NET types such as strings.
The simplest workaround is to access the underlying .NET instance with .psobject.baseobject, which bypasses the invisible wrapper object PowerShell uses to supply the extra properties:
PS> #{ something = (Get-Content -Raw $sourceFilePath).psobject.baseobject } |
ConvertTo-Json
{
"something": "this\nis\na\nsample\nfile\n"
}
Just a general recommendation apart from the actually issue described by #mklement0 and metadata added to the Get-Content results:
Do not poke (replace, insert, etc.) in any Json content.
Instead, modify the object (if necessary, use ConvertFrom-Json to restore the object) prior converting it into (ConvertTo-Json) a Json file.
In this example, I would use a hash-table with a here-string for this:
#{'something' = #'
this
is
a
sample
file
'#
} | ConvertTo-Json
Result:
{
"something": "this\nis\na\nsample\nfile"
}
You can use the Out-String cmdlet to coerce the output of Get-Content into a flat string first:
#{ "something" = (Get-Content lines.txt | Out-String) } | ConvertTo-Json
This produces:
{
"something": "this\r\nis\r\na\r\nsample\r\nfile\r\n"
}

Add windows env variables to json object using powershell

I'm quite new to powershell and just need it for a small task so please excuse my complete and utter ineptitude for the language. I was wondering if it were possible to form a json object based off environment variables and a variable that has already been declared earlier in my script. The variable that was already declared is based off a json config named optionsConfig.json and the contents of that file are here.
{"test1": ["options_size", "options_connection", "options_object"],
"test2":["options_customArgs", "options_noUDP", "options_noName"]}
The purpose of the $Options variable in the code below is to take each element in the list value for the respective test and assume that those elements are environment variables in the system, then find their values and form a dictionary object that will be used in the json.
Here is what I have so far.
# Read the JSON file into a custom object.
$configObj = Get-Content -Raw optionsConfig.json |
ConvertFrom-Json
# Retrieve the environment variables whose
# names are listed in the $env:test property
# as name-value pairs.
Get-Item -Path env:* -Include $configObj.$env:testTool
$Options = Get-Item -Path env:* -Include $configObj.$env:testTool |
% {$hash = #{}} {$hash[$_.Name]=$_.Value} {$hash}
The $Options variable looks like so when converted to json
{
"options_size": "default",
"options_object": "forward open",
"options_connection": "connected"
}
I have a few other environment variable values that I would like to be a part of the json object. Those 3 other environment variables I would like the value of are listed below.
$Env.testTool = "test1"
$Env.RecordName = "Record1"
$Env.Target = "Target1"
How would I construct a powershell statement to get the json object to be formatted like this? -
data = {"test": $Env.testTool, "target": "$Env.Target",
"options": "$Options", "RecordName': "$Env.RecordName"}
The keys are all predefined strings and $Options is the dict object from up above. Am I able to form a Json object like this in powershell and how would it be done? Any help would be appreciated. This appears to be the last step in my struggle with powershell.
Here is what I have done.
$jObj = [ordered]#{test= $Env:testTool}
When I change this variable to $jObj = [ordered]#{test= $Env:testTool,options= $Options} I get an error saying missing expression after ','
When I change this variable to $jObj = [ordered]#{test= $Env:testTool,options= $Options} I get an error saying missing expression after ','
Entries of a hashtable literal (#{ ... } or, in its ordered form, [ordered] #{ ... }) must be separated:
either by newlines (each entry on its own line)
or by ; if placed on the same line.
Thus, the following literals are equivalent:
# Multiline form
#{
test= $env:testTool
RecordName= $env:RecordName
Target= $env.Target
options=$Options
}
# Single-line form; separator is ";"
#{ test= $env:testTool; RecordName= $env:RecordName; Target= $env.Target; options=$Options }
Get-Help about_Hashtables has more information.
$jObj = #{test= $env:testTool
RecordName= $env:RecordName
Target= $env.Target
options=$Options}
$jObj | ConvertTo-Json | Set-Content jsonStuff.json
JsonStuff.json is the new json file for the new json object. This syntax for forming $jObj seems to have done the trick.

How to remove object from array in power shell?

I am trying to remove the complete object from the array not a member of the object.I am not able to find the way to remove the object there are so many solutions available to remove the item.
Can someone please suggest a way remove the complete object.
JSON Data: JSON data stored in the file.
{
"data": [
{
"id": "Caption",
"firstname": "Caption",
"lastname": "test",
"email": "test",
"requester": "test",
"password": "test",
"incNumber": "test"
}
]
}
Code : I have written the following code to read the object from the array and store into variables to do the task.Once the task is completed I want to remove the object from the array.
$file = Get-Content '\path\to\file' -Raw | ConvertFrom-Json
$file.data | % {if($_.id -eq 'Caption'){
$1 = $_.id
Write-Host $1
###Here I want to remove the whole object related to the id
}}
I think the answer in the comments is what you are looking for: file.data = $file.data | ? id -ne 'Caption'.
To give a little explanation to this, it uses ? which is actually an alias (alternative name) for the Where-Object cmdlet, which is what you use when you want to filter a collection based on some criteria (very similar to the WHERE statement in SQL if you are familiar).
The above answer is a short version, you might alternatively see this:
$file = Get-Content '\path\to\file' -Raw | ConvertFrom-Json
$Result = $file.data | Where-Object {$_.id -ne 'Caption'}
Which is passing a ScriptBlock { } to the Where-Object cmdlet and using $_ to represent each item in the pipeline, then using -ne as the not equals comparison operator to see if the ID property of that object does not match the string Caption. If that is evaluated as true, then it allows the item to pass through the pipeline, which in the above example means it ends up in the $Result variable. If the statement evaluates as false, then it discards that item in the collection and moves on to the next.

powershell piped input to export-csv different from -inputobject

In Powershell (3.0), I get different results when piping an object to Export-CSV than I do if I use the -IncludeObject parameter with the same object.
Example:
$a = Get-Process | select -first 5
$a | Export-CSV -Path '.\one.csv'
Export-CSV -InputObject $a -Path '.\two.csv'
Why are the contents of one.csv and two.csv different?
(And in case it's just me ...)
one.csv =
#TYPE System.Diagnostics.Process
"__NounName","Name","Handles","VM","WS","PM","NPM","Path","Company","CPU","FileVersion","ProductVersion","Description","Product","BasePriority","ExitCode","HasExited","ExitTime","Handle","HandleCount","Id","MachineName","MainWindowHandle","MainWindowTitle","MainModule","MaxWorkingSet","MinWorkingSet","Modules","NonpagedSystemMemorySize","NonpagedSystemMemorySize64","PagedMemorySize","PagedMemorySize64","PagedSystemMemorySize","PagedSystemMemorySize64","PeakPagedMemorySize","PeakPagedMemorySize64","PeakWorkingSet","PeakWorkingSet64","PeakVirtualMemorySize","PeakVirtualMemorySize64","PriorityBoostEnabled","PriorityClass","PrivateMemorySize","PrivateMemorySize64","PrivilegedProcessorTime","ProcessName","ProcessorAffinity","Responding","SessionId","StartInfo","StartTime","SynchronizingObject","Threads","TotalProcessorTime","UserProcessorTime","VirtualMemorySize","VirtualMemorySize64","EnableRaisingEvents","StandardInput","StandardOutput","StandardError","WorkingSet","WorkingSet64","Site","Container"
"Process","AATray","390","156721152","29769728","10678272","27600","C:\Program Files\IBM\ISAM ESSO\AA\AATray.exe","IBM Corporation","42.4166719","8.2.1.1143","8.2.1.1143","AccessAgent Tray Icon: Component of ISAM ESSO AccessAgent","ISAM ESSO AccessAgent","8",,"False",,"4844","390","7784",".","0","","System.Diagnostics.ProcessModule (AATray.exe)","1413120","204800","System.Diagnostics.ProcessModuleCollection","27600","27600","10678272","10678272","257536","257536","63672320","63672320","29806592","29806592","194101248","194101248","True","Normal","10678272","10678272","00:00:32.8070103","AATray","15","True","1","System.Diagnostics.ProcessStartInfo","8/2/2016 7:20:30 AM",,"System.Diagnostics.ProcessThreadCollection","00:00:42.4166719","00:00:09.6096616","156721152","156721152","False",,,,"29769728","29769728",,
"Process","ac.activclient.gui.scagent","547","155099136","22593536","8478720","33184","C:\Program Files\ActivIdentity\ActivClient\ac.activclient.gui.scagent.exe","HID Global Identity Assurance","2.2932147","7,0,5,17","7,0","ActivClient Agent","${release.product.name}","8",,"False",,"3648","547","7872",".","0","","System.Diagnostics.ProcessModule (ac.activclient.gui.scagent.exe)","1413120","204800","System.Diagnostics.ProcessModuleCollection","33184","33184","8478720","8478720","274408","274408","62431232","62431232","22659072","22659072","168542208","168542208","True","Normal","8478720","8478720","00:00:01.8876121","ac.activclient.gui.scagent","15","True","1","System.Diagnostics.ProcessStartInfo","8/2/2016 7:20:30 AM",,"System.Diagnostics.ProcessThreadCollection","00:00:02.2932147","00:00:00.4056026","155099136","155099136","False",,,,"22593536","22593536",,
"Process","accagt","166","508174336","17592320","25407488","26116",,,,,,,,"8",,,,,"166","2480",".","0","",,,,,"26116","26116","25407488","25407488","159440","159440","25657344","25657344","17694720","17694720","509747200","509747200",,,"25407488","25407488",,"accagt",,"True","0","System.Diagnostics.ProcessStartInfo",,,"System.Diagnostics.ProcessThreadCollection",,,"508174336","508174336","False",,,,"17592320","17592320",,
"Process","acevents","506","140414976","22474752","8048640","30856","C:\Program Files\ActivIdentity\ActivClient\acevents.exe","HID Global Identity Assurance","42.6350733","5,0,4,4","5,0","ActivIdentity Event Service","ActivClient Services","8",,"False",,"3872","506","8256",".","0","","System.Diagnostics.ProcessModule (acevents.exe)","1413120","204800","System.Diagnostics.ProcessModuleCollection","30856","30856","8048640","8048640","249632","249632","61378560","61378560","22528000","22528000","157003776","157003776","True","Normal","8048640","8048640","00:00:24.6013577","acevents","15","True","1","System.Diagnostics.ProcessStartInfo","8/2/2016 7:20:34 AM",,"System.Diagnostics.ProcessThreadCollection","00:00:42.6350733","00:00:18.0337156","140414976","140414976","False",,,,"22474752","22474752",,
"Process","acnamagent","395","98971648","14561280","6586368","28296",,,,,,,,"8",,,,,"395","2012",".","0","",,,,,"28296","28296","6586368","6586368","125784","125784","6647808","6647808","14594048","14594048","101597184","101597184",,,"6586368","6586368",,"acnamagent",,"True","0","System.Diagnostics.ProcessStartInfo",,,"System.Diagnostics.ProcessThreadCollection",,,"98971648","98971648","False",,,,"14561280","14561280",,
two.csv =
#TYPE System.Object[]
"Count","Length","LongLength","Rank","SyncRoot","IsReadOnly","IsFixedSize","IsSynchronized"
"5","5","5","1","System.Object[]","False","True","False"
For context, I'm trying to splat my parameters to Export-CSV, but I run into this when I pass -InputObject, and I can't pipe the input and then splat the rest of the parameters.
Thanks.
This is the expected behavior.
When you pipe in through the pipeline, arrays, collections, enumerable stuff, etc. gets processed item by item. This is usually what you want.
When you use -InputObject, it accepts the array as a single object.
The best way to see this is to use Get-Member:
$a = Get-Process | select -first 5
$a | Get-Member
Get-Member -InputObject $a
In the first invocation, you'll see the data type and members of each element. In the second invocation you'll see the type and members of the collection object.
Depending on the cmdlet, you may not notice difference at all because it's handling both cases (see the pipeline function at the end of my answer).
But in the case of Export-Csv, or ConvertTo-Json, or other serialization type cmdlets, you want this difference; otherwise it's very difficult to serialize the array explicitly when you want to.
Another way to demonstrate it:
$sb = {
$_
Write-Verbose $_.Count -Verbose
}
$a | ForEach-Object -Process $sb
ForEach-Object -Process $sb -InputObject $a
When writing your own pipeline functions, a common way to work around the different ways of receiving the object is to use the Process {} block along with foreach:
function Test-Pipeline {
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline)]
$MyVal
)
Process {
Write-Verbose $MyVal.Count -Verbose
foreach($v in $MyVal) {
$v
}
}
}
$a | Test-Pipeline
# Process block gets called once for each element
Test-Pipeline -MyVal $a
# Process block gets called once total, with the variable being an array
This works well because foreach doesn't fail if you give it a single non-array object, it just executes once.