Problem using Powershell to extract a value from a Json structure - json

Using Powershell, I get a json snippet returned into my First variable (this always works fine);
# Initialise variables...
$nMessage_id = "";
$whatStatusJsonContent = "";
# Function https://abc.googleapis.com/abc/load call here and returns...
$whatStatusJsonContent = '{"message_id":9093813071099257562}'
Then I call the convert function into a temp variable like this;
$ResponseBody = ConvertFrom-Json $whatStatusJsonContent;
which puts the Json into a nice little data structure like this;
message_id
----------
9093813071099257562
From which I can select the value I want by calling this;
$nMessage_id = $ResponseBody.message_id;
Usually, this works fine and I get the value into my second variable;
$nMessage_id = 9093813071099257562
The problem is: Sometimes I get nothing in $nMessage_id, even though $whatStatusJsonContent is definitely logged as having the Json returned correctly from the function.
My question is: Do I have to ConvertFrom-Json, or can I read it raw from the First variable..?
COMBINED SOLUTION: Thanks to #mklement() and #Bernard-Moeskops
# Initialise variables...
$nMessage_id = "";
$whatStatusJsonContent = "";
# Function https://abc.googleapis.com/abc/load call here and returns...
$whatStatusJsonContent = '{"message_id":9093813071099257562}'
$ResponseBody = ConvertFrom-Json $whatStatusJsonContent;
if($ResponseBody.message_id){
# ConvertFrom-Json got the value!
$nMessage_id = $ResponseBody.message_id
}else{
# ConvertFrom-Json didn't work!
$nMessage_id = = ($whatStatusJsonContent -split '[:}]')[1]
}

There's nothing overtly wrong with your code.
ConvertFrom-Json should work as expected and return a [pscustomobject] instance with a .message_id property.
In your example, the message_id JSON property value is a number that is an integer, for which ConvertTo-Json automatically chooses a suitable integer data type as follows: the smallest signed type >= [int] (System.Int32)[1] that can accommodate the value ([int] -> [long] (System.Int64) -> [decimal] (System.Decimal)); the caveat is that if the value can't even fit into a [decimal], an - inexact - [double] is used.[2]
With the sample JSON in your question, [long] is chosen.
In a follow-up comment you state:
The routine makes over 1000 calls/hour and for most of them the Json comes back and the $nMessage_id is yielded perfectly. Then, suddenly, the $nMessage_id is empty, even though the Json is logged as coming back fine. So, somewhere in the ConvertFrom-Json or $ResponseBody.message_id the value is going missing...
I have no explanation, but if - for whatever reason - ConvertFrom-Json is the culprit, you can try string manipulation as a workaround to extract the message ID and see if that helps:
$whatStatusJsonContent = '{"message_id":9093813071099257562}'
# Extract the message_id property value as a *string*
# (which you can cast to a numeric type if/as needed).
$message_id = ($whatStatusJsonContent -split '[:}]')[1]
The above stores a string with content 9093813071099257562 in $message_id; note that, as written, the input string must have the exact format as above with respect to whitespace; while it is possible to make the text parsing more robust, not having to worry about format variations is one good reason to use a dedicated parser such as ConvertFrom-Json.
Another option is to try a different JSON parser to see if that helps.
Json.NET is the preeminent JSON parser in the .NET world (which now underlies the JSON cmdlets in PowerShell Core):
$whatStatusJsonContent = '{"message_id":9093813071099257562}'
$message_id = [NewtonSoft.Json.Linq.JObject]::Parse($whatStatusJsonContent).message_id.Value
Note: Json.NET - like ConvetFrom-Json in PowerShell _Core - commendably uses the arbitrary large [bigint] type as well once a number is too large to fit into a [long].
Use of the Json.NET assembly has the added advantage of better performance than the ConvertFrom-Json cmdlet.
In PowerShell Core, you can run the above code as-is (the assembly is preloaded); in Windows PowerShell you'll have to download the package via the link above and add the assembly (NewtonSoft.Json.dll) to your session with Add-Type -LiteralPath.
[1] Curiously, in PowerShell Core, as of (at least) v6.2.0, the smallest type chosen is [long] (System.Int64).
[2] More helpfully, PowerShell Core, as of (at least) v6.2.0, creates an arbitrarily large [bigint] (System.Numerics.BigInteger) instance once a value doesn't fit into a [long] anymore; that is, the [decimal] type is skipped altogether.

You are going to have to convert it, so that PowerShell can understand it. It will convert from a string to a PSCustomObject. Just check by asking the type of the variable before and after.
$ResponseBody.message_id.GetType()
If sometimes the output is nothing, you could do something like:
if($ResponseBody.message_id){
$nMessage_id = $ResponseBody.message_id
}else{
throw "No message id found"
}
Hope this helps.

Related

PowerShell adds int indexes of items when returning object

I've seen a lot of similar questions on here but none of the solutions are working for me.
I have a function that looks like this
function Get-ManagedDevices {
# ...(params and stuff here)...
$batchResponse = Invoke-RestMethod #restParams
$responses = #{}
foreach ($response in $batchResponse.responses) {
$responses.Add($response.id, $response.body) > $null
}
return , $responses
}
that gets called like this
$data = Get-Devices -AccessToken $AccessToken -DeviceIds $DeviceIds
When I'm debugging inside of the function call I am getting what I expect when I look at the $response variable which is something like
PS> $responses
Name Value
---- -----
d1290207-c693-4663-88bf-58512… #{#odata.context=https://graph.microsoft.com/v1.0/$metadat…
7ad71b70-b992-490f-8822-1561a… #{#odata.context=https://graph.microsoft.com/v1.0/$metadat…
PS> $responses.count
1
However, when I try to use the $data variable from after the function call I get this
PS> $data
0
1
Name Value
---- -----
d1290207-c693-4663-88bf-58512… #{#odata.context=https://graph.microsoft.com/v1.0/$metadat…
7ad71b70-b992-490f-8822-1561a… #{#odata.context=https://graph.microsoft.com/v1.0/$metadat…
PS> $data.count
3
I'm so confused. I piped the addition of the variable to the hashtable to $null with >$null.
I added the unary operator with return like return , $responses which is what's recommended in the 'about_return' PowerShell documentation .
I tried omitting the return keyword, using an ArrayList instead, using an array instead, etc...
What on earth is going on and why can't I return a hashtable from this function?! I feel like I'm going crazy.
The about_Return Documentation makes no mention of returning a hash table directly from a function. The common problem your referring to is the ability to return an array in whole rather than iterated down the pipeline.
Its common and straight forward to return a hash table. At a glance your code could be:
function Get-ManagedDevices {
# ...(params and stuff here)...
$batchResponse = Invoke-RestMethod #restParams
$responses = #{}
foreach ($response in $batchResponse.responses) {
$responses.Add($response.id, $response.body)
}
return $responses
}
It shouldn't be more complicated that that. Remove the comma it has no effect when returning a hash. >$null has no effect on the HashTable .Add(...) method.
Given your results and the example, there may be some artifact from your earlier attempts using an ArrayList. The .Add(...) method on an ArrayList will emit the index # that it's adding to. When we forget to [void] or | Out-Null or $null = ...Add(...) this usually results in unwanted output polluting the pipeline. As you're getting the 2 index numbers then your data...

PowerShell Function parameters - by reference or by value?

So, I tried looking up the answer to this question, and found the generally available answer is that PowerShell passes parameters by value. These generally accepted solutions all post sample code to prove their assertions, similar to the following:
Function add1 ($parameter)
{
Write-Host " In Function: `$parameter = $parameter"
Write-Host " In Function: `$parameter += 1"
$parameter += 1
Write-Host " In Function: `$parameter = $parameter"
}
cls
$a = 1
Write-Host "Before function: `$a = $a"
add1 $a
Write-Host " After function: `$a = $a"
This gives the results:
Before function: Run Command: $a = 1
In Function: $parameter: 1
In Function: Run Command: $parameter += 1
In Function: $parameter: 2
After function: $a: 1
Thus proving that parameters are passed by value, right? Well, I was having a heck of a time troubleshooting a function I was writing. The function added a couple of additional NoteProperty items to a PSCustomObject I pass in to the function, and my program would throw all sorts of errors saying that the NoteProperty already existed, even though I had not modified the original object in the parent scope, only inside the function.
So, I set up a version of the above code to test using parameter of type [PSCustomObject], like so:
Function F1($Obj)
{
'Function F1: Run command: $Obj.FirstValue = 11'
$Obj.FirstValue = 11
" `$Obj.Name: $($StartObject.Name)"
" `$Obj.FirstValue: $($StartObject.FirstValue)"
" `$Obj.SecondValue: $($StartObject.SecondValue)"
}
Function F2($Obj)
{
'Function F2: Run command: $Obj | Add-Member -MemberType NoteProperty -Name SecondValue -Value 33'
$obj | Add-Member -MemberType NoteProperty -Name SecondValue -Value 33
" `$Obj.Name: $($StartObject.Name)"
" `$Obj.FirstValue: $($StartObject.FirstValue)"
" `$Obj.SecondValue: $($StartObject.SecondValue)"
}
cls
Remove-Variable StartObject
"Main script: Run command: `$StartObject = [PSCustomObject]#{Name='Original';FirstValue=22}"
$StartObject = [PSCustomObject]#{Name='Original';FirstValue=22}
" `$StartObject.Name: $($StartObject.Name)"
" `$StartObject.FirstValue: $($StartObject.FirstValue)"
" `$StartObject.SecondValue: $($StartObject.SecondValue)"
'Run command: F1 $StartObject'
" "
F1 $StartObject
" "
"Main script: `$StartObject.Name: $($StartObject.Name)"
" `$StartObject.FirstValue: $($StartObject.FirstValue)"
" `$StartObject.SecondValue: $($StartObject.SecondValue)"
"Run command: F2 $StartObject"
" "
F2 $StartObject
" "
"Main script: `$StartObject.Name = $($StartObject.Name)"
" `$StartObject.FirstValue = $($StartObject.FirstValue)"
" `$StartObject.SecondValue = $($StartObject.SecondValue)"
This messy piece of programming produces the following output:
Main script: Run command: $StartObject = [PSCustomObject]#{Name='Original';FirstValue=22}
$StartObject.Name: Original
$StartObject.FirstValue: 22
$StartObject.SecondValue:
Run command: F1 $StartObject
Function F1: Run command: $Obj.FirstValue = 11
$Obj.Name: Original
$Obj.FirstValue: 11
$Obj.SecondValue:
Main script: $StartObject.Name: Original
$StartObject.FirstValue: 11
$StartObject.SecondValue:
Run command: F2 #{Name=Original; FirstValue=11}
Function F2: Run command: $Obj | Add-Member -MemberType NoteProperty -Name SecondValue -Value 33
$Obj.Name: Original
$Obj.FirstValue: 11
$Obj.SecondValue: 33
Main script: $StartObject.Name = Original
$StartObject.FirstValue = 11
$StartObject.SecondValue = 33
These results clearly show that when [PSCustomObject] parameters are used, any modifications within the function take place on the passed object, thus pass by reference. This behavior happens regardless of defining my parameters as [PSCustomObject]$Obj, or leaving them untyped. This is not a huge problem in and of itself, but the problem is that I was unable to find this little gem of information in any of the documentation I looked through. I checked a few tutorial sites and Microsoft's own documentation on Function Parameters, but did not see this exception.
So, my question boils down to this: Has anyone found any documentation to support my theory that while most parameters default to passing by value, they are passed by reference when objects are concerned?
I am perfectly willing to believe that I missed some documentation somewhere, so please...point it out and show me the error of my ways! :)
Thanks very much
Note: The following also applies to assigning one variable to another: $b = $a ...
* makes $b reference the very same object that $a does if $a's value is an instance of a reference type,
* makes $b receive an independent copy of $a's value if the latter is an instance of a value type.
PowerShell uses by-(variable)-value passing by default; that is, the content of a variable is passed, not a reference to the variable itself.
Extra effort is needed if you want by-(variable)-reference passing, i.e. if you want to pass a reference to a variable itself, allowing the callee to both get the variable's content and to assign new content; in the simplest form, you can use a [ref]-typed parameter (akin to ref parameters in C#). However, note that this technique is rarely necessary in PowerShell.
Whether that content is a copy of what the caller sees or a reference to the very same object depends on the data type of the content:
If the content happens to be an instance of a .NET reference type - as [pscustomobject] is - that content is an object reference, and the callee can therefore potentially modify that object, by virtue of seeing the very same object as the caller.
If you want to pass a copy (clone) of a reference-type instance, note that there is no universal mechanism for creating one:
You can create copies of instances of types if they implement the System.ICloneable interface by calling their .Clone() method, but note that it is up to the implementing type whether to perform shallow or deep cloning[1]; it is for that reason that use of this interface is discouraged; in practice, types that do implement it typically perform shallow cloning, notably arrays, array lists (System.Collections.ArrayList) and hashtables (but note that an [ordered] hashtable (System.Collections.Specialized.OrderedDictionary) doesn't implement ICloneable at all.
Additionally, in PowerShell, you can call .psobject.Copy() on instances of type [pscustomobject] to create a shallow copy. (Do not use this method on objects of any other type, where it will effectively be a no-op.) Similarly, individual .NET types may implement custom cloning methods.
If, by contrast, that content is an instance of a .NET value type - e.g., [int] - or a
string[2], an independent copy of that instance is passed.
This distinction is fundamental to .NET, not specific to PowerShell; it is also how arguments are passed in C#, for instance.
To determine whether a given variable's value is an instance of a value type or a reference type, use something like the following:
1, (Get-Date), (Get-Item /) | # sample values
foreach {
'{0} is of type {1}; is it a value type? {2}' -f $_,
$_.GetType(),
$_.GetType().IsValueType
}
You'll see something like:
1 is of type System.Int32; is it a value type? True
4/30/2020 12:37:01 PM is of type System.DateTime; is it a value type? True
/ is of type System.IO.DirectoryInfo; is it a value type? False
If you look up the documentation for a given .NET type, say System.DateTime, the inheritance information will start with Object -> ValueType for value types; in C# terms, a value type is either a struct or an enum, whereas a reference type is a class.
Terminology and concepts:
There are two unrelated concepts at play here, and the fact that they both use the terms (by-)value and (by-)reference can get confusing:
By-(variable)-value vs. by-(variable)-reference parameter-passing is a data-holder (variable) concept:
It describes whether, on parameter passing, a variable's value is passed (by value) or a reference to the variable itself[3] (by reference).
Reference types vs. value types is purely a data concept:
That is, for technical reasons, any object in .NET is either an instance of a value type (stored on the stack) or a reference type (stored on the heap). Instances of the former are directly stored in variables, whereas the latter are stored by way of a reference. Therefore, copying a variable value - e.g., in the context of by-value parameter-passing - means:
either: making a copy of a value-type instance itself, resulting in an independent data copy.
or: making a copy of a reference-type instance reference; a copy of a reference still points to the same object, however, which is why even by-variable-value passed reference-type instances are directly seen by the callee (by way of their reference copy).
[1] Shallow cloning means that property values that are reference-type instances are copied as-is - as references - which means that the clone's property value again references the very same object as the original. Deep cloning means that such property values are cloned themselves, recursively. Deep cloning is expensive and isn't always possible.
[2] A string ([string]) is technically also an instance of a reference type, but, as an exception, it is treated like a value type; see this answer for the rationale behind this exception.
[3] Another way of thinking about it: a reference (pointer) to the location where the variable stores its value is passed. This allows the callee to not only access the variable's value, but also to assign a (new) value.

Pass strings by reference in Powershell?

How can I pass strings by reference to the parent scope?
This doesn't work since strings are not acceptable "values".
function Submit([ref]$firstName){
$firstName.value = $txtFirstName.Text
}
$firstName = $null
Submit([ref]$firstName)
$firstName
Error: "Property 'value' cannot be found on this object; make sure it exists and is settable"
Doing this doesn't give an error but it doesn't change the variable either:
$firstName = "nothing"
function Submit([ref]$firstName){
$firstName = $txtFirstName.Text
}
Submit([ref]$firstName)
$firstName
Edit:
Doing the first code block by itself works. However when trying to do it in my script it returns the error again. I fixed it enough for it to assign the variable and do what I want but it still throws up an error and I was wondering how to fix that. I think it's because it doesn't like variable;es changing during a running session. Here is a link to my script
https://github.com/InconspicuousIntern/Form/blob/master/Form.ps1
Your first snippet is conceptually correct and works as intended - by itself it does not produce the "Property 'Value' cannot be found on this object" error.
You're seeing the error only as part of the full script you link to, because of the following line:
$btnSubmit.Add_Click({ Submit })
This line causes your Submit function to be called without arguments, which in turn causes the $firstName parameter value to be $null, which in turn causes the error quoted above when you assign to $firstName.Value.
By contrast, the following invocation of Submit, as in your first snippet, is correct:
Submit ([ref] $firstName) # Note the recommended space after 'Submit' - see below.
[ref] $firstName creates a (transient) reference to the caller's $firstName variable, which inside Submit binds to (local) parameter variable $firstName (the two may, but needn't and perhaps better not have the same name), where $firstName.Value can then be used to modify the caller's $firstName variable.
Syntax note: I've intentionally placed a space between Submit and ([ref] $firstName) to make one thing clearer:
The (...) (parentheses) here do not enclose the entire argument list, as they would in a method call, they enclose the single argument [ref] $firstName - of necessity, because that expression wouldn't be recognized as such otherwise.
Function calls in PowerShell are parsed in so-called argument mode, whose syntax is more like that of invoking console applications: arguments are space-separated, and generally only need quoting if they contain special characters.
For instance, if you also wanted to pass string 'foo', as the 2nd positional parameter, to Submit:
Submit ([ref] $firstName) foo
Note how the two arguments are space-separated and how foo needn't be quoted.
As for an alternative approach:
[ref]'s primary purpose is to enable .NET method calls that have ref / out parameters, and, as shown above, using [ref] is nontrivial.
For calls to PowerShell functions there are generally simpler solutions.
For instance, you can pass a custom object to your function and let the function update its properties with the values you want to return, which naturally allows multiple values to be "returned"; e.g.:
function Submit($outObj){
$outObj.firstName = 'a first name'
}
# Initialize the custom object that will receive values inside
# the Submit function.
$obj = [pscustomobject] #{ firstName = $null }
# Pass the custom object to Submit.
# Since a custom object is a reference type, a *reference* to it
# is bound to the $outObj parameter variable.
Submit $obj
$obj.firstName # -> 'a first name'
Alternatively, you can just let Submit construct the custom object itself, and simply output it:
function Submit {
# Construct and (implicitly) output a custom
# object with all values of interest.
[pscustomobject] #{
firstName = 'a first name'
}
}
$obj = Submit
$obj.firstName # -> 'a first name'
Please try this out and see if you are getting the same results? It is working for me, and I really did not change much.
$txtFirstName = [PSCustomObject]#{
Text = "Something"
}
function Submit([ref]$var){
$var.value = $txtFirstName.Text
}
$firstName = $null
Submit([ref]$firstName)
$firstName

Setting lua table in redis

I have a lua script, which simplified is like this:
local item = {};
local id = redis.call("INCR", "counter");
item["id"] = id;
item["data"] = KEYS[1]
redis.call("SET", "item:" .. id, cjson.encode(item));
return cjson.encode(item);
KEYS[1] is a stringified json object:
JSON.stringify({name : 'some name'});
What happens is that because I'm using cjson.encode to add the item to the set, it seems to be getting stringified twice, so the result is:
{"id":20,"data":"{\"name\":\"some name\"}"}
Is there a better way to be handling this?
First, regardless your question, you're using KEYS the wrong way and your script isn't written according to the guidelines. You should not generate key names in your script (i.e. call SET with "item:" .. id as a keyname) but rather use the KEYS input array to declare any keys involved a priori.
Secondly, instead of passing the stringified string with KEYS, use the ARGV input array.
Thirdly, you can do item["data"] = json.decode(ARGV[1]) to avoid the double encoding.
Lastly, perhaps you should learn about Redis' Hash data type - it may be more suitable to your needs.

Problems parsing Reddit's JSON

I'm working on a perl script that parses reddit's JSON using the JSON module.
However I do have the problem of being very new to both perl and json.
I managed to parse the front page and subreddits successfully, but the comments have a different structure and I can't figure out how to access the data I need.
Here's the code that successfully finds the "data" hash for the front page and subreddits:
foreach my $children(#{$json_text->{"data"}->{"children"}}) #For values of children.
{
my $data = $children->{"data"}; #accessing each data hash.
my %phsh = (); #my hash to collect and print.
$phsh{author} = $data->{"author"};#Here I get the "author" value from "data"
*Etc....
This successfully gets what I need from http://www.reddit.com/.json
But when I go to the json of a comment, this one for example, it has a different format and I can't figure out how to parse it. If I try the same thing as before my parser crashes, saying it is not a HASH reference.
So my question is: How do access the "children" in the second JSON? I need to get both the data for the Post and the data for the comments. Can anybody help?
Thanks in advance!
(I know it may be obvious, but I'm running on very little sleep XD)
You need to either look at the JSON data or dump the decoded data to see what form it takes. The comment data, for example is an array at the top level.
Here is some code that prints the body field of all top-level comments. Note that a comment may have an array of replies in its replies field, and each reply may also have replies in turn.
Depending on what you want to do you may need to check whether a reference is to an array or a hash by checking the value returned by the ref operator.
use strict;
use warnings;
binmode STDOUT, ':utf8';
use JSON;
use LWP;
use Data::Dump;
my $ua = LWP::UserAgent->new;
my $resp = $ua->get('http://www.reddit.com/r/funny/comments/wx3n5/caption_win.json');
die $resp->status_line unless $resp->is_success;
my $json = $resp->decoded_content;
my $data = decode_json($json);
die "Error: $data->{error}" if ref $data eq 'HASH' and exists $data->{error};
dd $data->[1]{data}{children}[0];
print "\n\n";
my $children = $data->[1]{data}{children};
print scalar #$children, " comments:\n\n";
for my $child (#$children) {
print $child->{data}{body}, "\n";
}