There appears to be few options to validate the source JSON used when unmarshalling to a struct. By validate I mean 3 main things:
a required field exists in the JSON
the field is the correct type (e.g. don't force a string into an integer)
the field contains a valid value (value range / enum)
For nested structs, I simply mean where an attribute in one struct has the type of another struct:
type Example struct {
Attr1 int `json:"attr1"`
Attr2 ExampleToo `json:"attr2"`
}
type ExampleToo struct {
Attr3 int `json:"attr3"`
}
And this JSON would be valid:
{"attr1": 5, "attr2": {"attr3": 0}}
To keep this simple, I'll focus simply on integers. The concept of "zero values" is the first issue. I could create an UnmarshalJSON method, which is detected by JSON packages, including the standard encoding/json package. The problem with this approach is that is that is does not support nested structs. If ExampleToo has an UnmarshalJSON method, the ExampleToo.UnmarshalJSON() method is never called if unmarshalling to an Example object. It would be possible to write a method Example.UnmarshalJSON() that recursively handled validation, but that seems extremely complex, especially if ExampleToo is reused in many places.
So there appears to be some packages like the go-playground/validator where validation can be specified both as functions and tags. However, this works on the struct created, and not the JSON itself. So if a field is tagged as validation:"required" on an integer, and the integer value is 0, this will return an error because 0 is both a valid value and the "zero value" for integers.
An example of the latter here: https://go.dev/play/p/zqSUksPzUiq
I could also use pointers for everything, checking for nil as missing values. The main problem with that is that it requires dereferencing on each use and is a pretty uncommon practice for things like integers and strings.
One thing that I have also considered is a "sister struct" that uses pointers to do validation for required fields. The process would basically be to write a validation method for each struct, then validate that sister struct. If it works, then deserialize the main struct (without pointers). I haven't started on this, just a concept I've thought about, but I'm hoping there are better validation options.
So... is there a better way to do JSON/YAML input validation on nested structs? I'm happy to mix methods where say UnmarshalJSON is used for doing some work like verifying fields exist, but I'd like to pass that back to the library to let it continue to call UnmarshalJSON on subsequent nested structs. I'd also rather defer to the JSON library for casting values into the struct, etc.
I have an array of structs that has been decoded from a JSON file. Each struct has a stored property that is a variable-dimension array stored in an enum associated value.
Some of the structs have property Enum.array2D([[Float]]) and others have Enum.array3D([[[Float]]])
Is there a simple or elegant way to extract a variable-type associated value from the struct's enum property - maybe with a getter function? Currently, the only way I know how to do this is with an external switch anytime I want to access the underlaying value. For example, somewhere in external code I have to use this anytime I want to get these values and manipulate them:
switch structArray[index].enumProperty {
case .array2D(let array2Val):
// Do stuff with the 2D array
case .array3D(let array3Val):
// Do stuff with the 3D array
}
I have considered adding each of the two possible types as optionals and setting the correct one in the init function with a switch, but that seems inefficient as I’ll have the arrays stored in two places.
I making a foray into the world of JSON parsing and NewtonSoft and I'm confused, to say the least.
Take the below PowerShell script:
$json = #"
{
"Array1": [
"I am string 1 from array1",
"I am string 2 from array1"
],
"Array2": [
{
"Array2Object1Str1": "Object in list, string 1",
"Array2Object1Str2": "Object in list, string 2"
}
]
}
"#
#The newtonSoft way
$nsObj = [Newtonsoft.Json.JsonConvert]::DeserializeObject($json, [Newtonsoft.Json.Linq.JObject])
$nsObj.GetType().fullname #Type = Newtonsoft.Json.Linq.JObject
$nsObj[0] #Returns nothing. Why?
$nsObj.Array1 #Again nothing. Maybe because it contains no key:value pairs?
$nsObj.Array2 #This does return, maybe because has object with kv pairs
$nsObj.Array2[0].Array2Object1Str1 #Returns nothing. Why? but...
$nsObj.Array2[0].Array2Object1Str1.ToString() #Cool. I get the string this way.
$nsObj.Array2[0] #1st object has a Path property of "Array2[0].Array2Object1Str1" Great!
foreach( $o in $nsObj.Array2[0].GetEnumerator() ){
"Path is: $($o.Path)"
"Parent is: $($o.Parent)"
} #??? Why can't I see the Path property like when just output $nsObj.Array2[0] ???
#How can I find out what the root parent (Array2) is for a property? Is property even the right word?
I'd like to be able to find the name of the root parent for any given position. So above, I'd like to know that the item I'm looking at (Array2Object1Str1) belongs to the Array2 root parent.
I think I'm not understanding some fundamentals here. Is it possible to determine the root parent? Also, any help in understanding my comments in the script would be great. Namely why I can't return things like path or parent, but can see it when I debug in VSCode.
dbc's answer contains helpful background information, and makes it clear that calling the NewtonSoft Json.NET library from PowerShell is cumbersome.
Given PowerShell's built-in support for JSON parsing - via the ConvertFrom-Json and ConvertTo-Json cmdlets - there is usually no reason to resort to third-party libraries (directly[1]), except in the following cases:
When performance is paramount.
When the limitations of PowerShell's JSON parsing must be overcome (lack of support for empty key names and keys that differ in letter case only).
When you need to work with the Json.NET types and their methods rather than with the method-less "property-bag" [pscustomobject] instances ConvertFrom-Json constructs.
While working with NewtonSoft's Json.NET directly in PowerShell is awkward, it is manageable, if you observe a few rules:
Lack of visible output doesn't necessarily mean that there isn't any output at all:
Due to a bug in PowerShell (as of v7.0.0-preview.4), [JValue] instances and [JProperty] instances containing them produce no visible output by default; access their (strongly typed) .Value property instead (e.g., $nsObj.Array1[0].Value or $nsProp.Value.Value (sic))
To output the string representation of a [JObject] / [JArray] / [JProperty] / [JValue] instance, do not rely on output as-is (e.g, $nsObj), use explicit stringification with .ToString() (e.g., $nsObj.ToString()); while string interpolation (e.g., "$nsObj") does generally work, it doesn't with [JValue] instances, due to the above-mentioned bug.
[JObject] and [JArray] objects by default show a list of their elements' instance properties (implied Format-List applied to the enumeration of the objects); you can use the Format-* cmdlets to shape output; e.g., $nsObj | Format-Table Path, Type.
Due to another bug (which may have the same root cause), as of PowerShell Core 7.0.0-preview.4, default output for [JObject] instances is actually broken in cases where the input JSON contains an array (prints error format-default : Target type System.Collections.IEnumerator is not a value type or a non-abstract class. (Parameter 'targetType')).
To numerically index into a [JObject] instance, i.e. to access properties by index rather than by name, use the following idiom: #($nsObj)[<n>], where <n> is the numerical index of interest.
$nsObj[<n>] actually should work, because, unlike C#, PowerShell exposes members implemented via interfaces as directly callable type members, so the numeric indexer that JObject implements via the IList<JToken> interface should be accessible, but isn't, presumably due to this bug (as of PowerShell Core 7.0.0-preview.4).
The workaround based on #(...), PowerShell's array-subexpression operator, forces enumeration of a [JObject] instance to yield an array of its [JProperty] members, which can then be accessed by index; note that this approach is simple, but not efficient, because enumeration and construction of an aux. array occurs; however, given that a single JSON object (as opposed to an array) typically doesn't have large numbers of properties, this is unlikely to matter in practice.
A reflection-based solution that accesses the IList<JToken> interface's numeric indexer is possible, but may even be slower.
Note that additional .Value-based access may again be needed to print the result (or to extract the strongly typed property value).
Generally, do not use the .GetEnumerator() method; [JObject] and [JArray] instances are directly enumerable.
Keep in mind that PowerShell may automatically enumerate such instances in contexts where you don't expect it, notably in the pipeline; notably, when you send a [JObject] to the pipeline, it is its constituent [JProperty]s that are sent instead, individually.
Use something like #($nsObj.Array1).Value to extract the values of an array of primitive JSON values (strings, numbers, ...) - i.e, [JValue] instances - as an array.
The following demonstrates these techniques in context:
$json = #"
{
"Array1": [
"I am string 1 from array1",
"I am string 2 from array1",
],
"Array2": [
{
"Array2Object1Str1": "Object in list, string 1",
"Array2Object1Str2": "Object in list, string 2"
}
]
}
"#
# Deserialize the JSON text into a hierarchy of nested objects.
# Note: You can omit the target type to let Newtonsoft.Json infer a suitable one.
$nsObj = [Newtonsoft.Json.JsonConvert]::DeserializeObject($json)
# Alternatively, you could more simply use:
# $nsObj = [Newtonsoft.Json.Linq.JObject]::Parse($json)
# Access the 1st property *as a whole* by *index* (index 0).
#($nsObj)[0].ToString()
# Ditto, with (the typically used) access by property *name*.
$nsObj.Array1.ToString()
# Access a property *value* by name.
$nsObj.Array1[0].Value
# Get an *array* of the *values* in .Array1.
# Note: This assumes that the array elements are JSON primitives ([JValue] instances.
#($nsObj.Array1).Value
# Access a property value of the object contained in .Array2's first element by name:
$nsObj.Array2[0].Array2Object1Str1.Value
# Enumerate the properties of the object contained in .Array2's first element
# Do NOT use .GetEnumerator() here - enumerate the array *itself*
foreach($o in $nsObj.Array2[0]){
"Path is: $($o.Path)"
"Parent is: $($o.Parent.ToString())"
}
[1] PowerShell Core - but not Windows PowerShell - currently (v7) actually uses NewtonSoft's Json.NET behind the scenes.
You have a few separate questions here:
$nsObj[0] #Returns nothing. Why?
This is because nsObj corresponds to a JSON object, and, as explained in this answer to How to get first key from JObject?, JObject does not directly support accessing properties by integer index (rather than property name).
JObject does, however, implement IList<JToken> explicitly so if you could upcast nsObj to such a list you could access properties by index -- but apparently it's not straightforward in PowerShell to call an explicitly implemented method. As explained in the answers to How can I call explicitly implemented interface method from PowerShell? it's necessary to do this via reflection.
First, define the following function:
Function ChildAt([Newtonsoft.Json.Linq.JContainer]$arg1, [int]$arg2)
{
$property = [System.Collections.Generic.IList[Newtonsoft.Json.Linq.JToken]].GetProperty("Item")
$item = $property.GetValue($nsObj, #([System.Object]$arg2))
return $item
}
And then you can do:
$firstItem = ChildAt $nsObj 0
Try it online here.
#??? Why can't I see the Path property like when just output $nsObj.Array2[0] ???
The problem here is that JObject.GetEnumerator() does not return what you think it does. Your code assumes it returns the JToken children of the object, when in fact it is declared as
public IEnumerator<KeyValuePair<string, JToken>> GetEnumerator()
Since KeyValuePair<string, JToken> doesn't have the properties Path or Parent your output method fails.
JObject does implement interfaces like IList<JToken> and IEnumerable<JToken>, but it does so explicitly, and as mentioned above calling the relevant GetEnumerator() methods would require reflection.
Instead, use the base class method JContainer.Children(). This method works for both JArray and JObject and returns the immediate children in document order:
foreach( $o in $nsObj.Array2[0].Children() ){
"Path is: $($o.Path)"
"Parent is: $($o.Parent)"
}
Try it online here.
$nsObj.Array1 #Again nothing. Maybe because it contains no key:value pairs?
Actually this does return the value of Array1, if I do
$nsObj.Array1.ToString()
the JSON corresponding to the value of Array1 is displayed. The real issue seems to be that PowerShell doesn't know how to automatically print a JArray with JValue contents -- or even a simple, standalone JValue. If I do:
$jvalue = New-Object Newtonsoft.Json.Linq.JValue 'my jvalue value'
'$jvalue' #Nothing output
$jvalue
'$jvalue.ToString()' #my jvalue value
$jvalue.ToString()
Then the output is:
$jvalue
$jvalue.ToString()
my jvalue value
Try it online here and, relatedly, here.
Thus the lesson is: when printing a JToken hierarchy in PowerShell, always use ToString().
As to why printing a JObject produces some output while printing a JArray does not, I can only speculate. JToken implements the interface IDynamicMetaObjectProvider which is also implemented by PSObject; possibly something about the details of how this is implemented for JObject but not JValue or JArray are compatible with PowerShell's information printing code.
I have around 5 functions which exchange data among each other, each fuction has to return an array and around two integers. I went through "tuple" but we cant club integers and arrays together and im not very comfortable with "struts".
If i could please be guided of ways of doing the same or passing and returning two arrays ( the integers can be put in an array).
Thank You
returning and passing 2 one dimensional arrays
In C++, you can only return a single value. You cannot return multiple values, and the value that you return cannot be an array.
im not very comfortable with "struts" [sic]
I assume you mean structs. Well, now is the time to become comfortable, because a struct (also known as class) is great way to combine multiple values - even arrays - into a single object that can be returned.
Another option is to pass the function multiple references (or iterators or pointers) to objects that the function can modify instead of returning them.
I'm trying to create a json with multiple records by following this example: Generate a sample JSON with an array in it in Delphi XE5
must be the same way, except that when I add the array to the object
JSonObj.AddPair (TJSONPair.Create ('records', TJSONArray));
returns the error:
"There is the overloaded version of 'Create' that can be called with arguments These"
How do I add to the array object?
If I convert an array to string and add, to receive the amounts can not treat as an array ...
You're passing it the class reference for a JSON array. You need to pass it an instance.
arr := TJSONArray.Create;
JSONObj.AddPair(TJSONPair.Create('records', arr));
Look carefully at the answers in the question you link to, and you'll see this is exactly what they're doing, too.