Deserialize JSON without knowing full structure - json

I'm redoing the backend of a very basic framework that connects to a completely customizable frontend. It was originally in PHP but for the refactor have been plodding away in F#. Although it seems like PHP might be the more suited language. But people keep telling me you can do everything in F# and I like the syntax and need to learn and this seemingly simple project has me stumped when it comes to JSON. This is a further fleshed out version of my question yesterday, but it got alot more complex than I thought.
Here goes.
The frontend is basically a collection of HTML files, which are simply loaded in PHP and preg_replace() is used to replace things like [var: varName] or [var: array|key] or the troublesome one: [lang: hello]. That needs to be replaced by a variable defined in a translation dictionary, which is stored as JSON which is also editable by a non-programmer.
I can't change the frontend or the JSON files, and both are designed to be edited by non-programmers so it is very likely that there will be errors, calls to language variables that don't exist etc.
So we might have 2 json files, english.json and french.json
english.json contains:
{
"hello":"Hello",
"bye":"Goodbye"
}
french.json:
{
"hello": "Bonjour",
"duck": "Canard"
//Plus users can add whatever else they want here and expect to be able to use it in a template
}
There is a template that contains
<b>[lang: hello]</b>
<span>Favourite Animal: [lang:duck]</span>
In this case, if the language is set to "english" and english.json is being loaded, that should read:
<b>Hello</b>
<span>Favourite Animal: </span>
Or in French:
<b>Bonjour</b>
<span>Favourite Animal: Canard</span>
We can assume that the json format key: value is always string:string but ideally I'd like to handle string: 'T as well but that might be beyond the scope of this question.
So I need to convert a JSON file (called by dynamic name, which gave F# Data a bit of an issue I couldn't solve last night as it only allowed a static filename as a sample, and since these two files have potential to be different from sample and provided, the type provider doesn't work) to a dictionary or some other collection.
Now inside the template parsing function I need to replace [lang: hello] with something like
let key = "duck"
(*Magic function to convert JSON to usable collection*)
let languageString = convertedJSONCollection.[key] (*And obviously check if containsKey first*)
Which means I need to call the key dynamically, and I couldn't figure out how to do that with the type that FSharp.Data provided.
I have played around with some Thoth as well to some promising results that ended up going nowhere. I avoided JSON.NET because I thought it was paid, but just realised I am mistaken there so might be an avenue to explore
For comparison, the PHP function looks something like this:
function loadLanguage($lang='english){
$json = file_get_contents("$lang.json");
return json_decode($json, true);
}
$key = 'duck';
$langVars = loadLanguage();
$duck = $langVars[$key] || "";
Is there a clean way to do this in F#/.NET? JSON seems really painful to work with in comparison to PHP/Javascript and I'm starting to lose my mind. Am I going to have to write my own parser (which means probably going back to PHP)?
Cheers to all you F# geniuses who know the answer :p

open Thoth.Json.Net
let deserialiseDictionary (s: string) =
s
|> Decode.unsafeFromString (Decode.keyValuePairs Decode.string)
|> Map.ofList
let printDictionary json =
json
|> deserialiseDictionary
|> fun m -> printfn "%s" m.["hello"] // Hello
For the question about 'T the question becomes, what can 'T be? For json it very limited, it can be a number of things, string, json-object, number, bool or json array. What should happen if it is bool or a number?

Related

Convert terraform resourceData of type map[string]interface{} to struct

I'm creating a custom terraform provider and I came across this issue.
I was trying to convert a schema.TypeList field into a struct, the TypeList looks something like this:
"template": {
Type: schema.TypeList,
Required: true,
ForceNew: false,
Elem: &schema.Resource{
Schema: map[string]*schema.Schema{
"lists_test": {
Type: schema.TypeSet,
Required: true,
ForceNew: false,
Elem: &schema.Schema{
Type: schema.TypeString,
},
},
"name_test": {
Type: schema.TypeString,
Required: true,
ForceNew: false,
},
},},
and the struct that I'm trying to align to looks something like this:
type TestStruct struct {
NameTest string `json:"name_test"`
ListsTests []string `json:"lists_test"`
}
I tried a couple of solutions, for instance I tried unmarshalling it to json. Something like below:
template := d.Get("template").([]interface{})[0].(map[string]interface{})
templateStr, err := json.Marshal(template)
templateConverted := &TestStruct{}
json.Unmarshal(template, templateConverted)
however, I'm getting an error json: unsupported type: SchemaSetFunc, which is probably because it's trying to marshal a schema.Schema type instead of map[string]interface{} type, which confuses me. I also tried to use gohcl.DecodeBody but I abandoned the idea since it's usage seems more inclined into reading direct tf files rather than *schema.ResourceData types.
Does anyone had the same experience dealing with this type of scenario? Any help or suggestion is appreciated. Thank you!
Terraform's older SDK (SDKv2) is not designed around the paradigm of decoding into a tagged structure, and instead expects you to use d.Get and manually type-assert individual values, which in your case would perhaps look something like this:
raw := d.Get("template").([]interface{})[0].(map[string]interface{})
t := &TestStruct{
NameTest: raw["name_test"].(string),
ListsTests: make([]string, len(raw["lists_test"].([]interface{})),
}
for i, itemRaw := range raw["lists_test"].([]interface{}) {
t.ListsTests[i] = itemRaw.(string)
}
The idiomatic style for most Terraform providers is to write logic like this in separate functions for each complex-typed attribute, where each returns an object of the appropriate type in the target platform's SDK. There would typically also be a matching function for going in the opposite direction: given an object from the target platform's SDK, return a map[string]interface{} that can be assigned to this attribute using d.Set.
However, just because there isn't something built in to the SDK to handle this, that doesn't mean you can't use other libraries that are more general utilities for use in any Go programs.
One example library is github.com/mitchellh/mapstructure, which is designed for exactly the goal you have in mind: to take a value of some interface type and try to use reflection to fit it onto a tagged structure type.
If you want to use that library then you would need to annotate your structure with mapstructure:, instead of the json: ones, and then pass your raw value to the mapstructure.Decode function:
raw := d.Get("template").([]interface{})[0].(map[string]interface{})
var t TestStruct
err := mapstructure.Decode(raw, &t)
Since the schema.ResourceData abstraction in SDKv2 guarantees to return specific data types based on the schema you defined, you should not typically get errors from mapstructure.Decode as long as your schema and your target type match, but still a good idea to check for errors anyway because otherwise your t value may not be completely populated, causing confusing broken behavior downstream.
This is not a typical implementation style used in the official providers, but there's no real harm in writing your provider in this way if you find this style more convenient, or easier to maintain.
Alternatively, if you are not already deeply invested in SDKv2 then you may wish to consider using Terraform Plugin Framework instead. As well as being designed around the type system of modern Terraform (whereas SDKv2 was designed for Terraform v0.11 and earlier), it also supports a programming style more like what you are aiming for, with methods like tfsdk.Plan.Get and tfsdk.Plan.GetAttribute that can decode directly into an appropriately-shaped and appropriately tagged "normal" Go value.
I can't easily show an example of that because it would presume a provider written in quite a different way, but hopefully you can see from the signature of those two functions how they might be used. There's some more commentary and examples in Accessing State, Config, and Plan.

Accesing Json data after 'loading' it

With a lot of help from people in this site, I managed to get some Json data from an amazon page. The data, for example, looks like this.
https://jsoneditoronline.org/?id=9ea92643044f4ac88bcc3e76d98425fc
First I have a list of strings which is converted to a string.
script = response.xpath('//script/text()').extract()
#For example, I need the variationValues data
variationValues = re.findall(r'variationValues\" : ({.*?})', ' '.join(script))[0]
Then, in my code, I have this (not a great name, will be changed later)
variationValuesJson = json.loads(variationValues)
variationValuesJson is in fact a dictionary, so doing something like this
variationValues["size_name"][3]
Should return "5.5 M US"
My issue is that, when running the program, I get the string indices must be integers error. Anyone knows whats wrong?
Note: I have tried using 'size_name' instead of "size_name", same error
variationValues["size_name"][3] #this is the raw string which you have converted to variationValuesjson
I think this is not what you actually want.
Your code should be this.
variationValuesJson['size_name'][3] #use variationValuesjson ;)

How can I iterate Dynamic object in Haxe

I have Object parsed from JSON (haxe.Json.parse()) and I need to iterate over it.
I already tried to cast this object to Array<Dynamic>:
var data:String='{"data":{"0":0,"1":1},"method":"test"}';
var res:{method:String,data:Array<Dynamic>} = haxe.Json.parse(data);
for (n in res.data)
trace('aa')
There is no Can't iterate dynamic exception, just not working (iterating).
I completley don't understand why in Haxe iterating procedure is so difficult.
For the sake of posting a complete answer, and in case other people are wondering
In your first example, you've told the compiler that "res" contains two properties - one called "method" (which is a String) and one called "data" (which is Array). Now the JSON you're using doesn't actually have an Array<Dynamic>, it just has a dynamic object. An Array would look like: "data":[0,1].
So, assuming you meant for the JSON to have data as a Dynamic object, here is how you loop over it, using Reflect (as you mentioned in the comments):
var data:String='{"data":{"0":0,"1":1},"method":"test"}';
var res = haxe.Json.parse(data);
for (n in Reflect.fields(res.data))
trace(Reflect.field(res.data, n));
Note here we don't have to specify the type of "res", since we're using Reflection just leaving it as Dynamic will be fine.
Now, if your JSON actually contains an Array, the code might look like this:
var data:String='{"data":[0,1],"method":"test"}';
var res:{method:String,data:Array<Int>} = haxe.Json.parse(data);
for (n in res.data)
trace(n);
Here you use explicit typing to tell the compiler that res.data is an Array (and this time it actually is), and it can loop over it normally.
The reason you didn't get an error at compile-time is because the compiler thought there was genuinely going to be an array there, as you told it there was. At runtime, whether or not it throws an exception probably depends on the target... but you probably want to stay out of that anyway :)
Demo of both styles of code: http://try.haxe.org/#772A2

Using JSON in Haskell to Serialize a Record

I've been working on a very small program to grab details about Half Life 2 servers (using the protocol-srcds library). The workflow is pretty straightforward; it takes a list of servers from a file, queries each of them, and writes the output out to another file (which is read by a PHP script for display, as I'm tied to vBulletin). Would be nice if it was done in SQL or something, but seeing as I'm still just learning, that's a step too far for now!
Anyway, my question relates to serialization, namely, serializing to JSON. For now, I've written a scrappy helper function jsonify, such that:
jsonify (Just (SRCDS.GameServerInfo serverVersion
serverName
serverMap
serverMod
serverModDesc
serverAppId
serverPlayers
serverMaxPlayers
serverBots
serverType
serverOS
serverPassword
serverSecure
serverGameVersioning)) =
toJSObject [ ("serverName", serverName)
, ("serverMap", serverMap)
, ("serverPlayers", show serverPlayers)
, ("serverMaxPlayers", show serverMaxPlayers) ]
(I'm using the Text.JSON package). This is obviously not ideal. At this stage, however, I don't understand using instances to define serializers for records, and my attempts to do so met a wall of frustration in the type system.
Could someone please walk me through the "correct" way of doing this? How would I go about defining an instance that serializes the record? What functions should I use in the instance (showJSON?).
Thanks in advance for any help.
You might want to consider using Data.Aeson instead which might be regarded as the successor to Text.JSON.
With aeson you define separate instances for serialize/deserializing (with Text.JSON you have to define both directions even if you need only one, otherwise the compile will annoy you -- unless you silence the warning somehow), and it provides a few operators making defining instances a bit more compact, e.g. the example from #hammar's answer can be written a little bit less noisy as shown below with the aeson API:
instance ToJSON SRCDS.GameServerInfo where
toJSON (SRCDS.GameServerInfo {..}) = object
[ "serverName" .= serverName
, "serverMap" .= serverMap
, "serverPlayers" .= serverPlayers
, "serverMaxPlayers" .= serverMaxPlayers
]
One simple thing you can do is use record wildcards to cut down on the pattern code.
As for your type system problems, it's hard to give help without seeing error messages and what you've tried so far, however I suspect one thing that might be confusing is that the result of toJSObject will have to be wrapped in a JSObject data constructor as the return type of showJSON is supposed to be a JSValue. Similarly, the values of your object should also be of type JSValue. The easiest way to do this is to use their JSON instance and call showJSON to convert the values.
instance JSON SRCDS.GameServerInfo where
showJSON (SRCDS.GameServerInfo {..}) =
JSObject $ toJSObject [ ("serverName", showJSON serverName)
, ("serverMap", showJSON serverMap)
, ("serverPlayers", showJSON serverPlayers)
, ("serverMaxPlayers", showJSON serverMaxPlayers) ]

Is it okay to rely on automatic pass-by-reference to mutate objects?

I'm working in Python here (which is actually pass-by-name, I think), but the idea is language-agnostic as long as method parameters behave similarly:
If I have a function like this:
def changefoo(source, destination):
destination["foo"] = source
return destination
and call it like so,
some_dict = {"foo": "bar"}
some_var = "a"
new_dict = changefoo(some_var, some_dict)
new_dict will be a modified version of some_dict, but some_dict will also be modified.
Assuming the mutable structure like the dict in my example will almost always be similarly small, and performance is not an issue (in application, I'm taking abstract objects and changing into SOAP requests for different services, where the SOAP request will take an order of magnitude longer than reformatting the data for each service), is this okay?
The destination in these functions (there are several, it's not just a utility function like in my example) will always be mutable, but I like to be explicit: the return value of a function represents the outcome of a deterministic computation on the parameters you passed in. I don't like using out parameters but there's not really a way around this in Python when passing mutable structures to a function. A couple options I've mulled over:
Copying the parameters that will be mutated, to preserve the original
I'd have to copy the parameters in every function where I mutate them, which seems cumbersome and like I'm just duplicating a lot. Plus I don't think I'll ever actually need the original, it just seems messy to return a reference to the mutated object I already had.
Just use it as an in/out parameter
I don't like this, it's not immediately obvious what the function is doing, and I think it's ugly.
Create a decorator which will automatically copy the parameters
Seems like overkill
So is what I'm doing okay? I feel like I'm hiding something, and a future programmer might think the original object is preserved based on the way I'm calling the functions (grabbing its result rather than relying on the fact that it mutates the original). But I also feel like any of the alternatives will be messy. Is there a more preferred way? Note that it's not really an option to add a mutator-style method to the class representing the abstract data due to the way the software works (I would have to add a method to translate that data structure into the corresponding SOAP structure for every service we send that data off too--currently the translation logic is in a separate package for each service)
If you have a lot of functions like this, I think your best bet is to write a little class that wraps the dict and modifies it in-place:
class DictMunger(object):
def __init__(self, original_dict):
self.original_dict = original_dict
def changefoo(source)
self.original_dict['foo'] = source
some_dict = {"foo": "bar"}
some_var = "a"
munger = DictMunger(some_dict)
munger.changefoo(some_var)
# ...
new_dict = munger.original_dict
Objects modifying themselves is generally expected and reads well.