I need this result of JSON marshal:
["a", "b", ["c", "d"], "e"]
How correctly to do this in Go?
The trick to creating a slice/array of mixed types is to use the empty interface type that go offers
inner := []string{"c", "d"}
all := []interface{}{"a", "b", inner, "e"}
And then just json.Marshal the interface slice. This works because any and all values implement at least an empty interface. You can use the most bloated object as though it doesn't have any methods/receiver functions to call. That's why people sometimes refer to go's interface{} type as being a "generic" type. A lot of abstract stuff that packages do use interface{} arguments, and then reflection to work out what was actually passed. Just look at the source of the encoding/json package and see how it unmarshals values...
https://play.golang.org/p/96VVo2F1m7
Unmarshalling would work in pretty much the same way: you can unmarshal JSON strings into interface{} variables. You may need to use type-assertions and/or reflection to be able to work with the data though. That's why it's probably best to create types...
You ca use a slice of interface{}, for example https://play.golang.org/p/c8dPRPK6mr
Related
There appears to be few options to validate the source JSON used when unmarshalling to a struct. By validate I mean 3 main things:
a required field exists in the JSON
the field is the correct type (e.g. don't force a string into an integer)
the field contains a valid value (value range / enum)
For nested structs, I simply mean where an attribute in one struct has the type of another struct:
type Example struct {
Attr1 int `json:"attr1"`
Attr2 ExampleToo `json:"attr2"`
}
type ExampleToo struct {
Attr3 int `json:"attr3"`
}
And this JSON would be valid:
{"attr1": 5, "attr2": {"attr3": 0}}
To keep this simple, I'll focus simply on integers. The concept of "zero values" is the first issue. I could create an UnmarshalJSON method, which is detected by JSON packages, including the standard encoding/json package. The problem with this approach is that is that is does not support nested structs. If ExampleToo has an UnmarshalJSON method, the ExampleToo.UnmarshalJSON() method is never called if unmarshalling to an Example object. It would be possible to write a method Example.UnmarshalJSON() that recursively handled validation, but that seems extremely complex, especially if ExampleToo is reused in many places.
So there appears to be some packages like the go-playground/validator where validation can be specified both as functions and tags. However, this works on the struct created, and not the JSON itself. So if a field is tagged as validation:"required" on an integer, and the integer value is 0, this will return an error because 0 is both a valid value and the "zero value" for integers.
An example of the latter here: https://go.dev/play/p/zqSUksPzUiq
I could also use pointers for everything, checking for nil as missing values. The main problem with that is that it requires dereferencing on each use and is a pretty uncommon practice for things like integers and strings.
One thing that I have also considered is a "sister struct" that uses pointers to do validation for required fields. The process would basically be to write a validation method for each struct, then validate that sister struct. If it works, then deserialize the main struct (without pointers). I haven't started on this, just a concept I've thought about, but I'm hoping there are better validation options.
So... is there a better way to do JSON/YAML input validation on nested structs? I'm happy to mix methods where say UnmarshalJSON is used for doing some work like verifying fields exist, but I'd like to pass that back to the library to let it continue to call UnmarshalJSON on subsequent nested structs. I'd also rather defer to the JSON library for casting values into the struct, etc.
I'm trying to devise a service to convert a generic JSON data representation into XML data representation.
The first idea that came to my mind (and that I found on the internet) takes advantage of the Go templating utilities.
If I have a JSON data representation like the following:
{
"user": {
"name": "Luca",
"surname": "Rossi"
}
}
I could devise a template like the following:
<xml>
<user name="{{.user.name}}" surname="{{.user.surname}}" />
</xml>
to generate:
<xml>
<user name="Luca" surname="Rossi" />
</xml>
The problem is: Go requires the definition of a struct which declares how to marshal and unmarshal a JSON data representation; at the same time, however, I'd like to provide the template to generate the XML as a service configuration available at runtime.
The question is: "is it possible"?
I know (thanks tothis question) that I can take do something like this:
var anyJson map[string]interface{}
json.Unmarshal(bytes, &anyJson)
The problem comes when I have to access values: I'd be asked to do a type assertion, like
anyJson["id"].(string)
Now, I might be able to know the type of anyJson["id"] by means of JSON schema, for example, but I don't know if I can do a parametric type assertion, something like
anyJson["id"].(typeForIDFromJSONSchema)
When you unmarshal into map[string]interface{}, every nested JSON object will also be map[string]interface{}. So type assertions of the contained elements to string may typically work, but not to any struct type - the unmarshaller would always be unaware of your structs.
So the two options I suggest are
to do it 'the hard way' using type switches and type assertions - this is workable and fast, but not all that nice; or
to use a different tool such as jsoniter or gjson - these might be a little slower but they do allow you to traverse arbitrary JSON graphs
I have used both GJson and Jsoniter. GJson works by reading byte by byte through the input, using buffering to keep its speed up, providing an API that allows inspection of the current element and assertions to convert the values.
Jsoniter looks to me like a cleaner implementation along the lines of successful parsers in Java, but I haven't used it for parsing in this manner yet. (It can also be used simply as a fast drop-in replacement for the standard Go encoding/json.) I suggest you focus on using its Iterator and its WhatIsNext method.
Problem
Found many similar questions (title) but none solved my problem, so here is it.
I have a JSON string that contains some known fields (should always be present) plus any number of unknown/arbitrary fields.
Example
{"known1": "foo", "known2": "bar", "unknown1": "car", "unknown2": 1}
In this example known1 and known2 are known fields. unknown1 and unknown2 are arbitrary/unknown fields.
The unknown fields can have any name (key) and any value. The value type can be either a string, bool, float64 or int.
What I want is to find the simplest and most elegant (idiomatic) way to parse a message like this.
My solution
I've used the following struct:
type Message struct {
Known1 string `json:"known1"`
Known2 string `json:"known2"`
Unknowns []map[string]interface{}
}
Expected result
With this struct and the above sample JSON message I want to achieve a Message like the following (output from fmt.Printf("%+v", msg)):
{Known1:foo Known2:bar Unknowns:[map[unknown1:car] map[unknown2:1]]}
Attempts
1. Simple unmarshal
https://play.golang.org/p/WO6XlpK_vJg
This doesn't work, Unknowns is not filled with the remaining unknown key/value pairs as expected.
2. Double unmarshal
https://play.golang.org/p/Mw6fOYr-Km8
This works but I needed two unmarshals, one to fill the known fields (using an alias type to avoid an infinite loop) and a second one to get all fields as a map[string]interface{} and process the unknowns.
3. Unmarshal and type conversion
https://play.golang.org/p/a7itXObavrX
This works and seems the best option among my solutions.
Any other option?
Option 2 and 3 work but I'm curious if anyone has a simpler/more elegant solution.
TMTOWTDI, and I think you found a reasonable solution. Another option you could consider -- which I guess is similar to your option 2 -- is to unmarshal it onto a map[string]interface{}.
Then range over the keys and values and do whatever you need to with the data, running type assertions on the values as necessary. Depending on what you need to do, you may skip the struct entirely, or still populate it.
package main
import (
"encoding/json"
"fmt"
)
func main() {
jsonMsg := `{"known1": "foo", "known2": "bar", "unknown1": "car", "unknown2": 1}`
var msg map[string]interface{}
fmt.Println(json.Unmarshal([]byte(jsonMsg), &msg))
fmt.Printf("%+v", msg)
}
prints
<nil>
map[known1:foo known2:bar unknown1:car unknown2:1]
https://play.golang.org/p/Bl99Cq5tFWW
Probably there is no existing solution just for your situation.
As an addition to your solutions you may try to use raw parsing libraries like:
https://github.com/buger/jsonparser
https://github.com/json-iterator/go
https://github.com/nikandfor/json
Last one is mine and it has some unfinished features, but raw parsing is done.
You may also use reflection with libraries above to set known fields in the struct.
In decode.go, it mentions:
// To unmarshal JSON into a value implementing the Unmarshaler interface,
// Unmarshal calls that value's UnmarshalJSON method, including
// when the input is a JSON null.
// Otherwise, if the value implements encoding.TextUnmarshaler
// and the input is a JSON quoted string, Unmarshal calls that value's
// UnmarshalText method with the unquoted form of the string.
What are the differences between UnmarshalText and UnmarshalJSON? Which one is preferred?
Simply:
UnmarshalText unmarshals a text-encoded value.
UnmarshalJSON unmarshals a JSON-encoded value.
Which is preferred depends on what you're doing.
JSON encoding is defined by RFC 7159. If you're consuming or producing JSON documents, you should use JSON encoding.
Text encoding has no standard, and is entirely implementation-dependent. Go implements Text-(un)marshalers for a few types, but there's no guarantee that any other application will understand these formats.
Text-encoding is most commonly used for things like URL query parameters, HTML forms, or other loosely-defined formats.
If you have a choice in the matter, using JSON is probably a better way to go. But again, it depends on what you're doing what makes the most sense.
As it relates to Go's JSON unmarshaler, the JSON unmarshaler will call a type's UnmarshalJSON method, if it's defined, and fall back to UnmarshalText if that is defined.
If you know you'll be using JSON, you should absolutely define an UnmarshalJSON function.
You would generally create an UnmarshalText only if you expected it to be used in non-JSON contexts, with the added benefit that the JSON unmarshaler would also use it, without having to duplicate it (if indeed the same implementation would work for JSON).
Per the documentation:
To unmarshal JSON into a value implementing the Unmarshaler interface,
Unmarshal calls that value's UnmarshalJSON method, including when the
input is a JSON null. Otherwise, if the value implements
encoding.TextUnmarshaler and the input is a JSON quoted string,
Unmarshal calls that value's UnmarshalText method with the unquoted
form of the string.
Meaning: if you want to take some JSON and unmarshal it with some custom logic, you would use UnmarshalJSON. If you want to take the text in a string field of a JSON document and decode that in some special way (i.e. parse it rather than just write it into a string-typed field), you would use UnmarshalText. For example, net.IP implements UnmarshalText so that you can provide a string value like "ipAddress": "1.2.3.4" and unmarshal it into a net.IP field. If net.IP did not implement UnmarshalText, you would only be able to unmarshal the JSON representation of the underlying type ([]byte).
When you unmarshal JSON to []interface{} is there any way to automatically detect the type besides some standard types like bool, int and string?
What I noticed is the following, Let's say I marshal [uuid.UUID, bool] then the JSON I get looks like:
[[234,50,7,116,194,41,64,225,177,151,60,195,60,45,123,106],true]
When I unmarshal it again, I get the types as shown through reflect:
[]interface{}, bool
I don't understand why it picked []interface{}. If it cannot detect it, shouldn't it be at least interface{}?
In any case, my question is, is it possible to unmarshal any type when the target is of type []interface{}? It seems to work for standard types like string, bool, int but for custom types I don't think that's possible, is it? You can define custom JSON marshal/unmarshal methods but that only works if you decode it into a target type so that it can look up which custom marshal/unmarshal methods to use.
You can unmarshal any type into a value of type interface{}. If you use a value of type []interface{}, you can only unmarshal JSON arrays into it, but yes, the elements of the array may be of any type.
Since you're using interface{} or []interface{}, yes, type information is not available, and it's up to the encoding/json package to choose the best it sees fit. For example, for JSON objects it will choose map[string]interface{}. The full list of default types is documented in json.Unmarshal():
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Obviously if your JSON marshaling/unmarshaling logic needs some pre- / postprocessing, the json package will not miraculously find it out. It can know about those only if you unmarshal into values of specific types (which implement json.Unmarshaler). The json package will still be able to unmarshal them to the default types, but custom logic will obviously not run on them.