Is there a simple way in Go to check whether given JSON is either an Object {} or array []?
The first thing that comes to mind is to json.Unmarshal() into an interface, and then see if it becomes a map, or a slice of maps. But that seems quite inefficient.
Could I just check if the first byte is a { or a [? Or is there a better way of doing this that already exists.
Use the following to detect if JSON text in the []byte value data is an array or object:
// Get slice of data with optional leading whitespace removed.
// See RFC 7159, Section 2 for the definition of JSON whitespace.
x := bytes.TrimLeft(data, " \t\r\n")
isArray := len(x) > 0 && x[0] == '['
isObject := len(x) > 0 && x[0] == '{'
This snippet of code handles optional leading whitespace and is more efficient than unmarshalling the entire value.
Because the top-level value in JSON can also be a number, string, boolean or nil, it's possible that isArray and isObject both evaluate to false. The values isArray and isObject can also evaluate to false when the JSON is invalid.
Use a type switch to determine the type. This is similar to Xay's answer, but simpler:
var v interface{}
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
switch v := v.(type) {
case []interface{}:
// it's an array
case map[string]interface{}:
// it's an object
default:
// it's something else
}
Do step-by-step parsing of your JSON, using json.Decoder. This has the advantage over the other answers of:
Being more efficient than decoding the entire value
Using the official JSON parsing rules, and generating standard errors if you get invalid input.
Note, this code isn't tested, but should be enough to give you the idea. It can also be easily expanded to check for numbers, booleans, or strings, if desired.
func jsonType(in io.Reader) (string, error) {
dec := json.NewDecoder(in)
// Get just the first valid JSON token from input
t, err := dec.Token()
if err != nil {
return "", err
}
if d, ok := t.(json.Delim); ok {
// The first token is a delimiter, so this is an array or an object
switch (d) {
case '[':
return "array", nil
case '{':
return "object", nil
default: // ] or }, shouldn't be possible
return "", errors.New("Unexpected delimiter")
}
}
return "", errors.New("Input does not represent a JSON object or array")
}
Note that this consumed the first few bytes of in. It is an exercise for the reader to make a copy, if necessary. If you're trying to read from a byte slice ([]byte), convert it to a reader first:
t, err := jsonType(bytes.NewReader(myValue))
Go playground
Related
I have been writing in Go for a long time and recently while rewriting the code I came across a strange thing. I did a couple of tests and if request == nil check never worked. Previously, I was always afraid of getting a nil pointer exception, and so I inserted checks everywhere. But in this case, the json decode error handler seems to cover all cases.
var (
request *models.Owner
err error
)
err = json.NewDecoder(r.Body).Decode(&request)
if err != nil {
render.Render(w, r, response.ErrInvalidRequest(err))
return
}
if request == nil {
render.Render(w, r, response.ErrInvalidRequest("request is nil"))
return
}
if request == nil is it possible to catch this? Perhaps this check is unnecessary, and if I remove this check in my project, the code will become cleaner.
It is possible that nil error will be returned and request will still be nil, but only if the input JSON is the JSON null value.
For example:
type Owners struct {
Name string
}
var request *Owners
if err := json.Unmarshal([]byte("null"), &request); err != nil {
panic(err)
}
fmt.Println(request == nil)
fmt.Println(request)
This will output (try it on the Go Playground):
true
<nil>
This is documented at json.Unmarshal():
To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer. If the pointer is nil, Unmarshal allocates a new value for it to point to.
I'm trying to consume a rest endpoint in my golang project. The JSON structure is pretty large and is semi-structured, so I'm using reflection to iterate over it and grab the information that I am interested in.
Here is how I'm unmarshal-ing the response:
var m map[string]interface{}
json.Unmarshal(bytes, &m)
But the part I'm stuck at is - I'm iterating through a slice of maps (i think) but I'm unable to get the keys or values of the map. Here is the code in question.
if val, ok := m["messages"]; ok {
s := reflect.ValueOf(val)
if s.Kind() == reflect.Slice {
for i := 0; i < s.Len(); i++ {
item := s.Index(i)
fmt.Println("VALUE = ", item.Elem())
fmt.Println("KIND = ", item.Kind())
}
}
return
}
When I run the code the value that is displayed looks like a map:
map[decoration_stats:<nil> highlight_ranges:map[] index:graylog_7 message:map[_id:49272694-1834-11ea-8928-0242ac120004 docker:{"container_id":"0f9d97722c25240c6f99487b247b2416177a749de47d661cd661334514e0e74f"} facility:fluentd gl2_message_id:01DVDSM9VSDQ5PF81T4C31NSH6....
And the kind is:
KIND = interface
I tried various things like:
for _, e := range val.MapKeys() {
v := val.MapIndex(e)
fmt.Println(v)
}
But the code panics with:
panic: reflect: call of reflect.Value.MapKeys on interface Value
Sorry, I'm somewhat new to golang but have used other static typed language, mainly Java, when it comes to any reflection type programming.
My question is how to can I convert this interface to a map or some concrete type so that I can use it. Any help would be greatly appreciated.
Using reflection is an inefficient way to do this. JSON unmarshal, when used with an interface (and map[string]interface{}) produces a limited set of types, and you can use type assertions or a type-switch to deal with it:
if val, ok := m["messages"]; ok {
switch v:=val.(type) {
case map[string]interface{}: // JSON object
for key, value:=range v {
}
case []interface{}: // JSON array
for i,node:=range v {
}
case string: // string value
case float64: // numeric value
case bool: // boolean value
case json.Number: // If you use json.Decoder with UseNumber()
}
}
I am using Google's go-jsonnet library to evaluate some jsonnet files.
I have a function, like so, which renders a Jsonnet document:
// Takes a list of jsonnet files and imports each one and mixes them with "+"
func renderJsonnet(files []string, param string, prune bool) string {
// empty slice
jsonnetPaths := files[:0]
// range through the files
for _, s := range files {
jsonnetPaths = append(jsonnetPaths, fmt.Sprintf("(import '%s')", s))
}
// Create a JSonnet VM
vm := jsonnet.MakeVM()
// Join the slices into a jsonnet compat string
jsonnetImport := strings.Join(jsonnetPaths, "+")
if param != "" {
jsonnetImport = "(" + jsonnetImport + ")" + param
}
if prune {
// wrap in std.prune, to remove nulls, empty arrays and hashes
jsonnetImport = "std.prune(" + jsonnetImport + ")"
}
// render the jsonnet
out, err := vm.EvaluateSnippet("file", jsonnetImport)
if err != nil {
log.Panic("Error evaluating jsonnet snippet: ", err)
}
return out
}
This function currently returns a string, because the jsonnet EvaluateSnippet function returns a string.
What I now want to do is render that result JSON using the go-prettyjson library. However, because the JSON i'm piping in is a string, it's not rendering correctly.
So, some questions:
Can I convert the returned JSON string to a JSON type, without knowing beforehand what struct to marshal it into
if not, can I render the json in a pretty manner some other way?
Is there an option, function or method I'm missing here to make this easier?
Can I convert the returned JSON string to a JSON type, without knowing beforehand what struct to marshal it into
Yes. It's very easy:
var jsonOut interface{}
err := json.Unmarshal([]byte(out), &jsonOut)
if err != nil {
log.Panic("Invalid json returned by jsonnet: ", err)
}
formatted, err := prettyjson.Marshal([]byte(jsonOut))
if err != nil {
log.Panic("Failed to format jsonnet output: ", err)
}
More info here: https://blog.golang.org/json-and-go#TOC_5.
Is there an option, function or method I'm missing here to make this easier?
Yes. The go-prettyjson library has a Format function which does the unmarshalling for you:
formatted, err := prettyjson.Format([]byte(out))
if err != nil {
log.Panic("Failed to format jsonnet output: ", err)
}
can I render the json in a pretty manner some other way?
Depends on your definition of pretty. Jsonnet normally outputs every field of an object and every array element on a separate line. This is usually considered pretty printing (as opposed to putting everything on the same line with minimal whitespace to save a few bytes). I suppose this is not good enough for you. You can write your own manifester in jsonnet which formats it to your liking (see std.manifestJson as an example).
I have an application that consumes data from a third-party api. I need to decode the json into a struct, which requires the struct to have json tags of the "incoming" json fields. The outgoing json fields have a different naming convention, so I need different json tags for the encoding.
I will have to do this with many different structs, and each struct might have many fields.
What is the best way to accomplish this without repeating a lot of code?
Example Structs:
// incoming "schema" field names
type AccountIn struct {
OpenDate string `json:"accountStartDate"`
CloseDate string `json:"cancelDate"`
}
// outgoing "schema" field names
type AccountOut struct {
OpenDate string `json:"openDate"`
CloseDate string `json:"closeDate"`
}
Maybe the coming change on Go 1.8 would help you, it will allow to 'cast' types even if its JSON tags definition is different: This https://play.golang.org/p/Xbsoa8SsEk works as expected on 1.8beta, I guess this would simplify your current solution
A bit an uncommon but probably quite well working method would be to use a intermediate format so u can use different readers and writers and therefore different tags. For example https://github.com/mitchellh/mapstructure which allows to convert a nested map structure into struct
types. Pretty similar like json unmarshal, just from a map.
// incoming "schema" field names
type AccountIn struct {
OpenDate string `mapstructure:"accountStartDate" json:"openDate"`
CloseDate string `mapstructure:"cancelDate" json:"closeDate"`
}
// from json to map with no name changes
temporaryMap := map[string]interface{}{}
err := json.Unmarshal(jsonBlob, &temporaryMap)
// from map to structs using mapstructure tags
accountIn := &AccountIn{}
mapstructure.Decode(temporaryMap, accountIn)
Later when writing (or reading) u will use directly the json functions which will then use the json tags.
If it's acceptable to take another round trip through json.Unmarshal and json.Marshal, and you don't have any ambiguous field names within your various types, you could translate all the json keys in one pass by unmarshaling into the generic structures used by the json package:
// map incoming to outgoing json identifiers
var translation = map[string]string{
"accountStartDate": "openDate",
"cancelDate": "closeDate",
}
func translateJS(js []byte) ([]byte, error) {
var m map[string]interface{}
if err := json.Unmarshal(js, &m); err != nil {
return nil, err
}
translateKeys(m)
return json.MarshalIndent(m, "", " ")
}
func translateKeys(m map[string]interface{}) {
for _, v := range m {
if v, ok := v.(map[string]interface{}); ok {
translateKeys(v)
}
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
if newKey, ok := translation[k]; ok {
m[newKey] = m[k]
delete(m, k)
}
}
}
https://play.golang.org/p/nXmWlj7qH9
This might be a Naive Approach but is fairly easy to implement:-
func ConvertAccountInToAccountOut(AccountIn incoming) (AccountOut outcoming){
var outcoming AccountOut
outcoming.OpenDate = incoming.OpenDate
outcoming.CloseDate = incoming.CloseDate
return outcoming
}
var IncomingJSONData AccountIn
resp := getJSONDataFromSource() // Some method that gives you the Input JSON
err1 := json.UnMarshall(resp,&IncomingJSONData)
OutGoingJSONData := ConvertAccountInToAccountOut(IncomingJSONData)
if err1 != nil {
fmt.Println("Error in UnMarshalling JSON ",err1)
}
fmt.Println("Outgoing JSON Data: ",OutGoingJSONData)
I have a Json string that I want to unmarshal.
This is working:
jsonString := []byte(`{"my_int": 3, "my_string": null}`)
var data map[string]interface{}
err := json.Unmarshal(jsonString, &data)
if err != nil {
fmt.Println(err)
}
//avroJson := make(map[string]interface{})
for k, v := range data {
fmt.Printf("%v, %T\n", k, v)
}
My issue is: the value of my_int which is 3 is returned as float64.
My question is: how to parse a json string with the "minimum type" so that 3 will return int32 and not the maximum type 3 => float64?
Assumption: my Json is huge and only have primitive types and I want a minimum value that is really float64 to continue to show float64.
Clarification:
A "minimum type" means that if 3 can be considered both int32 and float64 the "minimum type" will be int32, which is the exact type you'll get when running this:
reflect.TypeOf(3).string()
Since you are unmarshaling into a map of interface{}, the following section of the golang json.Unmarshal documentation pertains:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
...
float64, for JSON numbers
string, for JSON strings
...
As such, to unmarshal your sample data into your desired types you should define a struct type which contains the desired field/type mappings, for example:
type MyType struct {
MyInt int `json:"my_int"`
MyString *string `json:"my_string"`
}
foo := MyType{}
jsonstr := `{"my_int": 3, "my_string": null}`
err := json.Unmarshal([]byte(jsonstr), &foo)
if err != nil {
panic(err)
}
// foo => main.MyType{MyInt:3, MyString:(*string)(nil)}
Since you cannot describe your data in a struct then your options are to:
Use a json.Decoder to convert the values to your desired types as they are parsed.
Parse the document into a generic interface and post-process the value types.
Option #1 is the most flexible and can likely be implemented to be more performant than the other option since parsing and transformation could be performed in a single pass of the data.
Option #2 might be simpler but will require two passes over the data. Here is an example of what the post-processing step might look like:
func TransformValueTypes(o map[string]interface{}) {
for k, v := range o {
// Convert nil values to *string type.
if v == interface{}(nil) {
o[k] = (*string)(nil)
}
// Convert numbers to int32 if possible
if x, isnumber := v.(float64); isnumber {
if math.Floor(x) == x {
if x >= math.MinInt32 && x <= math.MaxInt32 {
o[k] = int32(x)
}
// Possibly check for other integer sizes here?
}
// Possibly check if float32 is possible here?
}
// Check for maps and slices here...
}
}
So if you call TransformValueTypes(data) then your types will look like:
// my_int -> 3 (int32)
// my_string -> <nil> (*string)
// my_string2 -> "foo" (string)
// my_float -> 1.23 (float64)
Of course, your transform function could also apply type transformation logic based on the key name.
Importantly, note that if your document might have additional structure not mentioned in your question (such as nested objects or arrays) then your transform function will need to account for them by more value type checking, recursive calls, and iteration.