When marshaling from a map[string]interface{} and an equivalent struct, the JSON returned are similar but different in the internal order of the keys likewise:
var arg1 = map[string]interface{}{
"foo": "bar",
"baz": map[string]interface{}{
"bee": "boo",
},
}
type Arg struct {
Foo string `json:"foo"`
Baz map[string]interface{} `json:"baz"`
}
var arg2 = &Arg{
Foo: "bar",
Baz: map[string]interface{}{
"bee": "boo",
},
}
func main() {
result1, _ := json.Marshal(arg1)
result2, _ := json.Marshal(arg2)
fmt.Println(reflect.DeepEqual(result1, result2)
fmt.Println(string(result1))
fmt.Println(string(result2))
// false
// {"baz":{"bee":"boo"},"foo":"bar"}
// {"foo":"bar","baz":{"bee":"boo"}}
}
Check the playground here.
Although in JSON the order doesn't matter, I'd like to know why are the results ordered differently?
The encoding/json package marshals maps in sorted key order and structs in the order that the fields are declared.
Although the order used by the encoding/json package is not documented, it's safe to assume that maps are marshaled in sorted key order and structs are marshaled in field declaration order. There are many tests in the standard library and elsewhere that depend on these orders.
The order of elements in maps is not defined as Go uses a hash table to implement them. You should expect the elements to be marshalled in any order.
When marshalling a structure, the order of JSON fields should match the order of fields in structure. This is a side effect of accessing fields through reflection. Still you should not rely on this side effect.
The order of elements is not significant in JSON. The problem of comparing JSON structures is the same as the problem of comparing trees with unordered child nodes.
Related
I am connecting to an API which gives a rather large json payload. I need to add a key and value to the root object.Once I do ioutil.Readall from the package "net/http" the JSON is a byte array.
My goal is to just simply add to the structure and move on. As an example, the following the pretty similar to what I am doing: https://tutorialedge.net/golang/consuming-restful-api-with-go/
So how can I simply add to a JSON structure another element (key: value)?
If all you want to do is add a key and value to the root object and produce new JSON, and you don't care about having the data in a structure, you can unmarshal into map[string]interface{}, add your value, and then marshal again:
var m map[string]interface{}
err := json.Unmarshal(data, &m)
m["new_key"] = newValue
newData, err := json.Marshal(m)
(I'm not checking for errors, but you should do that of course.) Take a look at https://golang.org/pkg/encoding/json/ for more information about how to deal with JSON in Go.
Here's how you can do it in an efficient way while preserving the order of keys in the original JSON. The idea is to use json.RawMessage
// original JSON
bytes := []byte(`{"name": "Adele", "age": 24}`)
// let's try to add the kv pair "country": "USA" to it
type Person struct {
Bio json.RawMessage `json:"bio"`
Country string `json:"country"`
}
p := Person{
Bio: json.RawMessage(bytes),
Country: "USA",
}
// ignoring error for brevity
modifiedBytes, _ := json.Marshal(p)
fmt.Println(string(modifiedBytes))
Output:
{"bio":{"name":"Adele","age":24},"country":"USA"}
You can see that the ordering of original JSON is preserved, which wouldn't have been the case if you marshalled the JSON to a map[string]interface{}. This is also more efficient when you're dealing with huge JSONs since there's no reflection involved.
Complete code - https://play.golang.org/p/3hAPVbrAo_w
Since you have a byte data, you need to parse it and store the result in a variable that has your json structure using json.Marshal.
Then after, to add a new key value pair, you can define a new struct with the key and its data type
var variable type1
// Unmarshal the byte data in a variable
json.Unmarshall(data, &variable)
// to add a new key value you can define a new type
type type2 struct {
type1
key type
}
// and you can add
variable2 := type2{variable, newValueToAdd}
While deserializing & reserializing is the more "correct" approach, it may be overkill for just adding a value at the root, which can be done with simple string manipulation (really byte slice manipulation, but the semantics are similar and it's arguably easier):
data := []byte(`{"foo": 1, "bar": true}`)
ins := []byte(`, "baz": "hello"`) // Note the leading comma.
closingBraceIdx := bytes.LastIndexByte(data, '}')
data = append(data[:closingBraceIdx], ins...)
data = append(data, '}')
This is more error-prone, because it is unaware of JSON syntax entirely, but it's safe enough for most cases and for large JSON documents it is more efficient than a parse, insert, and reserialize.
Playground example: https://play.golang.org/p/h8kL4Zzp7rq
if you want to add key-value of json bytes to new json object, you can use json.RawMessage.
type Res struct {
Data interface{}
Message string
}
var row json.RawMessage
row = []byte(`{"Name":"xxx","Sex":1}`)
res := Res{Data:row,Message:"user"}
resBytes ,err := json.Marshal(res)
println(string(resBytes))
//print result："Data":{"Name":"xxx","Sex":1},"Message":"user"}
SJSON package is another way to modify JSON values.
In this example :
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Diamond","Gold","Silver"}
}`
To replace a "Diamond" with "Ultimate"
value, _ := sjson.Set(json, "license.0", "Ultimate")
fmt.Println(value)
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Ultimate","Gold","Silver"}
}`
Or to add a value "Ultimate" in the end:
value, _ := sjson.Set(json, "license.-1", "Ultimate")
fmt.Println(value)
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Diamond","Gold","Silver","Ultimate"}
}`
So I have a project with lots of incoming data about 15 sources in total, of course there are inconsistencies in how each label there data available in their rest api's. I need to Change some of their field names to be consistent with the others, but I am at a loss on how to do this when the data sources are json object arrays. A working example of what I am trying to do is found here playground and below
however I seem to lack the knowledge as to how to make this work when the data is not a single json object , but instead and array of objects that I am unmarshaling.
Another approach is using Maps like in this example but the result is the same, works great as is for single objects, but I can not seem to get it to work with json object arrays. Iteration through arrays is not a possibility as I am collecting about 8,000 records every few minutes.
package main
import (
"encoding/json"
"os"
)
type omit bool
type Value interface{}
type CacheItem struct {
Key string `json:"key"`
MaxAge int `json:"cacheAge"`
Value Value `json:"cacheValue"`
}
func NewCacheItem() (*CacheItem, error) {
i := &CacheItem{}
return i, json.Unmarshal([]byte(`{
"key": "foo",
"cacheAge": 1234,
"cacheValue": {
"nested": true
}
}`), i)
}
func main() {
item, _ := NewCacheItem()
json.NewEncoder(os.Stdout).Encode(struct {
*CacheItem
// Omit bad keys
OmitMaxAge omit `json:"cacheAge,omitempty"`
OmitValue omit `json:"cacheValue,omitempty"`
// Add nice keys
MaxAge int `json:"max_age"`
Value *Value `json:"value"`
}{
CacheItem: item,
// Set the int by value:
MaxAge: item.MaxAge,
// Set the nested struct by reference, avoid making a copy:
Value: &item.Value,
})
}
It appears your desired output is JSON. You can accomplish the conversion by unmarshaling into a slice of structs, and then iterating through each of those to convert them to the second struct type (your anonymous struct above), append them into a slice and then marshal the slice back to JSON:
package main
import (
"fmt"
"encoding/json"
)
type omit bool
type Value interface{}
type CacheItem struct {
Key string `json:"key"`
MaxAge int `json:"cacheAge"`
Value Value `json:"cacheValue"`
}
type OutGoing struct {
// Omit bad keys
OmitMaxAge omit `json:"cacheAge,omitempty"`
OmitValue omit `json:"cacheValue,omitempty"`
// Add nice keys
Key string `json:"key"`
MaxAge int `json:"max_age"`
Value *Value `json:"value"`
}
func main() {
objects := make([]CacheItem, 0)
sample := []byte(`[
{
"key": "foo",
"cacheAge": 1234,
"cacheValue": {
"nested": true
}},
{
"key": "baz",
"cacheAge": 123,
"cacheValue": {
"nested": true
}}]`)
json.Unmarshal(sample, &objects)
out := make([]OutGoing, 0, len(objects))
for _, o := range objects {
out = append(out, OutGoing{Key:o.Key, MaxAge:o.MaxAge, Value:&o.Value})
}
s, _ := json.Marshal(out)
fmt.Println(string(s))
}
This outputs
[{"key":"foo","max_age":1234,"value":{"nested":true}},{"key":"baz","max_age":123,"value":{"nested":true}}]
You could probably skip this iteration and conversion code if you wrote custom MarshalJSON and UnmarshalJSON methods for your CacheItem type, instead of relying on struct field tags. Then you could pass the same slice to both Unmarshal and Marshal.
To me there's no obvious performance mistake with these approaches -- contrast with building a string in a loop using the + operator -- and when that's the case it's often best to just get the software to work and then test for performance rather than ruling out a solution based on fears of performance issues without actually testing.
If there's a performance problem with the above approaches, and you really want to avoid marshal and unmarshal completely, you could look into byte replacement in the JSON data (e.g. regexp). I'm not recommending this approach, but if your changes are very simple and the inputs are very consistent it could work, and it would give another approach you could performance test, and then you could compare performance test results.
I have JSON (that I cannot control) like this:
{
"foo1":{
"a":{
"up":10,
"down":5
}
},
"foo2":{
"a":{
"up":1,
"down":1
}
},
"bar":{
"up":11,
"down":6
}
}
"foo1" and "foo2" are dynamic.
How can I properly unmarshal this structure in go?
It would be okay if I could just tell go to not try to deserialize "bar" (the inconsistent property).
Go will by default ignore fields unspecified in the struct you unmarshal into.
In this case, your structure would be set up like this:
type NestedProp2 struct {
Up int
Down int
}
type NestedProp struct {
A NestedProp2
}
type Prop struct {
Foo1 NestedProp
Foo2 NestedProp
}
When you call the the json.Unmarshal function, the extra property will not be deserialized:
var prop Prop
err := json.Unmarshal(jsonBlob, &prop)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", prop)
So you get the following output:
{Foo1:{A:{Up:10 Down:5}} Foo2:{A:{Up:1 Down:1}}}
You can see it in action here.
You said:
I have JSON (that I cannot control)
So to what extent you could control? Here I could provide you with some scenario, and hope some of them match your purpose :)
Remember the general rule first:
In Golang, if a JSON key failed to find a matched field in struct, it will not be unmarshalled.
This means, for a key name in a JSON struct, when unmarshalling, it will look for a field in a golang struct at the same level with the same name case-insensitively. If this search failed, this key won't be unmarshalled.
For example, a key named foo1 will look for a field name foo1 in a golang struct at the same indent level. However it also matches with Foo1 or FoO1, since this matching is case-insensitive.
Remember, you could use field tag to specify the field name as well. Please take a look at the official page.
The value of some of the JSON fields are not consistent, and they could be ignored.
This is the case #gnalck solved in his answer. According to the general rule, if those inconsistent field failed to find a match, they will not be unmarshalled. Therefore, just don't put those inconsistent fields in the struct and you will be fine.
The value of some of the JSON fields are not consistent, but they could not be ignored.
In this case, #gnalck failed since those fields could not be ignored. Now a better way is to unmarshal bar into a json.RawMessage, so that you could unmarshal later.
The keys of the JSON object is undetermined, and their value is undetermined as well.
In this case, we could unmarshal the whole JSON object into a map[string]json.RawMessage, and unmarshal each fields later. When unmarshalling to a map, you could iterate through the map to get all the fields, and unmarshal them into a proper struct later.
Lets say I have this type:
type Foo struct{
Bar string `json:"bar"`
}
and I want to unmarshal this json into it:
in := []byte(`{"bar":"aaa", "baz":123}`)
foo := &Foo{}
json.Unmarshal(in,foo)
will succeed just fine. I would like to at least know that there were fields that were skipped in the processing. Is there any good way to access that information?
playground snippet
As you're probably aware you can unmarshal any valid json into a map[string]interface{}. Having unmarsahled into an instance of Foo already there is no meta data available where you could check fields that were excluded or anything like that. You could however, unmarshal into both types and then check the map for keys that don't correspond to fields on Foo.
in := []byte(`{"bar":"aaa", "baz":123}`)
foo := &Foo{}
json.Unmarshal(in,foo)
allFields := &map[string]interface{}
json.Unmarshal(in, allFields)
for k, _ := range allFields {
fmt.Println(k)
// could also use reflect to get field names as string from Foo
// the get the symmetric difference by nesting another loop here
// and appending any key that is in allFields but not on Foo to a slice
}
In go you unmarshal json into a struct. The problem is that I have an api that might change the type of the value of a key from request to request.
For example objects that might be inlined objects like this:
{
"mykey": [
{obj1},
{obj2}
]
}
but also point to objects by keys, like this:
{
"mykey": [
"/obj1/is/at/this/path",
"/obj2/is/at/this/other/path"
]
}
Some objects can be inlined, but others are referenced from multiple locations.
In javascript or python this wouldn't be a problem. Just check the type.
What's the go-idiomatic way to unmarshal and parse these two objects? Is Reflect the only way?
You could unmarshal this JSON to a structure like the following:
type Data struct {
MyKey []interface{} `json:"mykey"`
}
If the JSON includes strings, they will be decoded as strings in the array. If the JSON includes objects, they will be decoded as map[string]interface{} values. You can distinguish between the two using a type switch. Something like this:
for i, v := range data.MyKey {
switch x := v.(type) {
case string:
fmt.Println("Got a string: ", x)
case map[string]interface{}:
fmt.Printf("Got an object: %#v\n", x)
}
}
You can play around with this example here: http://play.golang.org/p/PzwFI2FSav