Add a new key-value pair to a JSON object - json

I am connecting to an API which gives a rather large json payload. I need to add a key and value to the root object.Once I do ioutil.Readall from the package "net/http" the JSON is a byte array.
My goal is to just simply add to the structure and move on. As an example, the following the pretty similar to what I am doing: https://tutorialedge.net/golang/consuming-restful-api-with-go/
So how can I simply add to a JSON structure another element (key: value)?

If all you want to do is add a key and value to the root object and produce new JSON, and you don't care about having the data in a structure, you can unmarshal into map[string]interface{}, add your value, and then marshal again:
var m map[string]interface{}
err := json.Unmarshal(data, &m)
m["new_key"] = newValue
newData, err := json.Marshal(m)
(I'm not checking for errors, but you should do that of course.) Take a look at https://golang.org/pkg/encoding/json/ for more information about how to deal with JSON in Go.

Here's how you can do it in an efficient way while preserving the order of keys in the original JSON. The idea is to use json.RawMessage
// original JSON
bytes := []byte(`{"name": "Adele", "age": 24}`)
// let's try to add the kv pair "country": "USA" to it
type Person struct {
Bio json.RawMessage `json:"bio"`
Country string `json:"country"`
}
p := Person{
Bio: json.RawMessage(bytes),
Country: "USA",
}
// ignoring error for brevity
modifiedBytes, _ := json.Marshal(p)
fmt.Println(string(modifiedBytes))
Output:
{"bio":{"name":"Adele","age":24},"country":"USA"}
You can see that the ordering of original JSON is preserved, which wouldn't have been the case if you marshalled the JSON to a map[string]interface{}. This is also more efficient when you're dealing with huge JSONs since there's no reflection involved.
Complete code - https://play.golang.org/p/3hAPVbrAo_w

Since you have a byte data, you need to parse it and store the result in a variable that has your json structure using json.Marshal.
Then after, to add a new key value pair, you can define a new struct with the key and its data type
var variable type1
// Unmarshal the byte data in a variable
json.Unmarshall(data, &variable)
// to add a new key value you can define a new type
type type2 struct {
type1
key type
}
// and you can add
variable2 := type2{variable, newValueToAdd}

While deserializing & reserializing is the more "correct" approach, it may be overkill for just adding a value at the root, which can be done with simple string manipulation (really byte slice manipulation, but the semantics are similar and it's arguably easier):
data := []byte(`{"foo": 1, "bar": true}`)
ins := []byte(`, "baz": "hello"`) // Note the leading comma.
closingBraceIdx := bytes.LastIndexByte(data, '}')
data = append(data[:closingBraceIdx], ins...)
data = append(data, '}')
This is more error-prone, because it is unaware of JSON syntax entirely, but it's safe enough for most cases and for large JSON documents it is more efficient than a parse, insert, and reserialize.
Playground example: https://play.golang.org/p/h8kL4Zzp7rq

if you want to add key-value of json bytes to new json object, you can use json.RawMessage.
type Res struct {
Data interface{}
Message string
}
var row json.RawMessage
row = []byte(`{"Name":"xxx","Sex":1}`)
res := Res{Data:row,Message:"user"}
resBytes ,err := json.Marshal(res)
println(string(resBytes))
//print result:"Data":{"Name":"xxx","Sex":1},"Message":"user"}

SJSON package is another way to modify JSON values.
In this example :
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Diamond","Gold","Silver"}
}`
To replace a "Diamond" with "Ultimate"
value, _ := sjson.Set(json, "license.0", "Ultimate")
fmt.Println(value)
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Ultimate","Gold","Silver"}
}`
Or to add a value "Ultimate" in the end:
value, _ := sjson.Set(json, "license.-1", "Ultimate")
fmt.Println(value)
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Diamond","Gold","Silver","Ultimate"}
}`

Related

How to Change Field names of json object arrays

So I have a project with lots of incoming data about 15 sources in total, of course there are inconsistencies in how each label there data available in their rest api's. I need to Change some of their field names to be consistent with the others, but I am at a loss on how to do this when the data sources are json object arrays. A working example of what I am trying to do is found here playground and below
however I seem to lack the knowledge as to how to make this work when the data is not a single json object , but instead and array of objects that I am unmarshaling.
Another approach is using Maps like in this example but the result is the same, works great as is for single objects, but I can not seem to get it to work with json object arrays. Iteration through arrays is not a possibility as I am collecting about 8,000 records every few minutes.
package main
import (
"encoding/json"
"os"
)
type omit bool
type Value interface{}
type CacheItem struct {
Key string `json:"key"`
MaxAge int `json:"cacheAge"`
Value Value `json:"cacheValue"`
}
func NewCacheItem() (*CacheItem, error) {
i := &CacheItem{}
return i, json.Unmarshal([]byte(`{
"key": "foo",
"cacheAge": 1234,
"cacheValue": {
"nested": true
}
}`), i)
}
func main() {
item, _ := NewCacheItem()
json.NewEncoder(os.Stdout).Encode(struct {
*CacheItem
// Omit bad keys
OmitMaxAge omit `json:"cacheAge,omitempty"`
OmitValue omit `json:"cacheValue,omitempty"`
// Add nice keys
MaxAge int `json:"max_age"`
Value *Value `json:"value"`
}{
CacheItem: item,
// Set the int by value:
MaxAge: item.MaxAge,
// Set the nested struct by reference, avoid making a copy:
Value: &item.Value,
})
}
It appears your desired output is JSON. You can accomplish the conversion by unmarshaling into a slice of structs, and then iterating through each of those to convert them to the second struct type (your anonymous struct above), append them into a slice and then marshal the slice back to JSON:
package main
import (
"fmt"
"encoding/json"
)
type omit bool
type Value interface{}
type CacheItem struct {
Key string `json:"key"`
MaxAge int `json:"cacheAge"`
Value Value `json:"cacheValue"`
}
type OutGoing struct {
// Omit bad keys
OmitMaxAge omit `json:"cacheAge,omitempty"`
OmitValue omit `json:"cacheValue,omitempty"`
// Add nice keys
Key string `json:"key"`
MaxAge int `json:"max_age"`
Value *Value `json:"value"`
}
func main() {
objects := make([]CacheItem, 0)
sample := []byte(`[
{
"key": "foo",
"cacheAge": 1234,
"cacheValue": {
"nested": true
}},
{
"key": "baz",
"cacheAge": 123,
"cacheValue": {
"nested": true
}}]`)
json.Unmarshal(sample, &objects)
out := make([]OutGoing, 0, len(objects))
for _, o := range objects {
out = append(out, OutGoing{Key:o.Key, MaxAge:o.MaxAge, Value:&o.Value})
}
s, _ := json.Marshal(out)
fmt.Println(string(s))
}
This outputs
[{"key":"foo","max_age":1234,"value":{"nested":true}},{"key":"baz","max_age":123,"value":{"nested":true}}]
You could probably skip this iteration and conversion code if you wrote custom MarshalJSON and UnmarshalJSON methods for your CacheItem type, instead of relying on struct field tags. Then you could pass the same slice to both Unmarshal and Marshal.
To me there's no obvious performance mistake with these approaches -- contrast with building a string in a loop using the + operator -- and when that's the case it's often best to just get the software to work and then test for performance rather than ruling out a solution based on fears of performance issues without actually testing.
If there's a performance problem with the above approaches, and you really want to avoid marshal and unmarshal completely, you could look into byte replacement in the JSON data (e.g. regexp). I'm not recommending this approach, but if your changes are very simple and the inputs are very consistent it could work, and it would give another approach you could performance test, and then you could compare performance test results.

How to deal with Struct having different json key than json response

I have a struct VideoInfo that has a key in it called embedCode. The API I am querying returns the embed code as embed_code. During unmarshalling the response how do I ensure embed_code goes into embedCode?
Also is there an easy way to take a large json string and automatically turn it into a struct, or can one only use a map?
With respect to remapping the field names use the corresponding annotation in the structure declaration:
type VideoInfo struct {
EmbedCode string `json:"embed_code"`
}
The marshaller/un-marshaller will only process public field, so you need to capitalise the field name.
With respect to converting the whole structure, yes it is easy. Declare an instance to un-marshal into and pass a reference to the json.Unmarshal method (from a test):
data, _ := json.Marshal(request)
var resp response.VideoInfo
if err := json.Unmarshal(data, &resp); err != nil {
t.Errorf("unexpected error, %v", err)
}
At first, struct's field must be start from capital letter to be public. So you need something like that:
type VideoInfo struct {
EmbedCode string `json:"embed_code"`
}
And look at documentation for more info.

How to access fields of a JSON in GO

Hi everyone I'm trying to see what the proper way of accessing fields of a json object from a http.get request in go.
I first do an http.get call get the JSON and then print it (which works) but is there a way to access just a field?
for example:
response, err:= http.Get("URL")
//Error checking is done between
contents, err:=ioutil.Readall(response.Body)
//Now at this point I have a json that looks like
{"id": "someID",
"name": "someName",
"test": [{"Name":"Test1",
"Result": "Success"},
{"Name":"Test2",
"Result": "Success"},
{...},
]}
Is there a way to only print the "test" of the Json? What is the proper way of accessing that field?
Use encoding/json package to Unmarshal data into struct, like following.
type Result struct {
ID string `json:"id"`
Name string `json:"name"`
Test []interface{} `json:"test"`
}
var result Result
json.Unmarshal(contents, &result)
fmt.Println(result.Test)
You can also parse Test to specific struct.
Same as the previous answer, use encoding/json package to Unmarshal data. But if you don't want to specify the structure, you could use map[string]interface/bson.M{} to receive the data, and get the field, then cast into types your want.
m := make(map[string]interface{})
err := json.Unmarshal(data, &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m["id"])
You may want to try gabs container, if you are not sure how depth JSON hierarchy can be. Have a look at below resources
https://github.com/Jeffail/gabs
https://godoc.org/github.com/Jeffail/gabs
If you just want to access one field then you can use the jsonq module https://godoc.org/github.com/jmoiron/jsonq
For your example you could get the test object with code similar to
jq.Object("test")
Where jq is a jsonq query object constructed from your JSON above (see the godoc page for instructions on how to create a query object from a JSON stream or string).
You can also use this library for retrieving specific String, Integer, Float and Bool values at an arbitrary depth inside a JSON object.
Since you are starting with a URL, Decode is a better option than Unmarshal:
package main
import (
"encoding/json"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
defer r.Body.Close()
var s struct { Name string }
json.NewDecoder(r.Body).Decode(&s)
println(s.Name == "GitHub")
}
https://golang.org/pkg/encoding/json#Decoder.Decode
You may check this https://github.com/valyala/fastjson
s := []byte(`{"foo": [123, "bar"]}`)
fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))
// Output:
// foo.0=123

Can I access "extra" fields when unmarshalling json in Go?

Lets say I have this type:
type Foo struct{
Bar string `json:"bar"`
}
and I want to unmarshal this json into it:
in := []byte(`{"bar":"aaa", "baz":123}`)
foo := &Foo{}
json.Unmarshal(in,foo)
will succeed just fine. I would like to at least know that there were fields that were skipped in the processing. Is there any good way to access that information?
playground snippet
As you're probably aware you can unmarshal any valid json into a map[string]interface{}. Having unmarsahled into an instance of Foo already there is no meta data available where you could check fields that were excluded or anything like that. You could however, unmarshal into both types and then check the map for keys that don't correspond to fields on Foo.
in := []byte(`{"bar":"aaa", "baz":123}`)
foo := &Foo{}
json.Unmarshal(in,foo)
allFields := &map[string]interface{}
json.Unmarshal(in, allFields)
for k, _ := range allFields {
fmt.Println(k)
// could also use reflect to get field names as string from Foo
// the get the symmetric difference by nesting another loop here
// and appending any key that is in allFields but not on Foo to a slice
}

Golang Null Types and json.Decode()

I have not been able to find a way around this issue currently. If I have a structure i would like to populate with json from a http.Request I have no way to tell for instance what value was actually passed in for some values. For instance if I pass in an empty json object and run json.Decode on a structure that looks like this...
var Test struct {
Number int `json:"number"`
}
I now have a json object that supposedly was passed with a key of number and a value of zero when in fact I would rather have this return nothing at all. Does go provide another method that would actually allow me to see what JSON has been passed in or not.
Sorry for the rambling I have been trying to figure out how to to this for a few days now and it's driving me nuts.
Thanks for any help.
Edit:
I made this to depict exactly what I am talking about http://play.golang.org/p/aPFKSvuxC9
You could use pointers, for example:
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus"},
{"Name": "Quoll", "Order": 100}
]`)
type Animal struct {
Name string
Order *int
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
for _, a := range animals {
if a.Order != nil {
fmt.Printf("got order, %s : %d\n", a.Name, *a.Order)
}
}
}
I don't see how you could do this by giving a struct to the Unmarshal function. With the following structure for instance:
type A struct {
Hello string
Foo int
Baz string
}
var a A
json.Unmarshal(data, &a)
Even by doing another implementation of Unmarshal, there would be only two (simple) possibilities:
If baz is not in the json data, set a.Baz to a default value, compatible with its type: the empty string (or 0 if it's an integer). This is the current implementation.
If baz is not in the json data, return an error. That would be very inconvenient if the absence of baz is a normal behaviour.
Another possibility would be to use pointers, and use the default value nil in the same spirit than the default value I talked about, but there would still be issue if your json file could be filled with null values: you would not be able to distinguish values that were in the json file, but set as null, and values that were not in the json, and unmarshalled with nil as their default value.
However, this solution might suit you: instead of using a struct, why not using a map[string]interface{} ? The Unmarshall function would not have to add a default value to non-present fields, and it would be able to retrieve any type of data from the json file.
var b = []byte(`[{"Name": "Platypus"}, {"Name": "Quoll", "Order": 100}]`)
var m []map[string]interface{}
err := json.Unmarshal(b, &m)
fmt.Println(m)
// [map[Name:Platypus] map[Name:Quoll Order:100]]