Aggregating JSON objects in Go - json

I'm making a Go service that gathers JSON objects from different sources and aggregates them in a single JSON object.
I was wondering if there was any way to aggregate the child objects without having to unmarshal and re-marshal them again or having to manually build a JSON string.
I was thinking of using a struct containing the already marshalled parts, such as this:
type Event struct {
Place string `json:"place"`
Attendees string `json:"attendees"`
}
Where Place and Attendees are JSON strings themselves. I'd like to somehow mark them as "already marshalled" so they don't end up as escaped JSON strings but get used as is instead.
Is there any way to achieve this?

You can use json.RawMessage
RawMessage is a raw encoded JSON object. It implements Marshaler and Unmarshaler and can be used to delay JSON decoding or precompute a JSON encoding.
Also, json.RawMessage is an alias to []byte so you can values to it this way:
v := json.RawMessage(`{"foo":"bar"}`)
Example:
package main
import (
"encoding/json"
"fmt"
)
type Event struct {
Place json.RawMessage `json:"place"`
Attendees json.RawMessage `json:"attendees"`
}
func main() {
e := Event{
Place: json.RawMessage(`{"address":"somewhere"}`),
Attendees: json.RawMessage(`{"key":"value"}`),
}
c, err := json.Marshal(&e)
if err != nil {
panic(err)
}
fmt.Println(string(c))
// {"place":{"address":"somewhere"},"attendees":{"key":"value"}}
}

Yes, you can use a custom type that implements Marshaler interface.
https://play.golang.org/p/YB_eKlfOND
package main
import (
"fmt"
"encoding/json"
)
type Event struct {
Place RawString `json:"place"`
Attendees RawString `json:"attendees,omitempty"`
}
type RawString string
func (s RawString) MarshalJSON() ([]byte, error) {
return []byte(s), nil
}
func main() {
event := Event{
Place: RawString(`{"name":"Paris"}`),
Attendees: RawString(`[{"name":"John"}, {"name":"Juli"}]`),
}
s, err := json.Marshal(event)
fmt.Println(fmt.Sprintf("event: %v; err: %v", string(s), err))
}

Related

Unmarshall json value into []byte where the string could sometimes be escaped json

I have a json response that looks like this
{
"eventId":"fbf4a1a1-b4a3-4dfe-a01f-ec52c34e16e4",
"eventType":"event-type",
"eventNumber":0,
"data":"{\n \"a\": \"1\"\n}",
"metaData":"{\n \"yes\": \"no\"\n}",
"streamId":"test",
"isJson":true,
"isMetaData":true,
"isLinkMetaData":false,
"positionEventNumber":0,
"positionStreamId":"test",
"title":"0#test",
"id":"http://localhost:2113/streams/test/0",
"updated":"2017-12-14T05:09:58.816079Z"
}
the key value pairs of data, and metaData might sometimes be encoded json or it might not.
I want to decode those values into a byte array like this.
// Event represent an event to be stored.
type Event struct {
Data []byte `json:"data"`
Metadata []byte `json:"metaData"`
}
but when I try to unmarshal the json object I get the following error:
illegal base64 data at input byte 0
What could I be doing wrong here?
It works fine if I decode the data and metaData into a string, but I don't want to use a string.
You are looking for the json.RawMessage type
It is just a specialized []byte that you can then use as need be.
type Event struct {
Data json.RawMessage `json:"data"`
Metadata json.RawMessage `json:"metaData"`
}
Then you could treat it as a literal []byte via []byte(e.Data)
Here's an example of use, on play:
package main
import (
"encoding/json"
"fmt"
)
var RAW = []byte(`
{
"eventId":"fbf4a1a1-b4a3-4dfe-a01f-ec52c34e16e4",
"eventType":"event-type",
"eventNumber":0,
"data":"{\n \"a\": \"1\"\n}",
"metaData":"{\n \"yes\": \"no\"\n}",
"streamId":"test",
"isJson":true,
"isMetaData":true,
"isLinkMetaData":false,
"positionEventNumber":0,
"positionStreamId":"test",
"title":"0#test",
"id":"http://localhost:2113/streams/test/0",
"updated":"2017-12-14T05:09:58.816079Z"
}
`)
type Event struct {
Data json.RawMessage `json:"data"`
Metadata json.RawMessage `json:"metaData"`
}
func main() {
var e Event
err := json.Unmarshal(RAW, &e)
fmt.Printf("%v -- %+v\n", err, e)
b, err := json.Marshal(e)
fmt.Printf("%v -- %s\n", err, b)
}
I created a type that implements the TextUnmarshaler and TextMarshaler interfaces. The json decoder looks for this if the type doesn't implement MarshalJSON and UnmarshalJSON methods.
type RawData []byte
func (r RawData) MarshalText() (text []byte, err error) {
return r[:], nil
}
func (r *RawData) UnmarshalText(text []byte) error {
*r = text[:]
return nil
}
// Event represent an event to be stored.
type Event struct {
Data RawData `json:"data,omitempty"`
Metadata RawData `json:"metaData,omitempty"`
}
I needed this because sometimes the Data or Metadata would not be json encoded in a string, but could also be other formats like protocol buffers.

Go: Converting JSON string to map[string]interface{}

I'm trying to create a JSON representation within Go using a map[string]interface{} type. I'm dealing with JSON strings and I'm having a hard time figuring out how to avoid the JSON unmarshaler to automatically deal with numbers as float64s. As a result the following error occurs.
Ex.
"{ 'a' : 9223372036854775807}" should be map[string]interface{} = [a 9223372036854775807 but in reality it is map[string]interface{} = [a 9.2233720368547758088E18]
I searched how structs can be used to avoid this by using json.Number but I'd really prefer using the map type designated above.
The go json.Unmarshal(...) function automatically uses float64 for JSON numbers. If you want to unmarshal numbers into a different type then you'll have to use a custom type with a custom unmarshaler. There is no way to force the unmarshaler to deserialize custom values into a generic map.
For example, here's how you could parse values of the "a" property as a big.Int.
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type MyDoc struct {
A BigA `json:"a"`
}
type BigA struct{ *big.Int }
func (a BigA) UnmarshalJSON(bs []byte) error {
_, ok := a.SetString(string(bs), 10)
if !ok {
return fmt.Errorf("invalid integer %s", bs)
}
return nil
}
func main() {
jsonstr := `{"a":9223372036854775807}`
mydoc := MyDoc{A: BigA{new(big.Int)}}
err := json.Unmarshal([]byte(jsonstr), &mydoc)
if err != nil {
panic(err)
}
fmt.Printf("OK: mydoc=%#v\n", mydoc)
// OK: mydoc=main.MyDoc{A:9223372036854775807}
}
func jsonToMap(jsonStr string) map[string]interface{} {
result := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &result)
return result
}
Example - https://goplay.space/#ra7Gv8A5Heh
Related questions - create a JSON data as map[string]interface with the given data

Is there a way to extract JSON from an http response without having to build structs?

All of the ways I'm seeing involve building structs and unmarshalling the data into the struct. But what if I'm getting JSON responses with hundreds of fields? I don't want to have to create 100 field structs just to be able to get to the data I want. Coming from a Java background there are easy ways to simply get the http response as a string and then pass the JSON string into a JSON object that allows for easy traversal. It's very painless. Is there anything like this in Go?
Java example in pseudo code:
String json = httpResponse.getBody();
JsonObject object = new JsonObject(json);
object.get("desiredKey");
Golang: fetch JSON from an HTTP response without using structs as helpers
This is a typical scenario we come across. This is achieved by json.Unmarshal.
Here is a simple json
{"textfield":"I'm a text.","num":1234,"list":[1,2,3]}
which is serialized to send across the network and unmarshaled at Golang end.
package main
import (
"fmt"
"encoding/json"
)
func main() {
// replace this by fetching actual response body
responseBody := `{"textfield":"I'm a text.","num":1234,"list":[1,2,3]}`
var data map[string]interface{}
err := json.Unmarshal([]byte(responseBody), &data)
if err != nil {
panic(err)
}
fmt.Println(data["list"])
fmt.Println(data["textfield"])
}
Hope this was helpful.
The json.Unmarshal method will unmarshal to a struct that does not contain all the fields in the original JSON object. In other words, you can cherry-pick your fields. Here is an example where FirstName and LastName are cherry-picked and MiddleName is ignored from the json string:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
}
func main() {
jsonString := []byte("{\"first_name\": \"John\", \"last_name\": \"Doe\", \"middle_name\": \"Anderson\"}")
var person Person
if err := json.Unmarshal(jsonString, &person); err != nil {
panic(err)
}
fmt.Println(person)
}
The other answers here are misleading, as they don't show you what happens if you try to go deeper in the Map. This example works fine enough:
package main
import (
"encoding/json"
"fmt"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
body := map[string]interface{}{}
json.NewDecoder(r.Body).Decode(&body)
/*
[map[
id:com.github.android
platform:play
url:https://play.google.com/store/apps/details?id=com.github.android
]]
*/
fmt.Println(body["related_applications"])
}
but if you try to go one level deeper, it fails:
/*
invalid operation: body["related_applications"][0] (type interface {} does not
support indexing)
*/
fmt.Println(body["related_applications"][0])
Instead, you would need to assert type at each level of depth:
/*
map[
id:com.github.android
platform:play
url:https://play.google.com/store/apps/details?id=com.github.android
]
*/
fmt.Println(body["related_applications"].([]interface{})[0])
You can as well unmarshal it into a map[string]interface{}
body, err := ioutil.ReadAll(resp.Body)
map := &map[string]interface{}{}
json.Unmarshal(body, map)
desiredValue := map["desiredKey"]
The received json must have an object as the most outer element. The map can also contain lists or nested maps, depending on the json.

How do I Unmarshal JSON?

I am trying to unmarshal JSON into a struct. However, the struct has a field with a tag. Using reflection, and I try to see if the tag has the string "json" in it. If it does, then the json to unmarshal should simply be unmarshaled into the field as a string.
Example:
const data = `{"I":3, "S":{"phone": {"sales": "2223334444"}}}`
type A struct {
I int64
S string `sql:"type:json"`
}
Problem is simple - unmarshal "S" in the json as a string into the struct A.
This is how far I have come. But I am stuck here.
http://play.golang.org/p/YzrhjuXxGN
This is the go way of doing it - no reflection requred. Create a new type RawString and create MarshalJSON and UnmarshalJSON methods for it. (playground)
// RawString is a raw encoded JSON object.
// It implements Marshaler and Unmarshaler and can
// be used to delay JSON decoding or precompute a JSON encoding.
type RawString string
// MarshalJSON returns *m as the JSON encoding of m.
func (m *RawString) MarshalJSON() ([]byte, error) {
return []byte(*m), nil
}
// UnmarshalJSON sets *m to a copy of data.
func (m *RawString) UnmarshalJSON(data []byte) error {
if m == nil {
return errors.New("RawString: UnmarshalJSON on nil pointer")
}
*m += RawString(data)
return nil
}
const data = `{"i":3, "S":{"phone": {"sales": "2223334444"}}}`
type A struct {
I int64
S RawString `sql:"type:json"`
}
func main() {
a := A{}
err := json.Unmarshal([]byte(data), &a)
if err != nil {
log.Fatal("Unmarshal failed", err)
}
fmt.Println("Done", a)
}
I modified the implementation of RawMessage to create the above.
The problem here is that you are using one encoding format for two protocols and probably there is something wrong in your model.
That is a valid Json payload and should be handled as such. One trick that you can use is to create
another field to handle the "string" json and one to handle the "json struct".
See this modified example: I first unmarshal json and then marshal back to json to create the final string to upload to the database. One field is used for unmarshaling and the other to communicate with the DB.
package main
import(
"fmt"
"encoding/json"
)
const data = `{"i":3, "S":{"phone": {"sales": "2223334444"}}}`
type A struct {
I int64
Sjson struct {
Phone struct {
Sales string `json:"sales"`
} `json:"phone"`
} `json:"S", sql:"-"`
S string `sql:"type:json",json:"-"`
}
func main() {
msg := A{}
_ = json.Unmarshal([]byte(data), &msg)
data, _ := json.Marshal(msg.Sjson)
msg.S = string(data)
fmt.Println("Done", msg)
}
Looks like the problem is that s interface{} in your code was not addressable. For Value.SetString the Value has to be addressable and with Kind String. you can check the documentation for it - http://golang.org/pkg/reflect/#Value.SetString
How i understand it SetString would not change the value in a, since you are only working with interface s. in Laws of Reflection you can find "reflect.ValueOf is a copy of x, not x itself"(3rd Law).
To make your code work I made some type assertions, and I used reflect.ValueOf on a pointer to asserted struct.
To check if Value is settable or addressable you can use Value.CanSet ad Value.CanAddr
working code: http://play.golang.org/p/DTriENkzA8
No idea whether its correct way to do this

How to Unmarshal JSON into an interface in Go

I am trying to simultaneously unmarshal and strip fields from a number of different JSON responses into appropriate Go structs. To do this, I created a Wrappable interface that defines the Unwrap method (which strips the appropriate fields) and pass that interface to the code that unmarshals and unwraps. It looks like the following example (also at http://play.golang.org/p/fUGveHwiz9):
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
A string `json:"a"`
B string `json:"b"`
}
type DataWrapper struct {
Elements []Data `json:"elems"`
}
type Wrapper interface {
Unwrap() []interface{}
}
func (dw DataWrapper) Unwrap() []interface{} {
result := make([]interface{}, len(dw.Elements))
for i := range dw.Elements {
result[i] = dw.Elements[i]
}
return result
}
func unmarshalAndUnwrap(data []byte, wrapper Wrapper) []interface{} {
err := json.Unmarshal(data, &wrapper)
if err != nil {
panic(err)
}
return wrapper.Unwrap()
}
func main() {
data := `{"elems": [{"a": "data", "b": "data"}, {"a": "data", "b": "data"}]}`
res := unmarshalAndUnwrap([]byte(data), DataWrapper{})
fmt.Println(res)
}
However, when I run the code, Go panics with the following error:
panic: json: cannot unmarshal object into Go value of type main.Wrapper
It seems the unmarshaller doesn't want to be passed a pointer to an interface. I am somewhat surprised by this given that I can get at the underlying type and fields using the reflect package within the unmarshalAndUnwrap method. Can anyone provide insight into this problem and how I might work around it?
As you stated, passing a non-pointer fails. Why are you trying to do this anyway?
Replace
res := unmarshalAndUnwrap([]byte(data), DataWrapper{})
by
res := unmarshalAndUnwrap([]byte(data), &DataWrapper{})
It should do the trick and it avoid unnecessary copy.
This error should help you understand: http://play.golang.org/p/jXxCxPQDOw