I am trying to simultaneously unmarshal and strip fields from a number of different JSON responses into appropriate Go structs. To do this, I created a Wrappable interface that defines the Unwrap method (which strips the appropriate fields) and pass that interface to the code that unmarshals and unwraps. It looks like the following example (also at http://play.golang.org/p/fUGveHwiz9):
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
A string `json:"a"`
B string `json:"b"`
}
type DataWrapper struct {
Elements []Data `json:"elems"`
}
type Wrapper interface {
Unwrap() []interface{}
}
func (dw DataWrapper) Unwrap() []interface{} {
result := make([]interface{}, len(dw.Elements))
for i := range dw.Elements {
result[i] = dw.Elements[i]
}
return result
}
func unmarshalAndUnwrap(data []byte, wrapper Wrapper) []interface{} {
err := json.Unmarshal(data, &wrapper)
if err != nil {
panic(err)
}
return wrapper.Unwrap()
}
func main() {
data := `{"elems": [{"a": "data", "b": "data"}, {"a": "data", "b": "data"}]}`
res := unmarshalAndUnwrap([]byte(data), DataWrapper{})
fmt.Println(res)
}
However, when I run the code, Go panics with the following error:
panic: json: cannot unmarshal object into Go value of type main.Wrapper
It seems the unmarshaller doesn't want to be passed a pointer to an interface. I am somewhat surprised by this given that I can get at the underlying type and fields using the reflect package within the unmarshalAndUnwrap method. Can anyone provide insight into this problem and how I might work around it?
As you stated, passing a non-pointer fails. Why are you trying to do this anyway?
Replace
res := unmarshalAndUnwrap([]byte(data), DataWrapper{})
by
res := unmarshalAndUnwrap([]byte(data), &DataWrapper{})
It should do the trick and it avoid unnecessary copy.
This error should help you understand: http://play.golang.org/p/jXxCxPQDOw
Related
I'm trying to create a JSON representation within Go using a map[string]interface{} type. I'm dealing with JSON strings and I'm having a hard time figuring out how to avoid the JSON unmarshaler to automatically deal with numbers as float64s. As a result the following error occurs.
Ex.
"{ 'a' : 9223372036854775807}" should be map[string]interface{} = [a 9223372036854775807 but in reality it is map[string]interface{} = [a 9.2233720368547758088E18]
I searched how structs can be used to avoid this by using json.Number but I'd really prefer using the map type designated above.
The go json.Unmarshal(...) function automatically uses float64 for JSON numbers. If you want to unmarshal numbers into a different type then you'll have to use a custom type with a custom unmarshaler. There is no way to force the unmarshaler to deserialize custom values into a generic map.
For example, here's how you could parse values of the "a" property as a big.Int.
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type MyDoc struct {
A BigA `json:"a"`
}
type BigA struct{ *big.Int }
func (a BigA) UnmarshalJSON(bs []byte) error {
_, ok := a.SetString(string(bs), 10)
if !ok {
return fmt.Errorf("invalid integer %s", bs)
}
return nil
}
func main() {
jsonstr := `{"a":9223372036854775807}`
mydoc := MyDoc{A: BigA{new(big.Int)}}
err := json.Unmarshal([]byte(jsonstr), &mydoc)
if err != nil {
panic(err)
}
fmt.Printf("OK: mydoc=%#v\n", mydoc)
// OK: mydoc=main.MyDoc{A:9223372036854775807}
}
func jsonToMap(jsonStr string) map[string]interface{} {
result := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &result)
return result
}
Example - https://goplay.space/#ra7Gv8A5Heh
Related questions - create a JSON data as map[string]interface with the given data
In this stackoverflow post it's explained how to add arbitrary fields to a golang struct by using it as an anonymous. This works fine if you are working with known struct types, but I'm wondering how to do the same thing when dealing with an unknown struct or interface.
I wrote the following example to demonstrate:
package main
import (
"os"
"encoding/json"
"fmt"
)
type example interface{}
type Data struct {
Name string
}
func printInterface(val interface{}) {
example1 := struct {
example
Extra string
}{
example: val,
Extra: "text",
}
json.NewEncoder(os.Stdout).Encode(example1)
}
func printStructPointer(val *Data) {
example2 := struct {
*Data
Extra string
}{
Data: val,
Extra: "text",
}
json.NewEncoder(os.Stdout).Encode(example2)
}
func main() {
d := Data{Name:"name"}
fmt.Println("Example 1:")
printInterface(&d)
fmt.Println("Example 2:")
printStructPointer(&d)
}
This prints the following:
Example 1:
{"example":{"Name":"name"},"Extra":"text"}
Example 2:
{"Name":"name","Extra":"text"}
I'm so assuming that I was working within printInterface how do get the JSON output to look like the JSON output of printStructPointer?
There's an important difference between printInterface() and printStructPointer(). The first one embeds an interface type, while the second embeds a struct type (more specifically a pointer to a struct type).
When you embed a struct (or pointer to struct) type, the fields of the embedded type get promoted, so in the 2nd example it will be valid to write example2.Name. When you embed an interface type, an interface does not have fields, so no fields will be promoted. So it doesn't matter if the interface value wraps a struct (or pointer to struct), fields of that struct won't get promoted (they can't be).
Thus, in the printInterface() the interface wrapping a struct won't get "flattened" in the JSON result.
Solving it with generating a dynamic type using reflection
One way to solve this is to generate a dynamic type at runtime, using reflection (reflect package). This new type will be a struct, and it will contain an anonymous struct field being of the type that is wrapped in the passed interface, and will also contain our extra field (of type string).
This is how it could look like:
func printInterface(val interface{}) {
t2 := reflect.StructOf([]reflect.StructField{
reflect.StructField{
Name: "X",
Anonymous: true,
Type: reflect.TypeOf(val),
},
reflect.StructField{
Name: "Extra",
Type: reflect.TypeOf(""),
},
})
v2 := reflect.New(t2).Elem()
v2.Field(0).Set(reflect.ValueOf(val))
v2.FieldByName("Extra").SetString("text")
json.NewEncoder(os.Stdout).Encode(v2.Interface())
}
Output is as expected (try it on the Go Playground):
Example 1:
{"Name":"name","Extra":"text"}
Example 2:
{"Name":"name","Extra":"text"}
Solving it with marshaling twice
Another way would be to marshal the value, unmarshal it into a map, add the extra field and marshal it again:
func printInterface(val interface{}) error {
data, err := json.Marshal(val)
if err != nil {
return err
}
v2 := map[string]interface{}{}
if err := json.Unmarshal(data, &v2); err != nil {
return err
}
v2["Extra"] = "text"
return json.NewEncoder(os.Stdout).Encode(v2)
}
Output is the same. Try it on the Go Playground.
This solution is simpler, easier to follow, but it's slower as it marshals twice. Also note that in this example the fields in the result might be in different order, as iteration order on a map is not specified in Go (for details see Why can't Go iterate maps in insertion order?).
Here's one way:
package main
import (
"encoding/json"
"fmt"
"os"
)
type example interface{}
type Data struct {
Name string
}
func printInterface(val interface{}) {
if d, ok := val.(*Data); ok {
fmt.Println("Asserting type *Data for val is OK")
printStructPointer(d)
} else {
fmt.Println("Asserting type *Data for val is NOT OK")
}
}
func printStructPointer(val *Data) {
example2 := struct {
*Data
Extra string
}{
Data: val,
Extra: "text",
}
json.NewEncoder(os.Stdout).Encode(example2)
}
func main() {
d := Data{Name: "Testing"}
fmt.Println("Example 1:")
printInterface(&d)
fmt.Println("Example 2:")
printStructPointer(&d)
}
Playground: https://play.golang.org/p/OPotpTyUEz
You can also possibly use a type switch for an assertion, esp if you have many types. Hope this helps!
All of the ways I'm seeing involve building structs and unmarshalling the data into the struct. But what if I'm getting JSON responses with hundreds of fields? I don't want to have to create 100 field structs just to be able to get to the data I want. Coming from a Java background there are easy ways to simply get the http response as a string and then pass the JSON string into a JSON object that allows for easy traversal. It's very painless. Is there anything like this in Go?
Java example in pseudo code:
String json = httpResponse.getBody();
JsonObject object = new JsonObject(json);
object.get("desiredKey");
Golang: fetch JSON from an HTTP response without using structs as helpers
This is a typical scenario we come across. This is achieved by json.Unmarshal.
Here is a simple json
{"textfield":"I'm a text.","num":1234,"list":[1,2,3]}
which is serialized to send across the network and unmarshaled at Golang end.
package main
import (
"fmt"
"encoding/json"
)
func main() {
// replace this by fetching actual response body
responseBody := `{"textfield":"I'm a text.","num":1234,"list":[1,2,3]}`
var data map[string]interface{}
err := json.Unmarshal([]byte(responseBody), &data)
if err != nil {
panic(err)
}
fmt.Println(data["list"])
fmt.Println(data["textfield"])
}
Hope this was helpful.
The json.Unmarshal method will unmarshal to a struct that does not contain all the fields in the original JSON object. In other words, you can cherry-pick your fields. Here is an example where FirstName and LastName are cherry-picked and MiddleName is ignored from the json string:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
}
func main() {
jsonString := []byte("{\"first_name\": \"John\", \"last_name\": \"Doe\", \"middle_name\": \"Anderson\"}")
var person Person
if err := json.Unmarshal(jsonString, &person); err != nil {
panic(err)
}
fmt.Println(person)
}
The other answers here are misleading, as they don't show you what happens if you try to go deeper in the Map. This example works fine enough:
package main
import (
"encoding/json"
"fmt"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
body := map[string]interface{}{}
json.NewDecoder(r.Body).Decode(&body)
/*
[map[
id:com.github.android
platform:play
url:https://play.google.com/store/apps/details?id=com.github.android
]]
*/
fmt.Println(body["related_applications"])
}
but if you try to go one level deeper, it fails:
/*
invalid operation: body["related_applications"][0] (type interface {} does not
support indexing)
*/
fmt.Println(body["related_applications"][0])
Instead, you would need to assert type at each level of depth:
/*
map[
id:com.github.android
platform:play
url:https://play.google.com/store/apps/details?id=com.github.android
]
*/
fmt.Println(body["related_applications"].([]interface{})[0])
You can as well unmarshal it into a map[string]interface{}
body, err := ioutil.ReadAll(resp.Body)
map := &map[string]interface{}{}
json.Unmarshal(body, map)
desiredValue := map["desiredKey"]
The received json must have an object as the most outer element. The map can also contain lists or nested maps, depending on the json.
I'm making a Go service that gathers JSON objects from different sources and aggregates them in a single JSON object.
I was wondering if there was any way to aggregate the child objects without having to unmarshal and re-marshal them again or having to manually build a JSON string.
I was thinking of using a struct containing the already marshalled parts, such as this:
type Event struct {
Place string `json:"place"`
Attendees string `json:"attendees"`
}
Where Place and Attendees are JSON strings themselves. I'd like to somehow mark them as "already marshalled" so they don't end up as escaped JSON strings but get used as is instead.
Is there any way to achieve this?
You can use json.RawMessage
RawMessage is a raw encoded JSON object. It implements Marshaler and Unmarshaler and can be used to delay JSON decoding or precompute a JSON encoding.
Also, json.RawMessage is an alias to []byte so you can values to it this way:
v := json.RawMessage(`{"foo":"bar"}`)
Example:
package main
import (
"encoding/json"
"fmt"
)
type Event struct {
Place json.RawMessage `json:"place"`
Attendees json.RawMessage `json:"attendees"`
}
func main() {
e := Event{
Place: json.RawMessage(`{"address":"somewhere"}`),
Attendees: json.RawMessage(`{"key":"value"}`),
}
c, err := json.Marshal(&e)
if err != nil {
panic(err)
}
fmt.Println(string(c))
// {"place":{"address":"somewhere"},"attendees":{"key":"value"}}
}
Yes, you can use a custom type that implements Marshaler interface.
https://play.golang.org/p/YB_eKlfOND
package main
import (
"fmt"
"encoding/json"
)
type Event struct {
Place RawString `json:"place"`
Attendees RawString `json:"attendees,omitempty"`
}
type RawString string
func (s RawString) MarshalJSON() ([]byte, error) {
return []byte(s), nil
}
func main() {
event := Event{
Place: RawString(`{"name":"Paris"}`),
Attendees: RawString(`[{"name":"John"}, {"name":"Juli"}]`),
}
s, err := json.Marshal(event)
fmt.Println(fmt.Sprintf("event: %v; err: %v", string(s), err))
}
I have this struct
type SyncInfo struct {
Target string
}
Now I query some json data from ElasticSearch. Source is of type json.RawMessage.
All I want is to map source to my SyncInfo which I created the variable mySyncInfo for.
I even figured out how to do that...but it seems weird. I first call MarshalJSON() to get a []byte and then feed that to json.Unmarshal() which takes an []byte and a pointer to my struct.
This works fine but it feels as if I'm doing an extra hop. Am I missing something or is that the intended way to get from a json.RawMessage to a struct?
var mySyncInfo SyncInfo
jsonStr, _ := out.Hits.Hits[0].Source.MarshalJSON()
json.Unmarshal(jsonStr, &mySyncInfo)
fmt.Print(mySyncInfo.Target)
As said, the underlying type of json.RawMessage is []byte, so you can use a json.RawMessage as the data parameter to json.Unmarshal.
However, your problem is that you have a pointer (*json.RawMessage) and not a value. All you have to do is to dereference it:
err := json.Unmarshal(*out.Hits.Hits[0].Source, &mySyncInfo)
Working example:
package main
import (
"encoding/json"
"fmt"
)
type SyncInfo struct {
Target string
}
func main() {
data := []byte(`{"target": "localhost"}`)
Source := (*json.RawMessage)(&data)
var mySyncInfo SyncInfo
// Notice the dereferencing asterisk *
err := json.Unmarshal(*Source, &mySyncInfo)
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", mySyncInfo)
}
Output:
{Target:localhost}
Playground: http://play.golang.org/p/J8R3Qrjrzx
json.RawMessage is really just a slice of bytes. You should be able to feed it directly into json.Unmarshal directly, like so:
json.Unmarshal(out.Hits.Hits[0].Source, &mySyncInfo)
Also, somewhat unrelated, but json.Unmarshal can return an error and you want to handle that.
err := json.Unmarshal(*out.Hits.Hits[0].Source, &mySyncInfo)
if err != nil {
// Handle
}