I am writing golang struct, which are compatible with some json structure. However, those most of the fields are know, there will be few fields following some specific patterns(like "x-{randomName}") in the json definition, which I also want to get deserialized to a certain field as map[string]interface{} as well.
Is there any descent way to achieve it?
It's less efficient, but you could unmarshal twice to avoid manually mapping the fields. Once to put all the properly tagged fields into the struct, and then again into a map[string]interface{} to get everything else. If you don't care about the duplicate fields, you don't even need to filter the second map.
You can even do this in an UnmarshalJSONmethod to automatically populate the struct
type S struct {
A string `json:"a"`
B string `json:"b"`
All map[string]interface{}
}
func (s *S) UnmarshalJSON(b []byte) error {
// create a new type to hide the UnmarshalJSON method
// otherwise we'll recurse indefinitely.
type ss S
err := json.Unmarshal(b, (*ss)(s))
if err != nil {
return err
}
// now unmarshal again into the All map
err = json.Unmarshal(b, &s.All)
if err != nil {
return err
}
return nil
}
http://play.golang.org/p/VBVlRjNlHy
Related
I want to unmarshal a JSON object where one field contains a JSON string into one coherent object. How do I do that in Go?
Example:
Input:
{
"foo":1,
"bar":"{\\"a\\":\\"Hello\\"}"
}
Go type:
type Child struct {
A string `json:"a"`
}
type Main struct {
Foo int `json:"foo"`
Bar Child `json:"bar"`
}
I guess I'd need to implement a custom UnmarshalJSON implementation on one of the types, but its twisting my head to figure out on which one and how.
I guess you want to treat this as if the JSON String were just part of the surrounding JSON object? If so, then yes, as you suggest, a custom UnmarshalJSON method on Child should accomplish this.
func (c *Child) UnmarshalJSON(p []byte) error {
var jsonString string
if err := json.Unmarshal(p, &jsonString); err != nil {
return err // Means the string was invalid
}
type C Child // A new type that doesn't have UnmarshalJSON method
return json.Unmarshal([]byte(jsonString), (*C)(c))
}
See it in the playground
if i were to create a custom UnmarshalJson for that data, I would create an auxiliary struct auxMain that has the same fields as the main struct but with Bar field as string. Then it unmarshals the JSON data into this auxiliary struct, extracting the Foo field and the Bar field as a string. After that, it unmarshals the Bar field as string into the Child struct, and assigns the extracted Foo field and the Child struct to the Main struct.
It's a round about way but seems to work in the playground.
func (m *Main) UnmarshalJSON(b []byte) error {
type auxMain struct {
Foo int `json:"foo"`
Bar string `json:"bar"`
}
var a auxMain
if err := json.Unmarshal(b, &a); err != nil {
return err
}
var child Child
if err := json.Unmarshal([]byte(a.Bar), &child); err != nil {
return err
}
m.Foo = a.Foo
m.Bar = child
return nil
}
try it out in the PlayGround and see: https://go.dev/play/p/wWIceUxu1tj
Don't know if this is what you are looking for.
I thought I understood unmarshalling by now, but I guess not. I'm having a little bit of trouble unmarshalling a map in go. Here is the code that I have so far
type OHLC_RESS struct {
Pair map[string][]Candles
Last int64 `json:"last"`
}
type Candles struct {
Time uint64
Open string
High string
Low string
Close string
VWAP string
Volume string
Count int
}
func (c *Candles) UnmarshalJSON(d []byte) error {
tmp := []interface{}{&c.Time, &c.Open, &c.High, &c.Low, &c.Close, &c.VWAP, &c.Volume, &c.Count}
length := len(tmp)
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
g := len(tmp)
if g != length {
return fmt.Errorf("Lengths don't match: %d != %d", g, length)
}
return nil
}
func main() {
response := []byte(`{"XXBTZUSD":[[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5],[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5]],"last":15}`)
var resp OHLC_RESS
err := json.Unmarshal(response, &resp)
fmt.Println("resp: ", resp)
}
after running the code, the last field will unmarshal fine, but for whatever reason, the map is left without any value. Any help?
The expedient solution, for the specific example JSON, would be to NOT use a map at all but instead change the structure of OHLC_RESS so that it matches the structure of the JSON, i.e.
type OHLC_RESS struct {
Pair []Candles `json:"XXBTZUSD"`
Last int64 `json:"last"`
}
https://go.dev/play/p/Z9PhJt3wX33
However it's safe to assume, I think, that the reason you've opted to use a map is because the JSON object's key(s) that hold the "pairs" can vary and so hardcoding them into the field's tag is out of the question.
To understand why your code doesn't produce the desired result, you have to realize two things. First, the order of a struct's fields has no bearing on how the keys of a JSON object will be decoded. Second, the name Pair holds no special meaning for the unmarshaler. Therefore, by default, the unmarshaler has no way of knowing that your wish is to decode the "XXBTZUSD": [ ... ] element into the Pair map.
So, to get your desired result, you can have the OHLC_RESS implement the json.Unmarshaler interface and do the following:
func (r *OHLC_RESS) UnmarshalJSON(d []byte) error {
// first, decode just the object's keys and leave
// the values as raw, non-decoded JSON
var obj map[string]json.RawMessage
if err := json.Unmarshal(d, &obj); err != nil {
return err
}
// next, look up the "last" element's raw, non-decoded value
// and, if it is present, then decode it into the Last field
if last, ok := obj["last"]; ok {
if err := json.Unmarshal(last, &r.Last); err != nil {
return err
}
// remove the element so it's not in
// the way when decoding the rest below
delete(obj, "last")
}
// finally, decode the rest of the element values
// in the object and store them in the Pair field
r.Pair = make(map[string][]Candles, len(obj))
for key, val := range obj {
cc := []Candles{}
if err := json.Unmarshal(val, &cc); err != nil {
return err
}
r.Pair[key] = cc
}
return nil
}
https://go.dev/play/p/Lj8a8Gx9fWH
Golang encoding/json package lets you use ,string struct tag in order to marshal/unmarshal string values (like "309230") into int64 field. Example:
Int64String int64 `json:",string"`
However, this doesn't work for slices, ie. []int64:
Int64Slice []int64 `json:",string"` // Doesn't work.
Is there any way to marshal/unmarshal JSON string arrays into []int64 field?
Quote from https://golang.org/pkg/encoding/json:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:
For anyone interested, I found a solution using a custom type having MarshalJSON() and UnmarshalJSON() methods defined.
type Int64StringSlice []int64
func (slice Int64StringSlice) MarshalJSON() ([]byte, error) {
values := make([]string, len(slice))
for i, value := range []int64(slice) {
values[i] = fmt.Sprintf(`"%v"`, value)
}
return []byte(fmt.Sprintf("[%v]", strings.Join(values, ","))), nil
}
func (slice *Int64StringSlice) UnmarshalJSON(b []byte) error {
// Try array of strings first.
var values []string
err := json.Unmarshal(b, &values)
if err != nil {
// Fall back to array of integers:
var values []int64
if err := json.Unmarshal(b, &values); err != nil {
return err
}
*slice = values
return nil
}
*slice = make([]int64, len(values))
for i, value := range values {
value, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return err
}
(*slice)[i] = value
}
return nil
}
The above solution marshals []int64 into JSON string array. Unmarshaling works from both JSON string and integer arrays, ie.:
{"bars": ["1729382256910270462", "309286902808622", "23"]}
{"bars": [1729382256910270462, 309286902808622, 23]}
See example at https://play.golang.org/p/BOqUBGR3DXm
As you quoted from json.Marshal(), the ,string option only applies to specific types, namely:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types.
You want it to work with a slice, but that is not supported by the json package.
If you still want this functionality, you have to write your custom marshaling / unmarshaling logic.
What you presented works, but it is unnecessarily complex. This is because you created your custom logic on slices, but you only want this functionality on individual elements of the slices (arrays). You don't want to change how an array / slice (as a sequence of elements) is rendered or parsed.
So a much simpler solution is to only create a custom "number" type producing this behavior, and elements of slices of this custom type will behave the same.
Our custom number type and the marshaling / unmarshaling logic:
type Int64Str int64
func (i Int64Str) MarshalJSON() ([]byte, error) {
return json.Marshal(strconv.FormatInt(int64(i), 10))
}
func (i *Int64Str) UnmarshalJSON(b []byte) error {
// Try string first
var s string
if err := json.Unmarshal(b, &s); err == nil {
value, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return err
}
*i = Int64Str(value)
return nil
}
// Fallback to number
return json.Unmarshal(b, (*int64)(i))
}
And that's all!
The type using it:
type Foo struct {
Bars []Int64Str `json:"bars"`
}
Testing it the same way as you did yields the same result. Try it on the Go Playground.
I'm calling an API which will return Json objects like this:
{
name: "XXX"
type: "TYPE_1"
shared_fields: {...}
type_1_fields: {...}
..
type_2_fields: {...}
}
Based on different types, this object will have different kinds of fields, but these fields are certain for different types.
So, I unmarshal the Json string to map[string]interface{} to fetch the the different type but how can I turn these map[string]interface{} to a certain struct?
var f map[string]interface{}
err := json.Unmarshal(b, &f)
type := f["type"]
switch type {
case "type_1":
//initialize struct of type_1
case "type_2":
//initialize struct of type_2
}
For this sort of two-step json decoding you will probably want to check out json.RawMessage. It allows you to defer processing of parts of your json response. The example in the docs shows how.
One way to do it would be to have a constructor function (one that starts with New…) that takes a map as input parameter.
A second way, much slower in my opinion, would be to redo the unmarshalling to the right struct type.
If the types are different enough and you want to be lazy you can just try to decode it in each format:
f1 := type1{}
err := json.Unmarshal(b, &f1)
if err == nil {
return f1
}
f2 := type2{}
err := json.Unmarshal(b, &f2)
if err == nil {
return f2
}
...
If the objects are similar or you want to be less lazy, you could just decode the type and then do something:
type BasicInfo struct {
Type string `json:"type"`
}
f := BasicInfo{}
err := json.Unmarshal(b, &f)
switch f.Type {
case "type_1":
//initialize struct of type_1
case "type_2":
//initialize struct of type_2
}
I have this struct
type SyncInfo struct {
Target string
}
Now I query some json data from ElasticSearch. Source is of type json.RawMessage.
All I want is to map source to my SyncInfo which I created the variable mySyncInfo for.
I even figured out how to do that...but it seems weird. I first call MarshalJSON() to get a []byte and then feed that to json.Unmarshal() which takes an []byte and a pointer to my struct.
This works fine but it feels as if I'm doing an extra hop. Am I missing something or is that the intended way to get from a json.RawMessage to a struct?
var mySyncInfo SyncInfo
jsonStr, _ := out.Hits.Hits[0].Source.MarshalJSON()
json.Unmarshal(jsonStr, &mySyncInfo)
fmt.Print(mySyncInfo.Target)
As said, the underlying type of json.RawMessage is []byte, so you can use a json.RawMessage as the data parameter to json.Unmarshal.
However, your problem is that you have a pointer (*json.RawMessage) and not a value. All you have to do is to dereference it:
err := json.Unmarshal(*out.Hits.Hits[0].Source, &mySyncInfo)
Working example:
package main
import (
"encoding/json"
"fmt"
)
type SyncInfo struct {
Target string
}
func main() {
data := []byte(`{"target": "localhost"}`)
Source := (*json.RawMessage)(&data)
var mySyncInfo SyncInfo
// Notice the dereferencing asterisk *
err := json.Unmarshal(*Source, &mySyncInfo)
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", mySyncInfo)
}
Output:
{Target:localhost}
Playground: http://play.golang.org/p/J8R3Qrjrzx
json.RawMessage is really just a slice of bytes. You should be able to feed it directly into json.Unmarshal directly, like so:
json.Unmarshal(out.Hits.Hits[0].Source, &mySyncInfo)
Also, somewhat unrelated, but json.Unmarshal can return an error and you want to handle that.
err := json.Unmarshal(*out.Hits.Hits[0].Source, &mySyncInfo)
if err != nil {
// Handle
}