I have data like this
"descriptionMap": [[[1,2], "a"], [[3,4], "b"]]
and I was trying to decode it with
DescriptionMap []struct {
OpcodeTableIdPair []int
OpcodeDescription string
} `json:"descriptionMap"`
but I keep on getting empty arrays,
[[{[] } {[] }]]
You have a very unfortunate JSON schema which treats arrays as objects. The best you can do in this situation is something like this:
type Body struct {
DescriptionMap []Description `json:"descriptionMap"`
}
type Description struct {
IDPair []int
Description string
}
func (d *Description) UnmarshalJSON(b []byte) error {
arr := []interface{}{}
err := json.Unmarshal(b, &arr)
if err != nil {
return err
}
idPair := arr[0].([]interface{})
d.IDPair = make([]int, len(idPair))
for i := range idPair {
d.IDPair[i] = int(idPair[i].(float64))
}
d.Description = arr[1].(string)
return nil
}
Playground: https://play.golang.org/p/MPho12GJfc.
Notice though that this will panic if any of the types in the JSON don't match. You can create a better version which returns errors in such cases.
Related
I need to parse this json
{
"version": "1.1.29-snapshot",
"linux-amd64": {
"url": "https://origin/path",
"size": 7794688,
"sha256": "14b3c3ad05e3a98d30ee7e774646aec7ffa8825a1f6f4d9c01e08bf2d8a08646"
},
"windows-amd64": {
"url": "https://origin/path",
"size": 8102400,
"sha256": "01b8b927388f774bdda4b5394e381beb592d8ef0ceed69324d1d42f6605ab56d"
}
}
Keys like linux-amd64 are dynamic and theirs amount is arbitrary. I tried something like that to describe it and unmarshal. Obviously it doesn't work. Items is always empty.
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
It's similar to this use case, but has no parent key items. I suppose I can use 3rd party library or map[string]interface{} approach, but I'm interested in knowing how to achieve this with explicitly declared types.
The rest of the parsing code is:
func parseUpdateJson(jsonStr []byte) (UpdateInfo, error) {
var allInfo = UpdateInfo{Items: make(map[string]FileInfo)}
var err = json.Unmarshal(jsonStr, &allInfo)
return allInfo, err
}
Look at the link I attached and you will realize that is not that simple as you think. Also I pointed that I interested in typed approach. Ok, how to declare this map[string]FileInfo to get parsed?
You can create a json.Unmarshaller to decode the json into a map, then apply those values to your struct: https://play.golang.org/p/j1JXMpc4Q9u
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
func (i *UpdateInfo) UnmarshalJSON(d []byte) error {
tmp := map[string]json.RawMessage{}
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
err = json.Unmarshal(tmp["version"], &i.Version)
if err != nil {
return err
}
delete(tmp, "version")
i.Items = map[string]FileInfo{}
for k, v := range tmp {
var item FileInfo
err := json.Unmarshal(v, &item)
if err != nil {
return err
}
i.Items[k] = item
}
return nil
}
This answer is adapted from this recipe in my YouTube video on advanced JSON handling in Go.
func (u *UpdateInfo) UnmarshalJSON(d []byte) error {
var x struct {
UpdateInfo
UnmarshalJSON struct{}
}
if err := json.Unmarshal(d, &x); err != nil {
return err
}
var y map[string]json.RawMessage{}
if err := json.Unsmarshal(d, &y); err != nil {
return err
}
delete(y, "version"_ // We don't need this in the map
*u = x.UpdateInfo
u.Items = make(map[string]FileInfo, len(y))
for k, v := range y {
var info FileInfo
if err := json.Unmarshal(v, &info); err != nil {
return err
}
u.Items[k] = info
}
return nil
}
It:
Unmarshals the JSON into the struct directly, to get the struct fields.
It re-unmarshals into a map of map[string]json.RawMessage to get the arbitrary keys. This is necessary since the value of version is not of type FileInfo, and trying to unmarshal directly into map[string]FileInfo will thus error.
It deletes the keys we know we already got in the struct fields.
It then iterates through the map of string to json.RawMessage, and finally unmarshals each value into the FileInfo type, and stores it in the final object.
If you really don't want to unmarshal multiple times, your next best option is to iterate over the JSON tokens in your input by using the json.Decoder type. I've done this in a couple of performance-sensitive bits of code, but it makes your code INCREDIBLY hard to read, and in almost all cases is not worth the effort.
For the following JSON response {"table_contents":[{"id":100,"description":"text100"},{"id":101,"description":"text101"},{"id":1,"description":"text1"}]}
All you have to do is to produce the following code to execute it properly and be able to reads fields from the struct, such as:
package main
import (
"fmt"
"encoding/json"
)
type MyStruct1 struct {
TableContents []struct {
ID int
Description string
} `json:"table_contents"`
}
func main() {
result:= []byte(`{"table_contents":[{"id":100,"description":"text100"},{"id":101,"description":"text101"},{"id":1,"description":"text1"}]}`)
var container MyStruct1
err := json.Unmarshal(result, &container)
if err != nil {
fmt.Println(" [0] Error message: " + err.Error())
return
}
for i := range container.TableContents {
fmt.Println(container.TableContents[i].Description)
}
}
But how do you deal with the following JSON response? {"table_contents":[[{"id":100,"description":"text100"},{"id":101,"description":"text101"}],{"id":1,"description":"text1"}]} You can either get this response or the one above, it is important to modify the struct to accept both.
I did something like this, with the help of internet:
package main
import (
"fmt"
"encoding/json"
)
type MyStruct1 struct {
TableContents []TableContentUnion `json:"table_contents"`
}
type TableContentClass struct {
ID int
Description string
}
type TableContentUnion struct {
TableContentClass *TableContentClass
TableContentClassArray []TableContentClass
}
func main() {
result:= []byte(`{"table_contents":[[{"id":100,"description":"text100"},{"id":101,"description":"text101"}],{"id":1,"description":"text1"}]}`)
var container MyStruct1
err := json.Unmarshal(result, &container)
if err != nil {
fmt.Println(" [0] Error message: " + err.Error())
return
}
for i := range container.TableContents {
fmt.Println(container.TableContents[i])
}
}
but it does not go past the error message :(
[0] Error message: json: cannot unmarshal array into Go struct field MyStruct1.table_contents of type main.TableContentUnion*
Been struggling to come up with a solution for hours. If someone could help I would be happy. Thank you for reading. Let me know if you have questions
Inside table_contents you have two type options (json object or list of json objects). What you can do is to unmarshall into an interface and then run type-check on it when using it:
type MyStruct1 struct {
TableContents []interface{} `json:"table_contents"`
}
...
for i := range container.TableContents {
switch container.TableContents[i].(type){
case map[string]interface{}:
fmt.Println("json object")
case []interface{}:
fmt.Println("list")
}
}
From there you can use some library (e.g. https://github.com/mitchellh/mapstructure) to map unmarshalled struct to your TableContentClass type. See PoC playground here: https://play.golang.org/p/NhVUhQayeL_C
Custom UnmarshalJSON function
You can also create a custom UnmarshalJSON function on the object that has the 2 possibilities. In you case that would be TableContentUnion.
In the custom unmarshaller you can then decide how to unmarshal the content.
func (s *TableContentUnion) UnmarshalJSON(b []byte) error {
// Note that we get `b` as bytes, so we can also manually check to see
// if it is an array (starts with `[`) or an object (starts with `{`)
var jsonObj interface{}
if err := json.Unmarshal(b, &jsonObj); err != nil {
return err
}
switch jsonObj.(type) {
case map[string]interface{}:
// Note: instead of using json.Unmarshal again, we could also cast the interface
// and build the values as in the example above
var tableContentClass TableContentClass
if err := json.Unmarshal(b, &tableContentClass); err != nil {
return err
}
s.TableContentClass = &tableContentClass
case []interface{}:
// Note: instead of using json.Unmarshal again, we could also cast the interface
// and build the values as in the example above
if err := json.Unmarshal(b, &s.TableContentClassArray); err != nil {
return err
}
default:
return errors.New("TableContentUnion.UnmarshalJSON: unknown content type")
}
return nil
}
The rest then works like in your test code that was failing before. Here the working Go Playground
Unmarshal to map and manually build struct
You can always unmarshal a json (with an object at the root) into a map[string]interface{}. Then you can iterate things and further unmarshal them after checking what type they are.
Working example:
func main() {
result := []byte(`{"table_contents":[[{"id":100,"description":"text100"},{"id":101,"description":"text101"}],{"id":1,"description":"text1"}]}`)
var jsonMap map[string]interface{}
err := json.Unmarshal(result, &jsonMap)
if err != nil {
fmt.Println(" [0] Error message: " + err.Error())
return
}
cts, ok := jsonMap["table_contents"].([]interface{})
if !ok {
// Note: nil or missing 'table_contents" will also lead to this path.
fmt.Println("table_contents is not a slice")
return
}
var unions []TableContentUnion
for _, content := range cts {
var union TableContentUnion
if contents, ok := content.([]interface{}); ok {
for _, content := range contents {
contCls := parseContentClass(content)
if contCls == nil {
continue
}
union.TableContentClassArray = append(union.TableContentClassArray, *contCls)
}
} else {
contCls := parseContentClass(content)
union.TableContentClass = contCls
}
unions = append(unions, union)
}
container := MyStruct1{
TableContents: unions,
}
for i := range container.TableContents {
fmt.Println(container.TableContents[i])
}
}
func parseContentClass(value interface{}) *TableContentClass {
m, ok := value.(map[string]interface{})
if !ok {
return nil
}
return &TableContentClass{
ID: int(m["id"].(float64)),
Description: m["description"].(string),
}
}
This is most useful if the json has too many variations. For cases like this it might also make sense sometimes to switch to a json package that works differently like https://github.com/tidwall/gjson which gets values based on their path.
Use json.RawMessage to capture the varying parts of the JSON document. Unmarshal each raw message as appropriate.
func (ms *MyStruct1) UnmarshalJSON(data []byte) error {
// Declare new type with same base type as MyStruct1.
// This breaks recursion in call to json.Unmarshal below.
type x MyStruct1
v := struct {
*x
// Override TableContents field with raw message.
TableContents []json.RawMessage `json:"table_contents"`
}{
// Unmarshal all but TableContents directly to the
// receiver.
x: (*x)(ms),
}
err := json.Unmarshal(data, &v)
if err != nil {
return err
}
// Unmarahal raw elements as appropriate.
for _, tcData := range v.TableContents {
if bytes.HasPrefix(tcData, []byte{'{'}) {
var v TableContentClass
if err := json.Unmarshal(tcData, &v); err != nil {
return err
}
ms.TableContents = append(ms.TableContents, v)
} else {
var v []TableContentClass
if err := json.Unmarshal(tcData, &v); err != nil {
return err
}
ms.TableContents = append(ms.TableContents, v)
}
}
return nil
}
Use it like this:
var container MyStruct1
err := json.Unmarshal(result, &container)
if err != nil {
// handle error
}
Run it on the Go playground.
This approach does not add any outside dependencies. The function code does not need to modified when fields are added or removed from MyStruct1 or TableContentClass.
I am working with an API that sends JSON data. The problem is that an array of a single element shows up as a single value. For example, consider the following JSON:
{ "names": ["Alice","Bob"] }
The API sends this as an array. But when the names field has a single element, the API sends this:
{ "names": "Alice" }
This is how I would normally decode this of response in Go:
type Response struct {
Names []string `json:"names"`
}
// later
d := &Response{}
_ = json.NewDecoder(resp).Decode(d) // resp would be a http.Response.Body containing the problematic JSON
Go decodes the first JSON correctly. However, after decoding the second JSON, the object contains an empty array.
I don't have any control over the API, so I have to get around this problem. How could I decode this JSON correctly in Go, so that the Names slice contains a single element? Thank you for your help.
You could implement the json.Unmarshaler interface and have it check the 0th raw byte for [ or " to find out whether it's an array or a string respectivelly:
type StringSlice []string
func (ss *StringSlice) UnmarshalJSON(data []byte) error {
if data[0] == '[' {
return json.Unmarshal(data, (*[]string)(ss))
} else if data[0] == '"' {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*ss = append(*ss, s)
}
return nil
}
https://play.golang.com/p/2GEJsS2YOLJ
You'll have to decode this into an interface{} and then use a type assertion to check whether the underlying type is a slice or just a string.
type Response struct {
Names interface{} `json:"names"`
}
Then after decoding into d, you'd do something like:
slice, ok := d.Names.([]interface{})
if ok {
// it was a slice. use it.
} else {
// it wasn't a slice - so expect it to be a string
// and use that, etc.
}
Use json.RawMessage as type. RawMessage simply delays the decoding of part of a message, so we can do it ourselves later.
type Response struct {
NamesRaw json.RawMessage `json:"names"`
Names []string
}
First, decode the response and then using json.Unmarshal decode json.RawMessage
x := &Response{}
_ = json.NewDecoder(resp).Decode(x);
x.Names = DecodeName(x.NamesRaw)
DecodeName used for decoding NameRaw data
func DecodeName(nameRaw json.RawMessage) (data []string) {
var s string
if err := json.Unmarshal(nameRaw, &s); err == nil {
v := []string{s}
return v
}
var sn []string
if err := json.Unmarshal(nameRaw, &sn); err == nil {
return sn
}
return
}
I have a database table called http_requests. I have modelled following struct to represent rows in this table.
type Map map[string]interface{}
type HTTPRequest struct {
ID int64 `json:"id" gorm:"id"`
RequestURL string `json:"request_url,omitempty" gorm:"request_url"`
RequestParams *RequestParams `json:"request_params,omitempty" gorm:"request_params"`
}
// RequestParams is another struct that holds params from body and URL query
type RequestParams struct {
FromBody Map `json:"body,omitempty"`
FromQuery Map `json:"query,omitempty"`
}
Code to save HTTPRequest:
request := &HTTPRequest{
RequestURL: "dummy/url",
RequestParams: &RequestParams{FromBody: Map{"param1": "value1"}},
}
if err := gorm.DB.Create(request).Error; err != nil {
return err
}
When I try to save this HTTPRequest it results in error:
sql: Scan error on column index 9, name "request_params": unsupported Scan, storing driver.Value type []uint8 into type *RequestParams
I would like to have request_params column to store JSON like this:
{"body":{"param1":"value1"}, "query": {"param2" : "value2"} }
or
{"body":{"param1":"value1"}}
or
{"query": {"param2" : "value2"} }
And this should get parsed into RequestParams struct when reading from database.
As suggested by #mkopriva, I implemented Scan() and Value() methods for my RequestParams type. See code below.
import (
"database/sql/driver"
"encoding/json"
"strings"
)
// Value converts RequestParams to a map
func (reqParams RequestParams) Value() (driver.Value, error) {
reqMap, err := reqParams.ToMap()
if err != nil {
return nil, err
}
return reqMap.ForceJSON(), nil
}
// Scan converts value to RequestParams
func (reqParams *RequestParams) Scan(value interface{}) error {
// set empty struct by default
*reqParams = RequestParams{}
if value == nil {
return nil
}
if s, ok := value.([]byte); ok {
d := json.NewDecoder(strings.NewReader(string(s)))
d.UseNumber()
rp := &RequestParams{}
if err := d.Decode(rp); err == nil {
*reqParams = *rp
}
}
return nil
}
I'm using an API that formats its responses in this way:
{
"err": 0,
"data": **Other json structure**
}
The way I'm getting a response right now is I'm putting the response in an struct like this:
type Response struct {
Err int `json:"err"`
Data interface{} `json:"data"`
}
and then I'm doing this after getting a response
jsonbytes, _ := json.Marshal(resp.Data)
json.Unmarshal(jsonBytes, &dataStruct)
I'm only ignoring errors for this example.
It seems kinda weird to me that I'm marshaling and unmarshaling when I know what the data is supposed to look like and what type it's supposed to be.
Is there a more simple solution that I'm not seeing or is this a normal thing to do?
Edit: I should probably mention that the Data attribute in the response object can vary depending on what API call I'm doing.
The JSON unmarshaller uses reflection to look at the type it is unmarshalling to. Given an uninitialised interface{} as the destination for the unmarshalled data, a JSON object gets unmarshalled into a map[string]interface{} (example in playground).
Here are some ideas.
Option A
If you know the datatype, you can define a new response struct for each type. Example:
type FooResponse struct {
Err int `json:"err"`
Data Foo `json:"data"`
}
type Foo struct {
FooField string `json:"foofield"`
}
type BarResponse struct {
Err int `json:"err"`
Data Bar `json:"data"`
}
type Bar struct {
BarField string `json:"barfield"`
}
Option B
If you prefer to have a single Response struct instead of one per type, you can tell the JSON unmarshaller to avoid unmarshalling the data field until a later time by using the json.RawMessage data type:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Response struct {
Err int `json:"err"`
Data json.RawMessage `json:"data"`
}
type Foo struct {
FooField string `json:"foofield"`
}
type Bar struct {
BarField string `json:"barfield"`
}
func main() {
fooRespJSON := []byte(`{"data":{"foofield":"foo value"}}`)
barRespJSON := []byte(`{"data":{"barfield":"bar value"}}`)
var (
resp Response
foo Foo
bar Bar
)
// Foo
if err := json.Unmarshal(fooRespJSON, &resp); err != nil {
log.Fatal(err)
}
if err := json.Unmarshal(resp.Data, &foo); err != nil {
log.Fatal(err)
}
fmt.Println("foo:", foo)
// Bar
if err := json.Unmarshal(barRespJSON, &resp); err != nil {
log.Fatal(err)
}
if err := json.Unmarshal(resp.Data, &bar); err != nil {
log.Fatal(err)
}
fmt.Println("bar:", bar)
}
Output:
foo: {foo value}
bar: {bar value}
https://play.golang.org/p/Y7D4uhaC4a8
Option C
A third option, as pointed out by #mkopriva in a comment on the question, is to use interface{} as an intermediary datatype and pre-initialise this to a known datatype.
Emphasis lies on the word intermediary -- of course passing around an interface{} is best avoided (Rob Pike's Go Proverbs). The use-case here is to allow any datatype to be used without the need for multiple different Response types. On way to avoid exposing the interface{} is to wrap the response completely, exposing only the data and the error:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Foo struct {
FooField string `json:"foofield"`
}
type Bar struct {
BarField string `json:"barfield"`
}
type Error struct {
Code int
}
func (e *Error) Error() string {
return fmt.Sprintf("error code %d", e.Code)
}
func unmarshalResponse(data []byte, v interface{}) error {
resp := struct {
Err int `json:"err"`
Data interface{} `json:"data"`
}{Data: v}
if err := json.Unmarshal(data, &resp); err != nil {
return err
}
if resp.Err != 0 {
return &Error{Code: resp.Err}
}
return nil
}
func main() {
fooRespJSON := []byte(`{"data":{"foofield":"foo value"}}`)
barRespJSON := []byte(`{"data":{"barfield":"bar value"}}`)
errRespJSON := []byte(`{"err": 123}`)
// Foo
var foo Foo
if err := unmarshalResponse(fooRespJSON, &foo); err != nil {
log.Fatal(err)
}
fmt.Println("foo:", foo)
// Bar
var bar Bar
if err := unmarshalResponse(barRespJSON, &bar); err != nil {
log.Fatal(err)
}
fmt.Println("bar:", bar)
// Error response
var v interface{}
if err := unmarshalResponse(errRespJSON, &v); err != nil {
log.Fatal(err)
}
}
Output:
foo: {foo value}
bar: {bar value}
2009/11/10 23:00:00 error code 123
https://play.golang.org/p/5SVfQGwS-Wy