Is there a way to unmarshall JSON arrays into single objects in Go?
I have a json response from an endpoint:
{
"results": [
{
"key": "value"
}
]
}
I have a Go struct for the object inside the array:
type Object struct {
Key string `json:"key"`
}
...and a struct for the response object:
type Response struct {
Objects []Object `json:"results"`
}
results is an array of objects, but I know that the endpoint will only ever return an array with 1 object. Is there a way to unmarshall the data and avoid having reference the object by an index? I was hoping I could use something like json:"results[0]" as a field tag.
I'd prefer to be able to:
decoder.Decode(&response)
response.Object.Key
Rather than
decoder.Decode(&response)
response.Objects[0].Key
To does this you need to customize unmarshalling.
An way is create a ResponseCustom like:
//Response json (customized) that match with Unmarshaler interface
type ResponseCustom struct {
Object
}
func (r *ResponseCustom) UnmarshalJSON(b []byte) error{
rsp := &Response{}
err := json.Unmarshal(b, rsp)
if err != nil {
log.Fatalln("error:", err)
} else {
r.Object = rsp.Objects[0]
}
//
return err
}
So you can use ResponseCustom instead of you Response for get Object value.
Look:
func main() {
//
data := []byte(jsondata)
resp := &ResponseCustom{}
//
json.Unmarshal(data, resp)
//
fmt.Println("My Object.value is: " + resp.Object.Key)
}
The result is:
My Object.value is: value
In playground: https://play.golang.org/p/zo7wOSacA4w
Implement unmarshaler interface to convert array of object to object. Fetch the value for key and then unmarshal your json as
package main
import(
"fmt"
"encoding/json"
)
type Object struct {
Key string `json:"key"`
}
func (obj *Object) UnmarshalJSON(data []byte) error {
var v map[string]interface{}
if err := json.Unmarshal(data, &v); err != nil {
fmt.Printf("Error whilde decoding %v\n", err)
return err
}
obj.Key = v["results"].(interface{}).([]interface{})[0].(interface{}).(map[string]interface{})["key"].(interface{}).(string)
return nil
}
func main(){
var obj Object
data := []byte(`{
"results": [
{
"key": "value"
}
]
}`)
if err := json.Unmarshal(data, &obj); err!=nil{
fmt.Println(err)
}
fmt.Printf("%#v", obj)
}
Playground
Related
I'm trying to unmarshal the following JSON string
token = `{
"id": 1,
"token": {
"id": 2248637,
"metadata": {
"name": "Name #1",
"formats": "[{\"mimeType\": \"model/gltf-binary\", \"uri\": \"uri1\", {\"mimeType\": \"image/gif\", \"uri\": \"uri2\"}]"
}
}`
I can unmarshal it with 2 phases like this. However, I would like to use custom unmarshalJSON but I fail. I got error
My code as follow:
type FileFormat struct {
MIMEType string
URI string
}
func (format *FileFormat) UnmarshalJSON(data []byte) error {
var aux []interface{}
if err := json.Unmarshal(data, &aux); err != nil {
return err
}
format.MIMEType = aux[0].(string)
format.URI = aux[1].(string)
return nil
}
type TokenMetadata struct {
Name string `json:"name"`
Formats []FileFormat `json:"formats"`
}
type Token struct {
ID TokenID `json:"tokenId"`
Metadata TokenMetadata `json:"metadata"`
}
func main() {
var tokenRes OwnedToken
if err := json.Unmarshal([]byte(token), &tokenRes); err != nil {
fmt.Println(err)
}
}
And the error is
json: cannot unmarshal string into Go struct field TokenMetadata.token.metadata.formats of type []main.FileFormat
How can I fix this problem? Many thanks!
The JSON array of file formats is double encoded. Declare a Go type corresponding to the array. Double decode in the UnmarshalJSON method for that type.
type FileFormats []FileFormat
func (ff *FileFormats) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
return json.Unmarshal(
[]byte(s),
(*[]FileFormat)(ff))
}
type TokenMetadata struct {
Name string `json:"name"`
Formats FileFormats `json:"formats"`
}
Note: The conversion from from *FileFormats to *[]FileFormat is required to prevent recursion.
i'm having a lot of trouble getting my program to work. I want to unmarshal something pretty simple, but it's giving me a lot of issues, unfortunately.
Here is the response that I want to unmarshal:
{"error":[],"result":{"XXBTZUSD":[[1647365820,"39192.0","39192.0","39191.9","39191.9","39191.9","0.18008008",10],[1647365880,"39186.1","39186.1","39172.0","39176.0","39174.4","0.13120077",10]],"last":1647408900}}
I've wrote these structs to help with unmarshalling
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type Trades struct {
Pair []OHLC `json:"XXBTZUSD"`
Last float64 `json:"last"`
}
type OHLC struct {
Time float64
Open string
High string
Low string
Close string
Vwa string
Volume string
Count float64
}
I have a function call that makes the http request and then unmarshals the data. For whatever reason, my code will end before even starting the function call for the http request and subsequent unmarshalling when the Pair type is []OHLC or []*OHLC. If I change the Pair type to interface{}, then it runs. i want to make it work with the OHLC struct instead though. Below is the complete code:
package main
import (
"fmt"
"net/http"
//"strings"
"io/ioutil"
"encoding/json"
)
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type Trades struct {
Pair []OHLC `json:"XXBTZUSD"`
Last float64 `json:"last"`
}
type OHLC struct {
TT float64
Open string
High string
Low string
Close string
Vwap string
Volume string
Count float64
}
/*func main() {
var data = [...]Trade{
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
}
}*/
func main() {
fmt.Println("in main");
getOhlc()
}
func getOhlc() {
fmt.Println("in ohlc func")
resp, err := http.Get("https://api.kraken.com/0/public/OHLC?pair=XXBTZUSD");
if err != nil {
fmt.Errorf("error after request")
return;
}
defer resp.Body.Close();
body, err := ioutil.ReadAll(resp.Body);
if err != nil {
fmt.Errorf("error when reading")
return;
}
var jsonData Resp;
err = json.Unmarshal(body, &jsonData);
if err != nil {
fmt.Errorf("error when unmarshalling")
return
}
if(len(jsonData.Error) > 0) {
fmt.Errorf("error");
return;
}
fmt.Println(jsonData);
}
Any ideas about what might be happening?
"Any ideas about what might be happening?"
The elements in the "XXBTZUSD" JSON array are arrays themselves, i.e. "XXBTZUSD" is an array of arrays. The OHLC type is a struct type. The stdlib will not, by itself, unmarshal a JSON array into a Go struct. Go structs can be used to unmarshal JSON objects. JSON arrays can be unmarshaled into Go slices or arrays.
You would clearly see that that's the issue if you would just print the error from json.Unmarshal:
json: cannot unmarshal array into Go struct field
Trades.result.XXBTZUSD of type main.OHLC
https://go.dev/play/p/D4tjXZVzDI_w
If you want to unmarshal a JSON array into a Go struct you have to have the Go struct type implement a the json.Unmarshaler interface.
func (o *OHLC) UnmarshalJSON(data []byte) error {
// first unmarshal the array into a slice of raw json
raw := []json.RawMessage{}
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
// create a function that unmarshals each raw json element into a field
unmarshalFields := func(raw []json.RawMessage, fields ...interface{}) error {
if len(raw) != len(fields) {
return errors.New("bad number of elements in json array")
}
for i := range raw {
if err := json.Unmarshal([]byte(raw[i]), fields[i]); err != nil {
return err
}
}
return nil
}
// call the function
return unmarshalFields(
raw,
&o.Time,
&o.Open,
&o.High,
&o.Low,
&o.Close,
&o.Vwa,
&o.Volume,
&o.Count,
)
}
https://go.dev/play/p/fkFKLkaNaSU
Your code had some issues:
Remove semicolons from end of lines, it's redundant.
fmt.Errorf return error, and not print it, every time check your error and propagate it.
We can convert array of numbers and string to struct in golang.
for achieving your desired output we need to first convert to intermediate container and then convert to our wanted output:
package main
import (
"errors"
"fmt"
"log"
"net/http"
//"strings"
"encoding/json"
"io/ioutil"
)
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type IntermediateResp struct {
Error []string `json:"error"`
Result IntermediateTrades `json:"result"`
}
type IntermediateTrades struct {
Pair [][]interface{} `json:"XXBTZUSD"`
Last int `json:"last"`
}
type Trades struct {
Pair []OHLC `json:"result"`
Last int `json:"last"`
}
type OHLC struct {
TT float64
Open string
High string
Low string
Close string
Vwap string
Volume string
Count float64
}
/*func main() {
var data = [...]Trade{
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
}
}*/
func main() {
fmt.Println("in main")
err := getOhlc()
if err != nil {
log.Fatal(err)
}
}
func buildOHLC(l []interface{}) (*OHLC, error) {
if len(l) < 8 {
return nil, errors.New("short list")
}
return &OHLC{
TT: l[0].(float64),
Open: l[1].(string),
High: l[2].(string),
Low: l[3].(string),
Close: l[4].(string),
Vwap: l[5].(string),
Volume: l[6].(string),
Count: l[7].(float64),
}, nil
}
func convert(r IntermediateResp) (*Resp, error) {
result := &Resp{Error: r.Error, Result: Trades{Pair: make([]OHLC, len(r.Result.Pair)), Last: r.Result.Last}}
for i, v := range r.Result.Pair {
ohlc, err := buildOHLC(v)
if err != nil {
return nil, err
}
result.Result.Pair[i] = *ohlc
}
return result, nil
}
func getOhlc() error {
fmt.Println("in ohlc func")
resp, err := http.Get("https://api.kraken.com/0/public/OHLC?pair=XXBTZUSD")
if err != nil {
return fmt.Errorf("error after request, %v", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fmt.Println(string(body))
if err != nil {
return fmt.Errorf("error when reading %v", err)
}
var jsonData IntermediateResp
err = json.Unmarshal(body, &jsonData)
if err != nil {
return fmt.Errorf("error when unmarshalling %v", err)
}
if len(jsonData.Error) > 0 {
return fmt.Errorf("error")
}
convertedOhlc, err := convert(jsonData)
if err != nil {
return fmt.Errorf("error when convertedOhlc %v", err)
}
fmt.Println(convertedOhlc)
return nil
}
We define IntermediateResp and IntermediateTrades for Unmarshaling json and then convert it to actual Resp.
I think aother way is using custom Unmarshal for Trades struct.
My apologies for the basic question. I am new to Golang and I have the json to parse as below
{
"config1":{
"Parameters":{
"Pm1":"value",
"Pm2":"value",
"Pm3":"value"
},
"dynamic_key1":{
"Parameters":{
"a":"value",
"b":"value",
"c":"value",
"d":"value"
},
"Epoch":"value"
},
"Epoch":"value"
}
}
I am trying to write a struct to parse this json and wrote the struct in the following way.
type Parameters struct {
Pm1 string `json:"Pm1"`
Pm2 string `json:"Pm2"`
Pm3 string `json:"Pm3"`
}
type dynamicParametes struct {
a string `json:"a"`
b string `json:"b"`
c string `json:"c"`
d string `json:"d"`
}
type dynamic struct {
Parameters dynamicParametes `json:"Parameters"`
Epoch string `json:"Epoch"`
}
type config1 struct {
Parameters Parameters `json:"Parameters"`
Dynamic_keys map[string]dynamic `json:"-"`
Epoch string `json:"Epoch"`
}
type config struct {
config1 config1 `json:"config1"`
}
I was hoping that the map will match all the matching keys with dynamic structs and show them in the map. But, I see it created an empty map in the response.
Implemented custom unmarshler for config type.
Note
If you don't need Parameters and dynamicParametes as struct types, you can simply unmarshal them into map[string]string
you have to expose all fields in your structs to do json unmarshaling
validate your json string
type config struct {
Config1 config1 `json:"config1"`
}
type _config config
func (b *config) UnmarshalJSON(data []byte) error {
var v = struct {
Config1 map[string]interface{} `json:"config1"`
}{}
if err := json.Unmarshal(data, &v); err != nil {
return err
}
c := _config{}
err := json.Unmarshal(data, &c)
if err != nil {
return err
}
b.Config1.Parameters = c.Config1.Parameters
b.Config1.Epoch = c.Config1.Epoch
if b.Config1.Dynamic_keys == nil {
b.Config1.Dynamic_keys = map[string]dynamic{}
}
for key, config := range v.Config1 {
if key == `Parameters` || key == `Epoch` {
continue
}
data, err := json.Marshal(config)
if err != nil {
return err
}
d := dynamic{}
err = json.Unmarshal(data, &d)
if err != nil {
return err
}
b.Config1.Dynamic_keys[key] = d
}
return nil
}
you can see full code here
All you need is understand how base data types looks in json.
Field Parameters in your json is simple map[string]string and you can unmarshall it with standart json.Unmasrhall without any aditional implementation of interface json.Unmarshaler.
Link for Go Playground with code below
package main
import (
"encoding/json"
"fmt"
)
const jsonStr = `{
"config1":{
"Parameters":{
"Pm1":"value_1",
"Pm2":"value_2",
"Pm3":"value_3"
},
"dynamic_key1":{
"Parameters":{
"a":"value_1",
"b":"value_2",
"c":"value_3",
"d":"value_4"
},
"Epoch":"value"
},
"Epoch":"value"
}
}`
type Data struct {
Config1 struct {
Parameters map[string]string `json:"Parameters"`
Dynamic struct {
Parameters map[string]string `json:"Parameters"`
Epoch string `json:"Epoch"`
} `json:"dynamic_key1"`
Epoch string `json:"Epoch"`
} `json:"config1"`
}
func main() {
var data Data
_ = json.Unmarshal([]byte(jsonStr), &data)
fmt.Printf("%+v\n", data)
}
Output:
{Config1:{Parameters:map[Pm1:value_1 Pm2:value_2 Pm3:value_3] Dynamic:{Parameters:map[a:value_1 b:value_2 c:value_3 d:value_4] Epoch:value} Epoch:value}}
I have a database table called http_requests. I have modelled following struct to represent rows in this table.
type Map map[string]interface{}
type HTTPRequest struct {
ID int64 `json:"id" gorm:"id"`
RequestURL string `json:"request_url,omitempty" gorm:"request_url"`
RequestParams *RequestParams `json:"request_params,omitempty" gorm:"request_params"`
}
// RequestParams is another struct that holds params from body and URL query
type RequestParams struct {
FromBody Map `json:"body,omitempty"`
FromQuery Map `json:"query,omitempty"`
}
Code to save HTTPRequest:
request := &HTTPRequest{
RequestURL: "dummy/url",
RequestParams: &RequestParams{FromBody: Map{"param1": "value1"}},
}
if err := gorm.DB.Create(request).Error; err != nil {
return err
}
When I try to save this HTTPRequest it results in error:
sql: Scan error on column index 9, name "request_params": unsupported Scan, storing driver.Value type []uint8 into type *RequestParams
I would like to have request_params column to store JSON like this:
{"body":{"param1":"value1"}, "query": {"param2" : "value2"} }
or
{"body":{"param1":"value1"}}
or
{"query": {"param2" : "value2"} }
And this should get parsed into RequestParams struct when reading from database.
As suggested by #mkopriva, I implemented Scan() and Value() methods for my RequestParams type. See code below.
import (
"database/sql/driver"
"encoding/json"
"strings"
)
// Value converts RequestParams to a map
func (reqParams RequestParams) Value() (driver.Value, error) {
reqMap, err := reqParams.ToMap()
if err != nil {
return nil, err
}
return reqMap.ForceJSON(), nil
}
// Scan converts value to RequestParams
func (reqParams *RequestParams) Scan(value interface{}) error {
// set empty struct by default
*reqParams = RequestParams{}
if value == nil {
return nil
}
if s, ok := value.([]byte); ok {
d := json.NewDecoder(strings.NewReader(string(s)))
d.UseNumber()
rp := &RequestParams{}
if err := d.Decode(rp); err == nil {
*reqParams = *rp
}
}
return nil
}
I have a PostgreSQL schema with json field's (DisplayInfo, and FormatInfo). Structure of this field's is dynamic.
I'can read and render it only as string (string type in render struct) :
[
{
"ID":9,
"Name":"120 №1",
"DisplayInfo":"{\"path\": \"http://path/to/img.png\"}",
"Format":{
"Code":"frame-120",
"Width":120,
"Height":60,
"FormatInfo":"[{\"name\": \"\\u0413\\u043b\\u0430\\u0432\\u043d\\u043e\\u0435 \\u0438\\u0437\\u043e\\u0431\\u0440\\u0430\\u0436\\u0435\\u043d\\u0438\\u0435\", \"field_type\": \"img\", \"key\": \"path\"}]"
},
"Weight":0.075,
"Application":8,
"Url":"//path/to/game",
"Referrer":""
}
]
but i want output field DisplayInfo as JSON object. How ?
My render code:
func renderJSON(w http.ResponseWriter, obj models.Model) {
js, err := json.Marshal(obj)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Write(js)
}
UPD 1 : Structure of this field's is dynamic. DisplayInfo may have 'path' field, or may not. They may have additional fields.
UPD 2. I wana output DisplayInfo and FormatInfo as json-object(not string), as part of whole object, like this:
[
{
"ID":9,
"Name":"120 №1",
"DisplayInfo":{"path": "http://path/to/img.png"},
"Format":{
"Code":"frame-120",
"Width":120,
"Height":60,
"FormatInfo":[{"name": "\\u0413\\u043b\\u0430\\u0432\\u043d\\u043e\\u0435 \\u0438\\u0437\\u043e\\u0431\\u0440\\u0430\\u0436\\u0435\\u043d\\u0438\\u0435", "field_type": "img", "key": "path"}]
},
"Weight":0.075,
"Application":8,
"Url":"//path/to/game",
"Referrer":""
}
]
UPD 3: Structures
Actual structure is :
type BannerSerializer struct {
ID int
Name string
DisplayInfo string
Format formatSerializer
Weight float32
Application int
Url string
Referrer string
}
Then i trying this structure:
type BannerSerializer struct {
ID int
Name string
DisplayInfo json.RawMessage
Format formatSerializer
Weight float32
Application int
Url string
Referrer string
}
DisplayInfo serialize as base64 string (or like base64, don't know)
Use a pointer to json.RawMessage:
type Data struct {
Obj *json.RawMessage
}
Playground: http://play.golang.org/p/Qq9IUBDLzJ.
Assuming you have access to change models.Model, you can create your own type with a custom Unmarshaler that just returns the raw string:
type JSONString string
func (s JSONString) MarshalJSON() ([]byte, error) {
return []byte(s), nil
}
Working example:
package main
import (
"encoding/json"
"fmt"
)
type JSONString string
func (s JSONString) MarshalJSON() ([]byte, error) {
return []byte(s), nil
}
type Model struct {
ID int
Name string
DisplayInfo JSONString
}
func main() {
data := []byte(`{
"ID":9,
"Name":"120 №1",
"DisplayInfo":"{\"path\": \"http://path/to/img.png\"}"
}`)
var obj Model
err := json.Unmarshal(data, &obj)
if err != nil {
panic(err)
}
// Here comes your code
js, err := json.Marshal(obj)
if err != nil {
panic(err)
}
fmt.Println(string(js))
}
Output:
{"ID":9,"Name":"120 №1","DisplayInfo":{"path":"http://path/to/img.png"}}
Playground: http://play.golang.org/p/6bcnuGjlU8
You'd have to unmarshal it, here's an example:
var data []*struct {
ID int
DisplayInfo string
}
if err := json.Unmarshal([]byte(j), &data); err != nil {
log.Fatal(err)
}
for _, d := range data {
var displayInfo struct{ Path string }
if err := json.Unmarshal([]byte(d.DisplayInfo), &displayInfo); err != nil {
log.Fatal(err)
}
fmt.Println(d.ID, displayInfo.Path)
}
playground