I have a database table called http_requests. I have modelled following struct to represent rows in this table.
type Map map[string]interface{}
type HTTPRequest struct {
ID int64 `json:"id" gorm:"id"`
RequestURL string `json:"request_url,omitempty" gorm:"request_url"`
RequestParams *RequestParams `json:"request_params,omitempty" gorm:"request_params"`
}
// RequestParams is another struct that holds params from body and URL query
type RequestParams struct {
FromBody Map `json:"body,omitempty"`
FromQuery Map `json:"query,omitempty"`
}
Code to save HTTPRequest:
request := &HTTPRequest{
RequestURL: "dummy/url",
RequestParams: &RequestParams{FromBody: Map{"param1": "value1"}},
}
if err := gorm.DB.Create(request).Error; err != nil {
return err
}
When I try to save this HTTPRequest it results in error:
sql: Scan error on column index 9, name "request_params": unsupported Scan, storing driver.Value type []uint8 into type *RequestParams
I would like to have request_params column to store JSON like this:
{"body":{"param1":"value1"}, "query": {"param2" : "value2"} }
or
{"body":{"param1":"value1"}}
or
{"query": {"param2" : "value2"} }
And this should get parsed into RequestParams struct when reading from database.
As suggested by #mkopriva, I implemented Scan() and Value() methods for my RequestParams type. See code below.
import (
"database/sql/driver"
"encoding/json"
"strings"
)
// Value converts RequestParams to a map
func (reqParams RequestParams) Value() (driver.Value, error) {
reqMap, err := reqParams.ToMap()
if err != nil {
return nil, err
}
return reqMap.ForceJSON(), nil
}
// Scan converts value to RequestParams
func (reqParams *RequestParams) Scan(value interface{}) error {
// set empty struct by default
*reqParams = RequestParams{}
if value == nil {
return nil
}
if s, ok := value.([]byte); ok {
d := json.NewDecoder(strings.NewReader(string(s)))
d.UseNumber()
rp := &RequestParams{}
if err := d.Decode(rp); err == nil {
*reqParams = *rp
}
}
return nil
}
Related
My apologies for the basic question. I am new to Golang and I have the json to parse as below
{
"config1":{
"Parameters":{
"Pm1":"value",
"Pm2":"value",
"Pm3":"value"
},
"dynamic_key1":{
"Parameters":{
"a":"value",
"b":"value",
"c":"value",
"d":"value"
},
"Epoch":"value"
},
"Epoch":"value"
}
}
I am trying to write a struct to parse this json and wrote the struct in the following way.
type Parameters struct {
Pm1 string `json:"Pm1"`
Pm2 string `json:"Pm2"`
Pm3 string `json:"Pm3"`
}
type dynamicParametes struct {
a string `json:"a"`
b string `json:"b"`
c string `json:"c"`
d string `json:"d"`
}
type dynamic struct {
Parameters dynamicParametes `json:"Parameters"`
Epoch string `json:"Epoch"`
}
type config1 struct {
Parameters Parameters `json:"Parameters"`
Dynamic_keys map[string]dynamic `json:"-"`
Epoch string `json:"Epoch"`
}
type config struct {
config1 config1 `json:"config1"`
}
I was hoping that the map will match all the matching keys with dynamic structs and show them in the map. But, I see it created an empty map in the response.
Implemented custom unmarshler for config type.
Note
If you don't need Parameters and dynamicParametes as struct types, you can simply unmarshal them into map[string]string
you have to expose all fields in your structs to do json unmarshaling
validate your json string
type config struct {
Config1 config1 `json:"config1"`
}
type _config config
func (b *config) UnmarshalJSON(data []byte) error {
var v = struct {
Config1 map[string]interface{} `json:"config1"`
}{}
if err := json.Unmarshal(data, &v); err != nil {
return err
}
c := _config{}
err := json.Unmarshal(data, &c)
if err != nil {
return err
}
b.Config1.Parameters = c.Config1.Parameters
b.Config1.Epoch = c.Config1.Epoch
if b.Config1.Dynamic_keys == nil {
b.Config1.Dynamic_keys = map[string]dynamic{}
}
for key, config := range v.Config1 {
if key == `Parameters` || key == `Epoch` {
continue
}
data, err := json.Marshal(config)
if err != nil {
return err
}
d := dynamic{}
err = json.Unmarshal(data, &d)
if err != nil {
return err
}
b.Config1.Dynamic_keys[key] = d
}
return nil
}
you can see full code here
All you need is understand how base data types looks in json.
Field Parameters in your json is simple map[string]string and you can unmarshall it with standart json.Unmasrhall without any aditional implementation of interface json.Unmarshaler.
Link for Go Playground with code below
package main
import (
"encoding/json"
"fmt"
)
const jsonStr = `{
"config1":{
"Parameters":{
"Pm1":"value_1",
"Pm2":"value_2",
"Pm3":"value_3"
},
"dynamic_key1":{
"Parameters":{
"a":"value_1",
"b":"value_2",
"c":"value_3",
"d":"value_4"
},
"Epoch":"value"
},
"Epoch":"value"
}
}`
type Data struct {
Config1 struct {
Parameters map[string]string `json:"Parameters"`
Dynamic struct {
Parameters map[string]string `json:"Parameters"`
Epoch string `json:"Epoch"`
} `json:"dynamic_key1"`
Epoch string `json:"Epoch"`
} `json:"config1"`
}
func main() {
var data Data
_ = json.Unmarshal([]byte(jsonStr), &data)
fmt.Printf("%+v\n", data)
}
Output:
{Config1:{Parameters:map[Pm1:value_1 Pm2:value_2 Pm3:value_3] Dynamic:{Parameters:map[a:value_1 b:value_2 c:value_3 d:value_4] Epoch:value} Epoch:value}}
Is there a way to unmarshall JSON arrays into single objects in Go?
I have a json response from an endpoint:
{
"results": [
{
"key": "value"
}
]
}
I have a Go struct for the object inside the array:
type Object struct {
Key string `json:"key"`
}
...and a struct for the response object:
type Response struct {
Objects []Object `json:"results"`
}
results is an array of objects, but I know that the endpoint will only ever return an array with 1 object. Is there a way to unmarshall the data and avoid having reference the object by an index? I was hoping I could use something like json:"results[0]" as a field tag.
I'd prefer to be able to:
decoder.Decode(&response)
response.Object.Key
Rather than
decoder.Decode(&response)
response.Objects[0].Key
To does this you need to customize unmarshalling.
An way is create a ResponseCustom like:
//Response json (customized) that match with Unmarshaler interface
type ResponseCustom struct {
Object
}
func (r *ResponseCustom) UnmarshalJSON(b []byte) error{
rsp := &Response{}
err := json.Unmarshal(b, rsp)
if err != nil {
log.Fatalln("error:", err)
} else {
r.Object = rsp.Objects[0]
}
//
return err
}
So you can use ResponseCustom instead of you Response for get Object value.
Look:
func main() {
//
data := []byte(jsondata)
resp := &ResponseCustom{}
//
json.Unmarshal(data, resp)
//
fmt.Println("My Object.value is: " + resp.Object.Key)
}
The result is:
My Object.value is: value
In playground: https://play.golang.org/p/zo7wOSacA4w
Implement unmarshaler interface to convert array of object to object. Fetch the value for key and then unmarshal your json as
package main
import(
"fmt"
"encoding/json"
)
type Object struct {
Key string `json:"key"`
}
func (obj *Object) UnmarshalJSON(data []byte) error {
var v map[string]interface{}
if err := json.Unmarshal(data, &v); err != nil {
fmt.Printf("Error whilde decoding %v\n", err)
return err
}
obj.Key = v["results"].(interface{}).([]interface{})[0].(interface{}).(map[string]interface{})["key"].(interface{}).(string)
return nil
}
func main(){
var obj Object
data := []byte(`{
"results": [
{
"key": "value"
}
]
}`)
if err := json.Unmarshal(data, &obj); err!=nil{
fmt.Println(err)
}
fmt.Printf("%#v", obj)
}
Playground
I'm using an API that formats its responses in this way:
{
"err": 0,
"data": **Other json structure**
}
The way I'm getting a response right now is I'm putting the response in an struct like this:
type Response struct {
Err int `json:"err"`
Data interface{} `json:"data"`
}
and then I'm doing this after getting a response
jsonbytes, _ := json.Marshal(resp.Data)
json.Unmarshal(jsonBytes, &dataStruct)
I'm only ignoring errors for this example.
It seems kinda weird to me that I'm marshaling and unmarshaling when I know what the data is supposed to look like and what type it's supposed to be.
Is there a more simple solution that I'm not seeing or is this a normal thing to do?
Edit: I should probably mention that the Data attribute in the response object can vary depending on what API call I'm doing.
The JSON unmarshaller uses reflection to look at the type it is unmarshalling to. Given an uninitialised interface{} as the destination for the unmarshalled data, a JSON object gets unmarshalled into a map[string]interface{} (example in playground).
Here are some ideas.
Option A
If you know the datatype, you can define a new response struct for each type. Example:
type FooResponse struct {
Err int `json:"err"`
Data Foo `json:"data"`
}
type Foo struct {
FooField string `json:"foofield"`
}
type BarResponse struct {
Err int `json:"err"`
Data Bar `json:"data"`
}
type Bar struct {
BarField string `json:"barfield"`
}
Option B
If you prefer to have a single Response struct instead of one per type, you can tell the JSON unmarshaller to avoid unmarshalling the data field until a later time by using the json.RawMessage data type:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Response struct {
Err int `json:"err"`
Data json.RawMessage `json:"data"`
}
type Foo struct {
FooField string `json:"foofield"`
}
type Bar struct {
BarField string `json:"barfield"`
}
func main() {
fooRespJSON := []byte(`{"data":{"foofield":"foo value"}}`)
barRespJSON := []byte(`{"data":{"barfield":"bar value"}}`)
var (
resp Response
foo Foo
bar Bar
)
// Foo
if err := json.Unmarshal(fooRespJSON, &resp); err != nil {
log.Fatal(err)
}
if err := json.Unmarshal(resp.Data, &foo); err != nil {
log.Fatal(err)
}
fmt.Println("foo:", foo)
// Bar
if err := json.Unmarshal(barRespJSON, &resp); err != nil {
log.Fatal(err)
}
if err := json.Unmarshal(resp.Data, &bar); err != nil {
log.Fatal(err)
}
fmt.Println("bar:", bar)
}
Output:
foo: {foo value}
bar: {bar value}
https://play.golang.org/p/Y7D4uhaC4a8
Option C
A third option, as pointed out by #mkopriva in a comment on the question, is to use interface{} as an intermediary datatype and pre-initialise this to a known datatype.
Emphasis lies on the word intermediary -- of course passing around an interface{} is best avoided (Rob Pike's Go Proverbs). The use-case here is to allow any datatype to be used without the need for multiple different Response types. On way to avoid exposing the interface{} is to wrap the response completely, exposing only the data and the error:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Foo struct {
FooField string `json:"foofield"`
}
type Bar struct {
BarField string `json:"barfield"`
}
type Error struct {
Code int
}
func (e *Error) Error() string {
return fmt.Sprintf("error code %d", e.Code)
}
func unmarshalResponse(data []byte, v interface{}) error {
resp := struct {
Err int `json:"err"`
Data interface{} `json:"data"`
}{Data: v}
if err := json.Unmarshal(data, &resp); err != nil {
return err
}
if resp.Err != 0 {
return &Error{Code: resp.Err}
}
return nil
}
func main() {
fooRespJSON := []byte(`{"data":{"foofield":"foo value"}}`)
barRespJSON := []byte(`{"data":{"barfield":"bar value"}}`)
errRespJSON := []byte(`{"err": 123}`)
// Foo
var foo Foo
if err := unmarshalResponse(fooRespJSON, &foo); err != nil {
log.Fatal(err)
}
fmt.Println("foo:", foo)
// Bar
var bar Bar
if err := unmarshalResponse(barRespJSON, &bar); err != nil {
log.Fatal(err)
}
fmt.Println("bar:", bar)
// Error response
var v interface{}
if err := unmarshalResponse(errRespJSON, &v); err != nil {
log.Fatal(err)
}
}
Output:
foo: {foo value}
bar: {bar value}
2009/11/10 23:00:00 error code 123
https://play.golang.org/p/5SVfQGwS-Wy
Is there a way to serialize custom structs when encoding/decoding with json?
say you have 3 (in my actual code there are 10) different custom structs which are being sent over udp, and you use json for encoding:
type a struct {
Id int
Data msgInfo
}
type b struct {
Id int
Data msgInfo
Other metaInfo
}
type c struct {
Other metaInfo
}
On the recieving end you want to know if the struct recieved was of type a, b or c, so it can for example be passed to a type spesific channel.
type msgtype reflect.Type
.
.
nrOfBytes, err := udpConn.Read(recievedBytes)
if err != nil {...}
var msg interface{}
err = json.Unmarshal(recievedBytes[0:nrOfBytes], &msg)
if err != nil {...}
u := reflect.ValueOf(msg)
msgType := u.Type()
fmt.Printf("msg is of type: %s\n", msgType)
With gob this is easily done by registering the types, but i have to use json seeing as it's communication over udp, so is there anyway to serialize the custom structs? I want the print to be
msg is of type: a
but i'm only getting
msg is of type: map[string]interface {}
One thing you can do is using the json.RawMessage type and a custom Wrapper type.
Then, upon receiving a message, you can do a switch (or use a map of constructors) to get the right struct.
For example (omitting error checking):
package main
import (
"encoding/json"
"fmt"
)
type Message struct {
Type string
Data json.RawMessage
}
func (m Message) Struct() interface{} {
unmarshaller, found := unmarshallers[m.Type]
if !found {
return nil
}
return unmarshaller([]byte(m.Data))
}
type Foo struct {
ID int
}
var unmarshallers = map[string]func([]byte) interface{}{
"foo": func(raw []byte) interface{} {
var f Foo
json.Unmarshal(raw, &f)
return f
},
}
func main() {
var body = []byte(`{"Type":"foo","Data":{"ID":1}}`)
var msg Message
json.Unmarshal(body, &msg)
switch s := msg.Struct().(type) {
case Foo:
fmt.Println(s)
}
}
See this playground example for a live demo http://play.golang.org/p/7FmQqnWPaE
Base on your comment, this maybe a solution:
type Packet {
Type string
Data []byte
}
Encode function:
func packageEncode(i interface{}) ([]byte, error) {
b, err := json.Marshal(i)
if err != nil {
return nil, err
}
p := &Package{Data: b}
switch t.(type) {
case a:
p.Type = "a"
case b:
p.Type = "b"
case c:
p.Type = "c"
}
return json.Marshal(p)
}
Then, when recieved message and decode:
p := &Package{}
err = json.Unmarshal(recievedBytes[0:nrOfBytes], p)
...
if p.Type == "a" {
msg := &a{}
err = json.Unmarshal(p.Data, msg)
}
if p.Type == "b" {
...
}
I have a PostgreSQL schema with json field's (DisplayInfo, and FormatInfo). Structure of this field's is dynamic.
I'can read and render it only as string (string type in render struct) :
[
{
"ID":9,
"Name":"120 №1",
"DisplayInfo":"{\"path\": \"http://path/to/img.png\"}",
"Format":{
"Code":"frame-120",
"Width":120,
"Height":60,
"FormatInfo":"[{\"name\": \"\\u0413\\u043b\\u0430\\u0432\\u043d\\u043e\\u0435 \\u0438\\u0437\\u043e\\u0431\\u0440\\u0430\\u0436\\u0435\\u043d\\u0438\\u0435\", \"field_type\": \"img\", \"key\": \"path\"}]"
},
"Weight":0.075,
"Application":8,
"Url":"//path/to/game",
"Referrer":""
}
]
but i want output field DisplayInfo as JSON object. How ?
My render code:
func renderJSON(w http.ResponseWriter, obj models.Model) {
js, err := json.Marshal(obj)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Write(js)
}
UPD 1 : Structure of this field's is dynamic. DisplayInfo may have 'path' field, or may not. They may have additional fields.
UPD 2. I wana output DisplayInfo and FormatInfo as json-object(not string), as part of whole object, like this:
[
{
"ID":9,
"Name":"120 №1",
"DisplayInfo":{"path": "http://path/to/img.png"},
"Format":{
"Code":"frame-120",
"Width":120,
"Height":60,
"FormatInfo":[{"name": "\\u0413\\u043b\\u0430\\u0432\\u043d\\u043e\\u0435 \\u0438\\u0437\\u043e\\u0431\\u0440\\u0430\\u0436\\u0435\\u043d\\u0438\\u0435", "field_type": "img", "key": "path"}]
},
"Weight":0.075,
"Application":8,
"Url":"//path/to/game",
"Referrer":""
}
]
UPD 3: Structures
Actual structure is :
type BannerSerializer struct {
ID int
Name string
DisplayInfo string
Format formatSerializer
Weight float32
Application int
Url string
Referrer string
}
Then i trying this structure:
type BannerSerializer struct {
ID int
Name string
DisplayInfo json.RawMessage
Format formatSerializer
Weight float32
Application int
Url string
Referrer string
}
DisplayInfo serialize as base64 string (or like base64, don't know)
Use a pointer to json.RawMessage:
type Data struct {
Obj *json.RawMessage
}
Playground: http://play.golang.org/p/Qq9IUBDLzJ.
Assuming you have access to change models.Model, you can create your own type with a custom Unmarshaler that just returns the raw string:
type JSONString string
func (s JSONString) MarshalJSON() ([]byte, error) {
return []byte(s), nil
}
Working example:
package main
import (
"encoding/json"
"fmt"
)
type JSONString string
func (s JSONString) MarshalJSON() ([]byte, error) {
return []byte(s), nil
}
type Model struct {
ID int
Name string
DisplayInfo JSONString
}
func main() {
data := []byte(`{
"ID":9,
"Name":"120 №1",
"DisplayInfo":"{\"path\": \"http://path/to/img.png\"}"
}`)
var obj Model
err := json.Unmarshal(data, &obj)
if err != nil {
panic(err)
}
// Here comes your code
js, err := json.Marshal(obj)
if err != nil {
panic(err)
}
fmt.Println(string(js))
}
Output:
{"ID":9,"Name":"120 №1","DisplayInfo":{"path":"http://path/to/img.png"}}
Playground: http://play.golang.org/p/6bcnuGjlU8
You'd have to unmarshal it, here's an example:
var data []*struct {
ID int
DisplayInfo string
}
if err := json.Unmarshal([]byte(j), &data); err != nil {
log.Fatal(err)
}
for _, d := range data {
var displayInfo struct{ Path string }
if err := json.Unmarshal([]byte(d.DisplayInfo), &displayInfo); err != nil {
log.Fatal(err)
}
fmt.Println(d.ID, displayInfo.Path)
}
playground