Unmarshalling json to structure using json.RawMessage - json

I need to unmarshal json object which may have the following formats:
Format1:
{
"contactType": 2,
"value": "0123456789"
}
Format2:
{
"contactType": "MobileNumber",
"value": "0123456789"
}
The structure I'm using for unmarshalling is:-
type Contact struct {
ContactType int `json:"contactType"`
Value string `json:"value"`
}
But this works only for format 1. I don't want to change the datatype of ContactType but I want to accommodate the 2nd format as well. I heard about json.RawMarshal and tried using it.
type Contact struct {
ContactType int
Value string `json:"value"`
Type json.RawMessage `json:"contactType"`
}
type StringContact struct {
Type string `json:"contactType"`
}
type IntContact struct {
Type int `json:"contactType"`
}
This gets the unmarshalling done, but I'm unable to set the ContactType variable which depends on the type of json.RawMessage. How do I model my structure so that this problem gets solved?

You will need to do the unmarshalling yourself. There is a very good article that shows how to use the json.RawMessage right and a number of other solutions to this very problem, Like using interfaces, RawMessage, implemention your own unmarshal and decode functions etc.
You will find the article here: JSON decoding in GO by Attila Oláh
Note: Attila has made a few errors on his code examples.
I taken the liberty to put together (using some of the code from Attila) a working example using RawMessage to delay the unmarshaling so we can do it on our own version of the Decode func.
Link to GOLANG Playground
package main
import (
"fmt"
"encoding/json"
"io"
)
type Record struct {
AuthorRaw json.RawMessage `json:"author"`
Title string `json:"title"`
URL string `json:"url"`
Author Author
}
type Author struct {
ID uint64 `json:"id"`
Email string `json:"email"`
}
func Decode(r io.Reader) (x *Record, err error) {
x = new(Record)
if err = json.NewDecoder(r).Decode(x); err != nil {
return
}
if err = json.Unmarshal(x.AuthorRaw, &x.Author); err == nil {
return
}
var s string
if err = json.Unmarshal(x.AuthorRaw, &s); err == nil {
x.Author.Email = s
return
}
var n uint64
if err = json.Unmarshal(x.AuthorRaw, &n); err == nil {
x.Author.ID = n
}
return
}
func main() {
byt_1 := []byte(`{"author": 2,"title": "some things","url": "https://stackoverflow.com"}`)
byt_2 := []byte(`{"author": "Mad Scientist","title": "some things","url": "https://stackoverflow.com"}`)
var dat Record
if err := json.Unmarshal(byt_1, &dat); err != nil {
panic(err)
}
fmt.Printf("%#s\r\n", dat)
if err := json.Unmarshal(byt_2, &dat); err != nil {
panic(err)
}
fmt.Printf("%#s\r\n", dat)
}
Hope this helps.

Related

Unmarshalling of JSON with dynamic keys

I have a scenario where the JSON that has dynamic set of fields that need to get unmarshalled in to a struct.
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
type Status struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status map[string]Status `json:"status"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
var person Person
if err := dec.Decode(&person); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(person)
fmt.Println(person.Status["bvu62fu6dq"])
}
}
The output:
{john 23 map[]}
{false}
When it gets unmarshalled, the nested status struct is not being correctly resolved to the value in the JSON (shows false even with true value in JSON), is there any issue in the code?
Your types don't really match with the JSON you have:
type Status struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status map[string]Status `json:"status"`
}
Maps to JSON that looks something like this:
{
"name": "foo",
"age": 12,
"status": {
"some-string": {
"Status": true
}
}
}
The easiest way to unmarshal data with a mix of known/unknown fields in a go type is to have something like this:
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"` // skip this key
}
Then, first unmarshal the known data:
var p Person
if err := json.Unmarshal([]byte(jsonStream), &p); err != nil {
panic(err)
}
// then unmarshal the rest of the data
if err := json.Unmarshal([]byte(jsonStream), &p.Random); err != nil {
panic(err)
}
Now the Random map will contain every and all data, including the name and age fields. Seeing as you've got those tagged on the struct, these keys are known, so you can easily delete them from the map:
delete(p.Random, "name")
delete(p.Random, "age")
Now p.Random will contain all the unknown keys and their respective values. These values apparently will be an object with a field status, which is expected to be a boolean. You can set about using type assertions and convert them all over to a more sensible type, or you can take a shortcut and marshal/unmarshal the values. Update your Person type like so:
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"`
Statuses map[string]Status `json:"-"`
}
Now take the clean Random value, marshal it and unmarshal it back into the Statuses field:
b, err := json.Marshal(p.Random)
if err != nil {
panic(err)
}
if err := json.Unmarshal(b, &p.Statuses); err != nil {
panic(err)
}
// remove Random map
p.Random = nil
The result is Person.Statuses["bvu62fu6dq"].Status is set to true
Demo
Cleaning this all up, and marshalling the data back
Now because our Random and Statuses fields are tagged to be ignored for JSON marshalling (json:"-"), marshalling this Person type won't play nice when you want to output the original JSON from these types. It's best to wrap this logic up in a custom JSON (un)-Marshaller interface. You can either use some intermediary types in your MarshalJSON and UnmarshalJSON methods on the Person type, or just create a map and set the keys you need:
func (p Person) MarshalJSON() ([]byte, error) {
data := make(map[string]interface{}, len(p.Statuses) + 2) // 2 being the extra fields
// copy status fields
for k, v := range p.Statuses {
data[k] = v
}
// add known keys
data["name"] = p.Name
data["age"] = p.Age
return json.Marshal(data) // return the marshalled map
}
Similarly, you can do the same thing for UnmarshalJSON, but you'll need to create a version of the Person type that doesn't have the custom handling:
type intermediaryPerson struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"`
}
// no need for the tags and helper fields anymore
type Person struct {
Name string
Age int
Statuses map[string]Status // Status type doesn't change
}
func (p *Person) UnmarshalJSON(data []byte) error {
i := intermediaryPerson{}
if err := json.Unmarshal(data, &i); err != nil {
return err
}
if err := json.Unmarshal(data, &i.Random); err != nil {
return err
}
delete(i.Random, "name")
delete(i.Random, "age")
stat, err := json.Marshal(i.Random)
if err != nil {
return err
}
// copy known fields
p.Name = i.Name
p.Age = i.Age
return json.Unmarshal(stat, &p.Statuses) // set status fields
}
In cases like this, it's common to create a type that handles the known fields and embed that, though:
type BasePerson struct {
Name string `json:"name"`
Age int `json:"age"`
}
and embed that in both the intermediary and the "main"/exported type:
type interPerson struct {
BasePerson
Random map[string]interface{} `json:"-"`
}
type Person struct {
BasePerson
Statuses map[string]Status
}
That way, you can just unmarshal the known fields directly into the BasePerson type, assign it, and then handle the map:
func (p *Person) UnmarshalJSON(data []byte) error {
base := BasePerson{}
if err := json.Unmarshal(data, &base); err != nil {
return err
}
p.BasePerson = base // takes care of all known fields
unknown := map[string]interface{}{}
if err := json.Unmarshal(data, unknown); err != nil {
return err
}
// handle status stuff same as before
delete(unknown, "name") // remove known fields
// marshal unknown key map, then unmarshal into p.Statuses
}
Demo 2
This is how I'd go about it. It allows for calls to json.Marshal and json.Unmarshal to look just like any other type, it centralises the handling of unknown fields in a single place (the implementation of the marshaller/unmarshaller interface), and leaves you with a single Person type where every field contains the required data, in a usable format. It's a tad inefficient in that it relies on unmarshalling/marshalling/unmarshalling the unknown keys. You could do away with that, like I said, using type assertions and iterating over the unknown map instead, faffing around with something like this:
for k, v := range unknown {
m, ok := v.(map[string]interface{})
if !ok {
continue // not {"status": bool}
}
s, ok := m["status"]
if !ok {
continue // status key did not exist, ignore
}
if sb, ok := s.(bool); ok {
// ok, we have a status bool value
p.Statuses[k] = Status{
Status: sb,
}
}
}
But truth be told, the performance difference won't be that great (it's micro optimisation IMO), and the code is a tad too verbose to my liking. Be lazy, optimise when needed, not whenever
Type doesn't meet with your json value.
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
For above json your code should look like below snnipet to work (some modifications in your existing code).
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"strings"
)
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
type bvu62fu6dq struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status bvu62fu6dq `json:"bvu62fu6dq"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
var person Person
if err := dec.Decode(&person); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(person)
fmt.Println(person.Status)
}
}
Based on your json data you have to map with type fields.
Run code snippet

Unmarshalling Dynamic JSON Data With Overlapping Fields in Golang

Sorry If i'm posting a question that has already been answered, but I can't seem to find any similar situations on here. I have a websocket client that receives dynamic json data with overlapping fields. The fact that the fields overlap has has made Unmarshalling very difficult for me.
I have structs for the data types I receive, but I need a way to check the json data before I unmarshal it to a specific struct. I was hoping that an interface could act as a temporary holder and I would then be able to match the interface to the specific struct I want to unmarshal to, but that doesn't seem possible, or I just don't know how to go about it. Here are a few examples of the data types I'm receiving and structs to go along with it in case that helps.
response 1: {"connectionID":17973829270596587247,"event":"systemStatus","status":"online","version":"1.9.0"}
response 2: {"channelID":328,"channelName":"ohlc-5","event":"subscriptionStatus","pair":"XBT/USD","status":"subscribed","subscription":{"interval":5,"name":"ohlc"}}
response 3: [328,["1649576721.042916","1649577000.000000","42641.50000","42641.50000","42641.50000","42641.50000","42641.50000","0.00335101",2],"ohlc-5","XBT/USD"]
response 4: {"event":"heartbeat"}
structs below
import (
"time"
"encoding/json"
)
type ConnStatus struct {
ConnectionID uint64 `json:"connectionID"`
Event string `json:"event"`
Status string `json:"status"`
Version string `json:"version"`
}
type HeartBeat struct {
Event string `json:"event"`
}
type OHLCsuccess struct {
ChannelID int `json:"channelID"`
ChannelName string `json:"channelName"`
Event string `json:"event"`
Pair string `json:"pair"`
Status string `json:"status"`
Subscription OHLC `json:"subscription"`
}
type OHLC struct {
Interval int `json:"interval"`
Name string `json:"name"`
}
type OHLCUpdates struct {
ChannelID int
OHLCArray OHLCNewTrade
ChannelName string
Pair string
}
type OHLCNewTrade struct {
StartTime UnixTime
EndTime UnixTime
Open float64
High float64
Low float64
Close float64
VWAP float64
Volume float64
Count int
}
type UnixTime struct {
time.Time
}
func (u *UnixTime) UnmarshalJSON(d []byte) error {
var ts int64
err := json.Unmarshal(d, &ts)
if err != nil {
return err
}
u.Time = time.Unix(ts, 0).UTC()
return nil
}
Any idea(s) on how to go about this? Thanks in advance for the help!
Are you in control of the different responses? If so, wow about adding a "type" field to the top level?
See "How to put everything at the top level" section on https://eagain.net/articles/go-dynamic-json/ for more info.
E.g. (untested):
func UnmarshalJSON(d []byte) error {
var jsonValue map[string]interface{}
err := json.Unmarshal(d, &jsonValue)
if err != nil {
return err
}
switch jsonValue["type"] {
case 1:
// unmarshal into struct type 1
case 2:
// unmarshal into struct type 2
default:
// throw err
}
// or if you don't have access to type:
if jsonValue["connectionID"] != nil {
// unmarshal into struct type 1
}
return nil
}
Alternatively you could try to (strictly) unmarshal into each struct, until you don't get an error, e.g. something like:
func DetermineStruct(d []byte) int {
var connStatus *ConnStatus
reader := bytes.NewReader(d)
decoder := json.NewDecoder(reader)
decoder.DisallowUnknownFields()
err := decoder.Decode(connStatus)
if err == nil {
panic(err)
}
err = json.Unmarshal(d, &connStatus)
if err == nil {
return 1
}
var ohlcSuccess OHLCsuccess
err = json.Unmarshal(d, &ohlcSuccess)
if err == nil {
return 2
}
}

Is it possible to have a structure for dynamic keys along with static keys for json in Golang

My apologies for the basic question. I am new to Golang and I have the json to parse as below
{
"config1":{
"Parameters":{
"Pm1":"value",
"Pm2":"value",
"Pm3":"value"
},
"dynamic_key1":{
"Parameters":{
"a":"value",
"b":"value",
"c":"value",
"d":"value"
},
"Epoch":"value"
},
"Epoch":"value"
}
}
I am trying to write a struct to parse this json and wrote the struct in the following way.
type Parameters struct {
Pm1 string `json:"Pm1"`
Pm2 string `json:"Pm2"`
Pm3 string `json:"Pm3"`
}
type dynamicParametes struct {
a string `json:"a"`
b string `json:"b"`
c string `json:"c"`
d string `json:"d"`
}
type dynamic struct {
Parameters dynamicParametes `json:"Parameters"`
Epoch string `json:"Epoch"`
}
type config1 struct {
Parameters Parameters `json:"Parameters"`
Dynamic_keys map[string]dynamic `json:"-"`
Epoch string `json:"Epoch"`
}
type config struct {
config1 config1 `json:"config1"`
}
I was hoping that the map will match all the matching keys with dynamic structs and show them in the map. But, I see it created an empty map in the response.
Implemented custom unmarshler for config type.
Note
If you don't need Parameters and dynamicParametes as struct types, you can simply unmarshal them into map[string]string
you have to expose all fields in your structs to do json unmarshaling
validate your json string
type config struct {
Config1 config1 `json:"config1"`
}
type _config config
func (b *config) UnmarshalJSON(data []byte) error {
var v = struct {
Config1 map[string]interface{} `json:"config1"`
}{}
if err := json.Unmarshal(data, &v); err != nil {
return err
}
c := _config{}
err := json.Unmarshal(data, &c)
if err != nil {
return err
}
b.Config1.Parameters = c.Config1.Parameters
b.Config1.Epoch = c.Config1.Epoch
if b.Config1.Dynamic_keys == nil {
b.Config1.Dynamic_keys = map[string]dynamic{}
}
for key, config := range v.Config1 {
if key == `Parameters` || key == `Epoch` {
continue
}
data, err := json.Marshal(config)
if err != nil {
return err
}
d := dynamic{}
err = json.Unmarshal(data, &d)
if err != nil {
return err
}
b.Config1.Dynamic_keys[key] = d
}
return nil
}
you can see full code here
All you need is understand how base data types looks in json.
Field Parameters in your json is simple map[string]string and you can unmarshall it with standart json.Unmasrhall without any aditional implementation of interface json.Unmarshaler.
Link for Go Playground with code below
package main
import (
"encoding/json"
"fmt"
)
const jsonStr = `{
"config1":{
"Parameters":{
"Pm1":"value_1",
"Pm2":"value_2",
"Pm3":"value_3"
},
"dynamic_key1":{
"Parameters":{
"a":"value_1",
"b":"value_2",
"c":"value_3",
"d":"value_4"
},
"Epoch":"value"
},
"Epoch":"value"
}
}`
type Data struct {
Config1 struct {
Parameters map[string]string `json:"Parameters"`
Dynamic struct {
Parameters map[string]string `json:"Parameters"`
Epoch string `json:"Epoch"`
} `json:"dynamic_key1"`
Epoch string `json:"Epoch"`
} `json:"config1"`
}
func main() {
var data Data
_ = json.Unmarshal([]byte(jsonStr), &data)
fmt.Printf("%+v\n", data)
}
Output:
{Config1:{Parameters:map[Pm1:value_1 Pm2:value_2 Pm3:value_3] Dynamic:{Parameters:map[a:value_1 b:value_2 c:value_3 d:value_4] Epoch:value} Epoch:value}}

Map JSON array value to struct specific variable

Let's say I have this JSON structure:
{
"name":"repo",
"tags":["1.0","2.0","3.0"]
}
And I would like to map it to this Go struct:
type Repository struct {
Name string `json:"name"`
Tags []struct {
Tag string `json:"??"`
Sha256 string
}
}
How can I link the "tags" array JSON value to a struct field?
EDIT: The idea will be to access the tags array value like this
repository.Tags[0].Tag.
Implement json.Unmarshaler on a Tag type:
package main
import (
"encoding/json"
"log"
)
type Repository struct {
Name string
Tags []Tag
}
type Tag struct {
Tag string
Sha256 string
}
func (t *Tag) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err != nil {
return err
}
t.Tag = s
return nil
}
func main() {
b := []byte(`{ "name":"repo", "tags":["1.0","2.0","3.0"] }`)
var r Repository
err := json.Unmarshal(b, &r)
if err != nil {
log.Fatal(err)
}
log.Printf("%+v\n", r)
}
Try it on the playground: https://play.golang.org/p/ExwWhis0w0V
Marshaling back to JSON is left as an exercise for the reader.

go "encoding/json" : marshal json field

I have a PostgreSQL schema with json field's (DisplayInfo, and FormatInfo). Structure of this field's is dynamic.
I'can read and render it only as string (string type in render struct) :
[
{
"ID":9,
"Name":"120 №1",
"DisplayInfo":"{\"path\": \"http://path/to/img.png\"}",
"Format":{
"Code":"frame-120",
"Width":120,
"Height":60,
"FormatInfo":"[{\"name\": \"\\u0413\\u043b\\u0430\\u0432\\u043d\\u043e\\u0435 \\u0438\\u0437\\u043e\\u0431\\u0440\\u0430\\u0436\\u0435\\u043d\\u0438\\u0435\", \"field_type\": \"img\", \"key\": \"path\"}]"
},
"Weight":0.075,
"Application":8,
"Url":"//path/to/game",
"Referrer":""
}
]
but i want output field DisplayInfo as JSON object. How ?
My render code:
func renderJSON(w http.ResponseWriter, obj models.Model) {
js, err := json.Marshal(obj)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Write(js)
}
UPD 1 : Structure of this field's is dynamic. DisplayInfo may have 'path' field, or may not. They may have additional fields.
UPD 2. I wana output DisplayInfo and FormatInfo as json-object(not string), as part of whole object, like this:
[
{
"ID":9,
"Name":"120 №1",
"DisplayInfo":{"path": "http://path/to/img.png"},
"Format":{
"Code":"frame-120",
"Width":120,
"Height":60,
"FormatInfo":[{"name": "\\u0413\\u043b\\u0430\\u0432\\u043d\\u043e\\u0435 \\u0438\\u0437\\u043e\\u0431\\u0440\\u0430\\u0436\\u0435\\u043d\\u0438\\u0435", "field_type": "img", "key": "path"}]
},
"Weight":0.075,
"Application":8,
"Url":"//path/to/game",
"Referrer":""
}
]
UPD 3: Structures
Actual structure is :
type BannerSerializer struct {
ID int
Name string
DisplayInfo string
Format formatSerializer
Weight float32
Application int
Url string
Referrer string
}
Then i trying this structure:
type BannerSerializer struct {
ID int
Name string
DisplayInfo json.RawMessage
Format formatSerializer
Weight float32
Application int
Url string
Referrer string
}
DisplayInfo serialize as base64 string (or like base64, don't know)
Use a pointer to json.RawMessage:
type Data struct {
Obj *json.RawMessage
}
Playground: http://play.golang.org/p/Qq9IUBDLzJ.
Assuming you have access to change models.Model, you can create your own type with a custom Unmarshaler that just returns the raw string:
type JSONString string
func (s JSONString) MarshalJSON() ([]byte, error) {
return []byte(s), nil
}
Working example:
package main
import (
"encoding/json"
"fmt"
)
type JSONString string
func (s JSONString) MarshalJSON() ([]byte, error) {
return []byte(s), nil
}
type Model struct {
ID int
Name string
DisplayInfo JSONString
}
func main() {
data := []byte(`{
"ID":9,
"Name":"120 №1",
"DisplayInfo":"{\"path\": \"http://path/to/img.png\"}"
}`)
var obj Model
err := json.Unmarshal(data, &obj)
if err != nil {
panic(err)
}
// Here comes your code
js, err := json.Marshal(obj)
if err != nil {
panic(err)
}
fmt.Println(string(js))
}
Output:
{"ID":9,"Name":"120 №1","DisplayInfo":{"path":"http://path/to/img.png"}}
Playground: http://play.golang.org/p/6bcnuGjlU8
You'd have to unmarshal it, here's an example:
var data []*struct {
ID int
DisplayInfo string
}
if err := json.Unmarshal([]byte(j), &data); err != nil {
log.Fatal(err)
}
for _, d := range data {
var displayInfo struct{ Path string }
if err := json.Unmarshal([]byte(d.DisplayInfo), &displayInfo); err != nil {
log.Fatal(err)
}
fmt.Println(d.ID, displayInfo.Path)
}
playground