Unmarshal JSON that could be a string or an object - json

I have a third party service that is returning JSON where one field contains a collection of data. Here is an example of the structure it returns.
{
"title": "Afghanistan",
"slug": "afghanistan",
"fields": {
"fieldOne": "",
"fieldTwo": {
"new1": {
"type": "contentBlock",,
"fields": {
"richTextBlick": "<p>This is the travel advice.<\/p>"
}
}
},
"fieldThree": {
"new1": {
"type": "introBlock",
"fields": {
"text": "This is a title"
"richText": "<p>This is the current travel summary for Afganistan.<\/p>"
}
},
"new2": {
"type": "contentBlock",
"fields": {
"richText": "<p>It has a second block of content!<\/p>"
}
}
},
"fieldfour": "country"
}
}
Each of the "field" entries could be either a string or another object. I'd like to decode these into something like the structs below.
type EntryVersion struct {
Slug string `json:"slug"`
Fields map[string][]EntryVersionBlock `json:"fields"`
}
type EntryVersionBlock struct {
Type string `json:"type"`
Fields map[string]string `json:"fields"`
}
If the field value is simply a string, I would wrap it in an EntryVersionBlock with the type of "Text" and a single entry in the Fields map.
Any ideas how can I do this in an efficient manner? I may have to do it several hundred times in an extreme edge-case.
Thanks

Your structure is a little bit off from the json's. Fields in EntryVersion is an object not an array so making it a slice is not what you want. I would recommend you change it to this:
type EntryVersion struct {
Slug string `json:"slug"`
Fields map[string]EntryVersionBlock `json:"fields"`
}
The EntryVersionBlock does not have a "type" field in the corresponding json, it has entries with field names "new1", "new2", etc. So I would recommend you add a 3rd type Entry into which you unmarshal those fields.
type Entry struct {
Type string `json:"type"`
Fields map[string]string `json:"fields"`
}
And you would update your EntryVersionBlock to look something like this:
type EntryVersionBlock struct {
Value string `json:"-"`
Fields map[string]Entry `json:"fields"`
}
And to deal with your original problem you can have the EntryVersionBlock type implement the json.Unmarshaler interface, check the first byte in the data passed to the UnmarshalJSON method and if it's a double quote it's a string and if it's a curly opening brace it's an object. Something like this:
func (evb *EntryVersionBlock) UnmarshalJSON(data []byte) error {
switch data[0] {
case '"':
if err := json.Unmarshal(data, &evb.Value); err != nil {
return err
}
case '{':
evb.Fields = make(map[string]Entry)
if err := json.Unmarshal(data, &evb.Fields); err != nil {
return err
}
}
return nil
}
Playground: https://play.golang.org/p/IsTXI5202m

You can use GJson library to unmarshal your JSON to map, then iterate on this map and do the conversion that you need using your structs and based on fields type (map or string).
Update: Example using this method with similar case
http://blog.serverbooter.com/post/parsing-nested-json-in-go/

Related

How to convert string to JSON in Go?

Let’s say I have this string:
`
{
"testCode": 0,
"replyTest": "OK",
"data": {
"001": {
"fields": {
"name": "arben",
"fav_color": "blue",
"address": "PH",
}
},
"002": {
"fields": {
"name": "john",
"fav_color": "black",
"address": "PH",
}
},
}
}
`
How to convert this string to JSON where data is in form of list in order for me to loop this list in a process?
Whenever you have json property names that aren't known upfront, or they don't lend themselves very well to be represented as fields of a struct then you're left more or less with one option, a map. So first you need to unmarshal the "data" property into a map.
However maps in Go are implemented as unordered groups of elements indexed by a set of unique keys. So basically there's no way to ensure the order of a map in Go, and therefore you'll have to transfer the data from the map into something that can be ordered, like a slice for example.
After you've got your data in the slice you can use the standard sort package to sort the slice in the order you want.
You can start by declaring the types you need:
type DataItem struct {
Key string `json:"-"`
Fields DataFields `json:"fields"`
}
type DataFields struct {
Name string `json:"name"`
FavColor string `json:"fav_color"`
Address string `json:"address"`
}
Then unmarshal the json
var obj struct {
TestCode int `json:"testCode"`
ReplyTest string `json:"replyTest"`
Data map[string]*DataItem `json:"data"`
}
if err := json.Unmarshal(data, &obj); err != nil {
panic(err)
}
Transfer the contents of the map into a slice
items := []*DataItem{}
for key, item := range obj.Data {
item.Key = key // keep track of the key because it will be used to order the contents of the slice
items = append(items, item)
}
Finally sort the slice
sort.Slice(items, func(i, j int) bool {
return items[i].Key < items[j].Key
})
https://play.golang.com/p/D2u46veOQwD

How to handle missing fields in a JSON response dynamically in Go

I'm working on a Go wrapper for an API and I noticed that two of the JSON fields stay empty when they don't have any data.
Basically the API returns a set of information on a given url, and if it was visited at least once, everything is okay and I get a full json that I then Unmarshal into a struct:
{
"stats":{
"status":1,
"date":"09.07.2019",
"title":"Test",
"devices":{
"dev":[
{
"tag":"Desktop"
}
],
"sys":[
{
"tag":"GNU/Linux "
},
{
"tag":"Windows 10"
}
],
"bro":[
{
"tag":"Firefox 67.0"
},
{
"tag":"Chrome 62.0"
}
]
},
"refs":[
{
"link":"www.google.com"
}
]
}
}
This is the struct I'm using:
type Stats struct {
Stats struct {
Status int `json:"status"`
Date string `json:"date"`
Title string `json:"title"`
Devices struct {
Dev []struct {
Tag string `json:"tag"`
} `json:"dev"`
Sys []struct {
Tag string `json:"tag"`
} `json:"sys"`
Bro []struct {
Tag string `json:"tag"`
} `json:"bro"`
} `json:"devices"`
Refs []struct {
Link string `json:"link"`
} `json:"refs"`
} `json:"stats"`
}
When a new url is given, then things become a little bit weird:
{
"stats": {
"status": 1,
"date": "09.07.2019",
"title": "Test2",
"devices": [
],
"refs": [
]
}
}
As you can see, the fields "dev", "sys" and "bro" just disappear because they're not used and when I try to Unmarshal the JSON into the same struct I get json: cannot unmarshal array into Go struct field Stats.device of type [...]
I tried to use two different structs to handle both the responses but I'm sure that there's a way to handle them gracefully with just one.
Any help would be appreciated, thanks!
I finally managed to make it work with an ugly workaround.
I changed my struct to
type Stats struct {
Status int `json:"status"`
Date string `json:"date"`
Title string `json:"title"`
Devices interface{} `json:"devices"`
Refs interface{} `json:"refs"`
}
Then I can finally Unmarshal the JSON in both cases, but I get a map[string]interface{} when an object is passed and an empty interface{} when an empty array is passed. In order to fix this inconsistency, I simply check for the data type and force the use of a JSON intermediate conversion in order to unpack the map[string]interface{} value inside a custom Devices struct:
// Devices contains devices information
type Devices struct {
Dev []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"dev"`
Sys []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"sys"`
Bro []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"bro"`
}
The algorithms I use are the following:
//ForceDevicesToRightType uses a json conversion as intermediary for filling the Stats.Devices
// struct with map[string]interface{} values
func ForceDevicesToRightType(dev interface{}) (Devices, error) {
temp, err := json.Marshal(dev)
if err != nil {
return Devices{}, err
}
// Use a temporary variable of the right type
var devices Devices
err = json.Unmarshal(temp, &devices)
if err != nil {
return Devices{}, err
}
return devices, nil
}
// ForceRefsToRightType uses a json conversion as intermediary for filling the Stats.Refs
// struct with map[string]interface{} values
func ForceRefsToRightType(refs interface{}) (Refs, error) {
temp, err := json.Marshal(refs)
if err != nil {
return Refs{}, err
}
// Use a temporary variable of the right type
var references Refs
err = json.Unmarshal(temp, &references)
if err != nil {
return Refs{}, err
}
return references, nil
}
Since the compiler knows that both Devices and Refs fields are interface{} I cannot simply access any methods after the conversion, so I simply make a cast of the right type and everything works fine.
For example, if I wanted to access the Dev sub-struct, this is the proper way:
y, _ := GetStats()
fmt.Println(y.Devices.(Devices).Dev)
It's ugly, but it works.
Thank you very much for your help, I hope that this method will save you an headache!

How to show empty object instead of empty struct or nil in Go json marshal

I need to show json's empty object {} when do json.Marshal() for a struct pointer. I can only output either null value or empty struct value.
If the person key is filled with &Person{} or new(Person), it will show empty struct like below:
{
"data": {
"person": {
"name": "",
"age": 0
},
"created_date": "2009-11-10T23:00:00Z"
}
}
And if we don't initialize it at all, it will show null.
{
"data": {
"person": null,
"created_date": "2009-11-10T23:00:00Z"
}
}
I want to show "person": {}. Is it possible?
Go Playground for the complete code: https://play.golang.org/p/tT15G2ESPVc
Option A, use the omitempty tag option on all of the Person's fields and make sure the response's field is allocated before marshaling.
type Person struct {
Name string `json:"name,omitempty"`
Age int `json:"age,omitempty"`
}
// ...
resp.Person = new(Person)
https://play.golang.org/p/o3jWdru_8bC
Option B, use a non-pointer wrapper type that embeds the Person pointer type.
type PersonJSON struct {
*Person
}
type Response struct {
Person PersonJSON `json:"person"`
CreatedDate time.Time `json:"created_date"`
}
https://play.golang.org/p/EKQc7uf1_Vk
Option C, have the Reponse type implement the json.Marshaler interface.
func (r *Response) MarshalJSON() ([]byte, error) {
type tmp Response
resp := (*tmp)(r)
var data struct {
Wrapper struct {
*Person
} `json:"person"`
*tmp
}
data.Wrapper.Person = resp.Person
data.tmp = resp
return json.Marshal(data)
}
https://play.golang.org/p/1qkSCWZ225j
There may be other options...
In Go, an empty struct by definition assigns zero values to field elements. Eg: for int 0, "" for string, etc.
For your case, simply comparing to null would work out. Or, you could define an emptyPerson as:
var BAD_AGE = -1
emptyPerson := &Person{"", BAD_AGE} // BAD_AGE indicates no person
if person[age] == BAD_AGE {
// handle case for emptyPerson}

Go - JSON Decoder not initializing my struct

I am trying to decode some JSON retrieved via http.Get. However, when I check the structs I initialize with fmt.Println, they are always empty.
I suspect it is because my struct's structure does not agree with the returned JSON, but I am not sure how to fix it. In general, I am not exactly quite sure how the decoder works.
This is the JSON:
{
"response":[
{
"list": {
"category":"(noun)",
"synonyms":"histrion|player|thespian|role player|performer|performing artist"
}
},
{
"list": {
"category":"(noun)",
"synonyms":"doer|worker|person|individual|someone|somebody|mortal|soul"
}
}
]
}
Here is what i have tried so far:
type SynonymResponse struct {
lists []SynonymList
}
type SynonymList struct {
category string
synonyms string
}
var synonyms SynonymResponse;
dec := json.NewDecoder(response.Body)
err := dec.Decode(&synonyms)
if err != nil {
log.Fatal(err)
}
fmt.Println(synonyms)
EDIT: Per #Leo's answer and #JimB's hint, there are two issues with my attempt. Below is the proper set of structs, though as Leo pointed out, this will be empty:
type SynonymResponses struct {
resp []SynonymResponse
}
type SynonymResponse struct {
listo SynonymList
}
type SynonymList struct {
cat string
syns string
}
In order for your JSON to be picked up by the decoder, the fields in your struct must be exported.
This means you need you capitalize the field names. If you have custom naming on your fields -> json conversion, you can add json tags to your structs.
This will fix your issue:
type SynonymResponse struct {
Lists []SynonymList `json:"response"`
}
type SynonymList struct {
Category string `json:"category"`
Synonyms string `json:"synonyms"`
}

How can I parse JSON in GoLang if nested content uses dynamic keys?

I received JSON below from a client API but i am struggling to get nested JSON content. How can I parse it if the inner keys are dynamic?
const jsonStream = `
{
"items": {
"bvu62fu6dq": {
"name": "john",
"age": 23,
"xyz": "weu33s"
},
"iaxdw23fq": {
"name": "kelly",
"age": 21,
"xyz": "weu33s"
}
}
}`
This is what i have tried below by looping the map to extract the value of name and age from above JSON string; but it returns map with nil value as a result.
goplaygound
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
type Item struct {
Contact struct {
Info map[string]Person
} `json:"items"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
var item Item
if err := dec.Decode(&item); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", item.Contact.Info["bvu62fu6dq"].Name)
}
}
Try this instead, looks like you just have your structure set up incorrectly:
http://play.golang.org/p/VRKbv-GVQB
You need to parse the entire json string, which is an object that contains a single element named items. items then contains a map of string -> Person objects.
If you only want to extract name and age from each person, you do it by grabbing data.Items["bvu62fu6dq"].Name.
If you want dynamic keys inside the Person, you'll need to do map[string]interface{} instead of Person in order to capture the dynamic keys again. It would look something like:
type Data struct {
Items map[string]map[string]interface{} `json:"items"`
}
...
fmt.Printf("%v\n", data.Items["bvu62fu6dq"]["name"]
fmt.Printf("%v\n", data.Items["bvu62fu6dq"]["age"]
fmt.Printf("%v\n", data.Items["bvu62fu6dq"]["xyz"]