JSON Unmarshal Irregular JSON field - json

I have this code:
type Response struct {
ID string `json:"id"`
Tags Tags `json:"tags,omitempty"`
}
type Tags struct {
Geo []string `json:"geo,omitempty"`
Keyword []string `json:"keyword,omitempty"`
Storm []string `json:"storm,omitempty"`
}
func (t *Tags) UnmarshalJSON(b []byte) (err error) {
str := string(b)
if str == "" {
t = &Tags{}
return nil
}
err = json.Unmarshal(b, t)
if err != nil {
return err
}
return nil
}
Now, my JSON response looks like this:
[{
"id": "/cms/v4/assets/en_US",
"doc": [{
"id": "af02b41d-c2c5-48ec-9dbc-ceed693bdbac",
"tags": {
"geo": [
"DMA:US.740:US"
]
}
},
{
"id": "6a90d9ed-7978-4c18-8e36-c01cf4260492",
"tags": ""
},
{
"id": "32cfd045-98ac-408c-b464-c74e02466339",
"tags": {
"storm": [
"HARVEY - AL092017"
],
"keyword": [
"hurrcane",
"wunderground"
]
}
}
]
}]
Preferably, I'd change the JSON response to be done correctly, but I cannot. Unmarshaling continues to error out (goroutine stack exceeds 1000000000-byte limit). Preferably, I'd rather do this using easyjson or ffjson but doubt it is possible. Suggestions?

Your UnmarshalJSON function calls itself recursively, which will cause the stack to explode in size.
func (t *Tags) UnmarshalJSON(b []byte) (err error) {
str := string(b)
if str == "" {
t = &Tags{}
return nil
}
err = json.Unmarshal(b, t) <--- here it calls itself again
if err != nil {
return err
}
return nil
}
If you have a reason to call json.Unmarshal from within a UnmarshalJSON function, it must be on a different type. A common way to do this is to use a local alias:
type tagsAlias Tags
var ta = &tagsAlias
err = json.Unmarshal(b, ta)
if err != nil {
return err
}
*t = Tags(ta)
Also note that t = &Tags{} does nothing in your function; it assigns a new value to t, but that value is lost as soon as the function exits. If you really want to assign to t, you need *t; but you also don't need that at all, unless you're trying to unsset a previously set instance of *Tags.

Related

How to parse json without a defined stucture?

I want to parse this JSON :
{
"error": null,
"id": "tutu",
"result": {
"param1": 559,
"param2": "yo",
"param3": {"tab":["a", "b"], "param4":"hello"},
}
}
The problem is that I want a flexible solution if the JSON structure changes: I want to be able to access each field with a key system (like in Javascript) without knowing in advance the JSON structure:
fmt.Println(jsonObj["result"]["param3"]["tab"][1])
Is it possible to do it ?
First, in your case unmarshall directly into map[string]interface{} to save an assertion.
var jsonObj map[string]interface{}
err := json.Unmarshal(b, &jsonObj)
Second, remember to check the assertion
result, ok := jsonObj["result"].(map[string]interface{})
if !ok {
panic("not json obj")
}
param3, ok := result["param3"].(map[string]interface{})
if !ok {
panic("not json obj")
}
tab, ok := param3["tab"].([]interface{})
if !ok {
panic("not json arr")
}
Lastly, you can declare a new type and "hide" the assertions in its methods
type AnyObj map[string]interface{}
func (obj AnyObj) MustObject(name string) AnyObj {
v, ok := obj[name].(map[string]interface{})
if !ok {
panic("not json obj")
}
return AnyObj(v)
}
func (obj AnyObj) MustArray(name string) []interface{} {
v, ok := obj[name].([]interface{})
if !ok {
panic("not json arr")
}
return v
}
Then use like this:
func main() {
var jsonObj AnyObj
err := json.Unmarshal(b, &jsonObj)
if (err != nil) {
panic(err);
}
tab := jsonObj.MustObject("result").MustObject("param3").MustArray("tab")
fmt.Println(tab[1])
}
check it here https://play.golang.org/p/ucdMZ0VEKcr
Yes, it is possible, although the fact that you need to assert the types makes it a bit unwieldy:
package main
import (
"encoding/json"
"fmt"
)
func main() {
b := []byte(`
{
"error": null,
"id": "tutu",
"result": {
"param1": 559,
"param2": "yo",
"param3": {"tab":["a", "b"], "param4": "hello"}
}
}
`)
var jsonObj interface{}
err := json.Unmarshal(b, &jsonObj)
if err != nil {
panic(err)
}
msg := jsonObj.(map[string]interface{})
result := msg["result"].(map[string]interface{})
param3 := result["param3"].(map[string]interface{})
tab := param3["tab"].([]interface{})
fmt.Println(tab[1])
}
This prints
b
like you would expect.
Note that this program will panic: not just if the JSON fails to parse, but also if the JSON does not exactly match the program's expectations: missing keys, different types and so on.
The Generic JSON with interface section of the JSON and Go article on The Go Blog has an example of checking the actual type of the thing at runtime; as was suggested in the comments, it's easy to forget.
this is mine used fastjson:
package main
import (
"fmt"
"github.com/valyala/fastjson"
)
type ParseValue struct {
*fastjson.Value
}
func Parse(b []byte) (*ParseValue, error) {
var p fastjson.Parser
v, err := p.ParseBytes(b)
if err != nil {
return nil, err
}
return &ParseValue{v}, nil
}
func (p *ParseValue) GetString(keys ...string) string {
return string(p.GetStringBytes(keys...))
}
func main() {
b := []byte(`
{
"error": null,
"id": "tutu",
"result": {
"param1": 559,
"param2": "yo",
"param3": {"tab":["a", "b"], "param4": "hello"}
}
}
`)
v, err := Parse(b)
if err != nil {
panic(err)
}
r := v.GetString("result", "param3", "tab", "1")
fmt.Println(r)
rr := v.GetUint("result", "param1")
fmt.Println(rr)
// output:
// b
// 559
}

Reading the data which is in a map[string]interface{}

Source server return the data in a Json format of multiple object
interfaces, how can we parse such data ?
I am using a variable of JSON map[string]interface{} type to hold the result from server
The data return from Server.
"data": [
{
"group": "PAA_TEST",
"id": "2018-04-10T09:24:18.000000Z",
"name": "PAA_STATION",
"released": true,
"version": 33
},
{
"group": "PAA_TEST",
"id": "2018-03-19T10:50:21.000000Z",
"name": "PAA_STATION",
"released": false,
"version": 32
}
my fmt.print output outputdata["data"] //where output data is of JSON
map[string]interface{}
[
map[group:PAA_TEST id:2018-04-10T09:24:18.000000Z name:PAA_STATION
released:true version:33]
map[group:PAA_TEST id:2018-03-19T10:50:21.000000Z name:PAA_STATION
released:false version:32]
]
How can we iterate with multiple Map interfaces? For example, if I just want to process the information with released status as true. I am trying various method for indexing but no luck yet.
The best solution is to decode the JSON directly to a Go type that matches the structure of the data. This avoids the type assertions required to dig through a map[string]interface{}.
I'll assume that the common function looks something like this:
func request(path string, ... more arguments) (map[string]interface{}}, error) {
...
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
...
var result map[string]interface{}
err := json.NewDecoder(resp.Body).Decode(&result)
return result, err
}
Change the function to take a pointer to the result as an argument:
func request(pv interface{}, path string, ... more arguments) error {
...
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
...
err := json.NewDecoder(resp.Body).Decode(pv)
return err
}
Call this modified function like this:
var result struct {
Data []struct{
Group, ID, Name string
Released bool
Version int
}
}
err := request(&result, "some/path", ... more arguments )
if err != nil {
// handle error
}
for _, d := range result.Data {
if !d.Released {
continue
}
fmt.Println(d.Group, d.ID, d.Name, d.Version)
... process d
}

How to list unknown fields after json parsing

Imagine we have following Go structs:
type Config struct {
Name string `json:"name,omitempty"`
Params []Param `json:"params,omitempty"`
}
type Param struct {
Name string `json:"name,omitempty"`
Value string `json:"value,omitempty"`
}
and following json:
{
"name": "parabolic",
"subdir": "pb",
"params": [{
"name": "input",
"value": "in.csv"
}, {
"name": "output",
"value": "out.csv",
"tune": "fine"
}]
}
and we do unmarshalling:
cfg := Config{}
if err := json.Unmarshal([]byte(cfgString), &cfg); err != nil {
log.Fatalf("Error unmarshalling json: %v", err)
}
fmt.Println(cfg)
https://play.golang.org/p/HZgo0jxbQrp
Output would be {parabolic [{input in.csv} {output out.csv}]} which makes sense - unknown fields were ignored.
Question: how to find out which fields were ignored?
I.e. getIgnoredFields(cfg, cfgString) would return ["subdir", "params[1].tune"]
(There is a DisallowUnknownFields option but it's different: this option would result Unmarshal in error while question is how to still parse json without errors and find out which fields were ignored)
Not sure if that is the best way but what I did is:
If current-level type is map:
Check that all map keys are known.
It could be either if keys are struct field names or map keys.
If not known - add to list of unknown fields
Repeat recursively for value that corresponds to each key
If current level type is array:
Run recursively for each element
Code:
// ValidateUnknownFields checks that provided json
// matches provided struct. If that is not the case
// list of unknown fields is returned.
func ValidateUnknownFields(jsn []byte, strct interface{}) ([]string, error) {
var obj interface{}
err := json.Unmarshal(jsn, &obj)
if err != nil {
return nil, fmt.Errorf("error while unmarshaling json: %v", err)
}
return checkUnknownFields("", obj, reflect.ValueOf(strct)), nil
}
func checkUnknownFields(keyPref string, jsn interface{}, strct reflect.Value) []string {
var uf []string
switch concreteVal := jsn.(type) {
case map[string]interface{}:
// Iterate over map and check every value
for field, val := range concreteVal {
fullKey := fmt.Sprintf("%s.%s", keyPref, field)
subStrct := getSubStruct(field, strct)
if !subStrct.IsValid() {
uf = append(uf, fullKey[1:])
} else {
subUf := checkUnknownFields(fullKey, val, subStrct)
uf = append(uf, subUf...)
}
}
case []interface{}:
for i, val := range concreteVal {
fullKey := fmt.Sprintf("%s[%v]", keyPref, i)
subStrct := strct.Index(i)
uf = append(uf, checkUnknownFields(fullKey, val, subStrct)...)
}
}
return uf
}
Full version: https://github.com/yb172/json-unknown/blob/master/validator.go

How to decode complicated unnamed JSON in Golang

I am currently trying to decode the following JSON structure:
[
{
"2015-08-14 19:29:48-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1a",
"UTC_time": "2015-08-14 23:29:48",
"error": "0"
}
},
{
"2015-08-14 19:37:07-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:37:07",
"error": "0"
}
},
{
"2015-08-14 19:44:16-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:44:16",
"error": "0"
}
}
]
This is to eventually have a slice of Reading structs, formatted as the following:
type reading struct {
Value string `json:"value"`
MTID string `json:"measurement_tag_id"`
UTCTime string `json:"UTC_time"`
Error string `json:"error"`
}
I would then like to add this into an existing structure nested as:
type site struct {
Name string
ID string
Tags []tag
}
type tag struct {
ID string
Readings []reading
}
I've currently been able to create the base structure for sites and tags from a more typical JSON payload with appropriate keys. I have been unsuccessful though in figuring out how to decode the reading JSON. So far the closest I have gotten is via map[string]interface{} chaining, but this feels incredibly clunky and verbose.
Solution so far for reference:
var readingData []interface{}
if err := json.Unmarshal(file, &readingData); err != nil {
panic(err)
}
readings := readingData[0].(map[string]interface{})
firstReading := readings["2015-08-14 19:29:48-04:00"].(map[string]interface{})
fmt.Println(firstReading)
value := firstReading["value"].(string)
error := firstReading["error"].(string)
MTID := firstReading["measurement_tag_id"].(string)
UTCTime := firstReading["UTC_time"].(string)
fmt.Println(value, error, MTID, UTCTime)
While I am not sure if its necessary yet, I would also like to hold on to the arbitrary date keys. My first thought was to create a function that returned a map[string]reading but I am not sure how feasible this is.
Thanks for the help in advance!
You can have your reading type implement the json.Unmarshaler interface.
func (r *reading) UnmarshalJSON(data []byte) error {
type _r reading // same structure, but no methods, avoids infinite calls to this method
m := map[string]_r{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
for _, v := range m {
*r = reading(v)
}
return nil
}
https://play.golang.org/p/7X1oB77XL4
Another way is to use a slice of maps to parse, then copy to slice of readings, eg:
var readingMaps []map[string]reading //slice of maps of string key to reading value
if err := json.Unmarshal([]byte(data), &readingMaps); err != nil {
panic(err)
}
readings := []reading{}
for _, m := range readingMaps {
for _, r := range m {
readings = append(readings, r)
}
}
play.golang.org/p/jXTdmaZz7s

How to save JSON response in dynamodb in GO

I want to save a JSON response in aws-dynamodb, I am using aws-dynamodb-sdk. What I'm currently doing is:
func (e *DB) saveToDynamodb(data map[string]interface{}){
params := &dynamodb.PutItemInput{
Item: map[string]*dynamodb.AttributeValue{
"Key": {
M: data,
},
},
TableName: aws.String("Asset_Data"),
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}
But as you can see data is of map[string]interface{} type while the expected type is map[string]*AttributeValue that's why giving compilation error.
Is there any workaround to save a json response?
The best way to put json into DynamoDB is to use the helper functions.
func ExampleMarshal() (map[string]*dynamodb.AttributeValue, error) {
type Record struct {
Bytes []byte
MyField string
Letters []string
Numbers []int
}
r := Record{
Bytes: []byte{48, 49},
MyField: "MyFieldValue",
Letters: []string{"a", "b", "c", "d"},
Numbers: []int{1, 2, 3},
}
av, err := dynamodbattribute.Marshal(r)
return map[string]*dynamodb.AttributeValue{"object":av}, err
}
You should use type assertion in this case.
Give it a try to this:
func (e *DB) saveToDynamodb(data map[string]interface{}){
params := &dynamodb.PutItemInput{
Item: map[string]*dynamodb.AttributeValue{
"Key": {
M: data.(map[string]*dynamodb.AttributeValue), // assert interface to *dynamodb.AttributeValue
},
},
TableName: aws.String("Asset_Data"),
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}
To put JSON in aws-dynamodb we first need to iterate through each attribute of JSON struct and convert it to dynamodb.AttributeValue in the following manner:
func (e *DB) saveToDynamodb(data map[string]interface{}){
var vv=make(map[string]*dynamodb.AttributeValue)
for k,v:=range data{
x:=(v.(string)) //assert string type
xx:=&(x)
vv[k]=&dynamodb.AttributeValue{S: xx,}
}
//s:=data["asset_id"].(string)
params := &dynamodb.PutItemInput{
Item: vv,
TableName: aws.String("Asset_Data"), // Required
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
// Print the error, cast err to awserr.Error to get the Code and
// Message from an error.
fmt.Println(err.Error())
return
}
// Pretty-print the response data.
fmt.Println(resp)
}