Reading the data which is in a map[string]interface{} - json

Source server return the data in a Json format of multiple object
interfaces, how can we parse such data ?
I am using a variable of JSON map[string]interface{} type to hold the result from server
The data return from Server.
"data": [
{
"group": "PAA_TEST",
"id": "2018-04-10T09:24:18.000000Z",
"name": "PAA_STATION",
"released": true,
"version": 33
},
{
"group": "PAA_TEST",
"id": "2018-03-19T10:50:21.000000Z",
"name": "PAA_STATION",
"released": false,
"version": 32
}
my fmt.print output outputdata["data"] //where output data is of JSON
map[string]interface{}
[
map[group:PAA_TEST id:2018-04-10T09:24:18.000000Z name:PAA_STATION
released:true version:33]
map[group:PAA_TEST id:2018-03-19T10:50:21.000000Z name:PAA_STATION
released:false version:32]
]
How can we iterate with multiple Map interfaces? For example, if I just want to process the information with released status as true. I am trying various method for indexing but no luck yet.

The best solution is to decode the JSON directly to a Go type that matches the structure of the data. This avoids the type assertions required to dig through a map[string]interface{}.
I'll assume that the common function looks something like this:
func request(path string, ... more arguments) (map[string]interface{}}, error) {
...
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
...
var result map[string]interface{}
err := json.NewDecoder(resp.Body).Decode(&result)
return result, err
}
Change the function to take a pointer to the result as an argument:
func request(pv interface{}, path string, ... more arguments) error {
...
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
...
err := json.NewDecoder(resp.Body).Decode(pv)
return err
}
Call this modified function like this:
var result struct {
Data []struct{
Group, ID, Name string
Released bool
Version int
}
}
err := request(&result, "some/path", ... more arguments )
if err != nil {
// handle error
}
for _, d := range result.Data {
if !d.Released {
continue
}
fmt.Println(d.Group, d.ID, d.Name, d.Version)
... process d
}

Related

Go reading map from json stream

I need to parse really long json file (more than million items). I don't want to load it to the memory and read it chunk by chunk. There's a good example with the array of items here. The problem is that I deal with the map. And when I call Decode I get not at beginning of value.
I can't get what should be changed.
const data = `{
"object1": {"name": "cattle","location": "kitchen"},
"object2": {"name": "table","location": "office"}
}`
type ReadObject struct {
Name string `json:"name"`
Location string `json:"location"`
}
func ParseJSON() {
dec := json.NewDecoder(strings.NewReader(data))
tkn, err := dec.Token()
if err != nil {
log.Fatalf("failed to read opening token: %v", err)
}
fmt.Printf("opening token: %v\n", tkn)
objects := make(map[string]*ReadObject)
for dec.More() {
var nextSymbol string
if err := dec.Decode(&nextSymbol); err != nil {
log.Fatalf("failed to parse next symbol: %v", err)
}
nextObject := &ReadObject{}
if err := dec.Decode(&nextObject); err != nil {
log.Fatalf("failed to parse next object")
}
objects[nextSymbol] = nextObject
}
tkn, err = dec.Token()
if err != nil {
log.Fatalf("failed to read closing token: %v", err)
}
fmt.Printf("closing token: %v\n", tkn)
fmt.Printf("OBJECTS: \n%v\n", objects)
}
TL,DR: when you are calling Token() method for a first time, you move offset from the beginning (of a JSON value) and therefore you get the error.
You are working with this struct (link):
type Decoder struct {
// others fields omits for simplicity
tokenState int
}
Pay attention for a tokenState field. This value could be one of (link):
const (
tokenTopValue = iota
tokenArrayStart
tokenArrayValue
tokenArrayComma
tokenObjectStart
tokenObjectKey
tokenObjectColon
tokenObjectValue
tokenObjectComma
)
Let's back to your code. You are calling Token() method. This method obtains first JSON-valid token { and changes tokenState from tokenObjectValue to the tokenObjectStart (link). Now you are "in-an-object" state.
If you try to call Decode() at this point you will get an error (not at beginning of value). This is because allowed states of tokenState for calling Decode() are tokenTopValue, tokenArrayStart, tokenArrayValue, tokenObjectValue, i.e. "full" value, not part of it (link).
To avoid this you can just don't call Token() at all and do something like this:
dec := json.NewDecoder(strings.NewReader(dataMapFromJson))
objects := make(map[string]*ReadObject)
if err := dec.Decode(&objects); err != nil {
log.Fatalf("failed to parse next symbol: %v", err)
}
fmt.Printf("OBJECTS: \n%v\n", objects)
Or, if you want to read chunk-by-chunk, you could keep calling Token() until you reach "full" value. And then call Decode() on this value (I guess this should work).
After consuming the initial { with your first call to dec.Token(), you must :
use dec.Token() to extract the next key
after extracting the key, you can call dec.Decode(&nextObject) to decode an entry
example code :
for dec.More() {
key, err := dec.Token()
if err != nil {
// handle error
}
var val interface{}
err = dec.Decode(&val)
if err != nil {
// handle error
}
fmt.Printf(" %s : %v\n", key, val)
}
https://play.golang.org/p/5r1d8MsNlKb

Parsing Json in variable in golang

how do I parse this json
[
[
[
"Odio los lunes",
"i hate mondays",
null,
null,
1
]
],
null,
"en"
]
to show just Odio los lunes?
Implement unmarshalar to fetch the value required from nested array and parse it into and struct using unmarshal like:
package main
import (
"fmt"
"encoding/json"
)
func(item *Result) UnmarshalJSON(data []byte)error{
var v []interface{}
if err:= json.Unmarshal(data, &v);err!=nil{
fmt.Println(err)
return err
}
item.Data = v[0].(interface{}).([]interface{})[0].([]interface{})[0].(string)
return nil
}
type Result struct {
Data string
}
func main() {
var result Result
jsonString := []byte(`[[["Odio los lunes", "i hate mondays", null, null, 1]], null, "en"]`)
if err := json.Unmarshal(jsonString, &result); err != nil{
fmt.Println(err)
}
//fmt.Printf("%+v\n",result)
value := result.Data
fmt.Println(value)
}
Playground example
A simple approach would be to parse the JSON document as an interface{} ("any type") and grab the target string value by asserting the parsed structure. For example (on the go playground):
func main() {
str := GetTargetString([]byte(jsonstr))
fmt.Println(str) // => "Odio los lunes"
}
func GetTargetString(bs []byte) string {
var doc interface{}
if err := json.Unmarshal(bs, &doc); err != nil {
panic(err)
}
return doc.([]interface{})[0].([]interface{})[0].([]interface{})[0].(string)
}
The "GetTargetString" function will panic if the given byte slice does not contain a valid JSON document or if the structure of the document isn't adequately similar (that is, an array whose first element is an array, whose first element is an array, whose first element is a string).
A more idiomatic (and generally safer) approach would be to inspect the types using the special two-return-value form of the type assertion and return a tuple of (string, error), e.g.:
if err := json.Unmarshal(jsonString, &result); err != nil {
return "", err
}
array0, ok := doc.([]interface{})
if !ok {
return "", fmt.Errorf("JSON document is not an array")
}
array1, ok := array0[0].([]interface{})
if !ok {
return "", fmt.Errorf("first element is not an array")
}
// ...
It is much simpler (and faster) with fastjson:
var p fastjson.Parser
v, err := p.Parse(input)
if err != nil {
log.Fatal(err)
}
fmt.Printf("v[0][0][0]: %s", v.GetStringBytes("0", "0", "0"))

How to decode complicated unnamed JSON in Golang

I am currently trying to decode the following JSON structure:
[
{
"2015-08-14 19:29:48-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1a",
"UTC_time": "2015-08-14 23:29:48",
"error": "0"
}
},
{
"2015-08-14 19:37:07-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:37:07",
"error": "0"
}
},
{
"2015-08-14 19:44:16-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:44:16",
"error": "0"
}
}
]
This is to eventually have a slice of Reading structs, formatted as the following:
type reading struct {
Value string `json:"value"`
MTID string `json:"measurement_tag_id"`
UTCTime string `json:"UTC_time"`
Error string `json:"error"`
}
I would then like to add this into an existing structure nested as:
type site struct {
Name string
ID string
Tags []tag
}
type tag struct {
ID string
Readings []reading
}
I've currently been able to create the base structure for sites and tags from a more typical JSON payload with appropriate keys. I have been unsuccessful though in figuring out how to decode the reading JSON. So far the closest I have gotten is via map[string]interface{} chaining, but this feels incredibly clunky and verbose.
Solution so far for reference:
var readingData []interface{}
if err := json.Unmarshal(file, &readingData); err != nil {
panic(err)
}
readings := readingData[0].(map[string]interface{})
firstReading := readings["2015-08-14 19:29:48-04:00"].(map[string]interface{})
fmt.Println(firstReading)
value := firstReading["value"].(string)
error := firstReading["error"].(string)
MTID := firstReading["measurement_tag_id"].(string)
UTCTime := firstReading["UTC_time"].(string)
fmt.Println(value, error, MTID, UTCTime)
While I am not sure if its necessary yet, I would also like to hold on to the arbitrary date keys. My first thought was to create a function that returned a map[string]reading but I am not sure how feasible this is.
Thanks for the help in advance!
You can have your reading type implement the json.Unmarshaler interface.
func (r *reading) UnmarshalJSON(data []byte) error {
type _r reading // same structure, but no methods, avoids infinite calls to this method
m := map[string]_r{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
for _, v := range m {
*r = reading(v)
}
return nil
}
https://play.golang.org/p/7X1oB77XL4
Another way is to use a slice of maps to parse, then copy to slice of readings, eg:
var readingMaps []map[string]reading //slice of maps of string key to reading value
if err := json.Unmarshal([]byte(data), &readingMaps); err != nil {
panic(err)
}
readings := []reading{}
for _, m := range readingMaps {
for _, r := range m {
readings = append(readings, r)
}
}
play.golang.org/p/jXTdmaZz7s

JSON Unmarshal Irregular JSON field

I have this code:
type Response struct {
ID string `json:"id"`
Tags Tags `json:"tags,omitempty"`
}
type Tags struct {
Geo []string `json:"geo,omitempty"`
Keyword []string `json:"keyword,omitempty"`
Storm []string `json:"storm,omitempty"`
}
func (t *Tags) UnmarshalJSON(b []byte) (err error) {
str := string(b)
if str == "" {
t = &Tags{}
return nil
}
err = json.Unmarshal(b, t)
if err != nil {
return err
}
return nil
}
Now, my JSON response looks like this:
[{
"id": "/cms/v4/assets/en_US",
"doc": [{
"id": "af02b41d-c2c5-48ec-9dbc-ceed693bdbac",
"tags": {
"geo": [
"DMA:US.740:US"
]
}
},
{
"id": "6a90d9ed-7978-4c18-8e36-c01cf4260492",
"tags": ""
},
{
"id": "32cfd045-98ac-408c-b464-c74e02466339",
"tags": {
"storm": [
"HARVEY - AL092017"
],
"keyword": [
"hurrcane",
"wunderground"
]
}
}
]
}]
Preferably, I'd change the JSON response to be done correctly, but I cannot. Unmarshaling continues to error out (goroutine stack exceeds 1000000000-byte limit). Preferably, I'd rather do this using easyjson or ffjson but doubt it is possible. Suggestions?
Your UnmarshalJSON function calls itself recursively, which will cause the stack to explode in size.
func (t *Tags) UnmarshalJSON(b []byte) (err error) {
str := string(b)
if str == "" {
t = &Tags{}
return nil
}
err = json.Unmarshal(b, t) <--- here it calls itself again
if err != nil {
return err
}
return nil
}
If you have a reason to call json.Unmarshal from within a UnmarshalJSON function, it must be on a different type. A common way to do this is to use a local alias:
type tagsAlias Tags
var ta = &tagsAlias
err = json.Unmarshal(b, ta)
if err != nil {
return err
}
*t = Tags(ta)
Also note that t = &Tags{} does nothing in your function; it assigns a new value to t, but that value is lost as soon as the function exits. If you really want to assign to t, you need *t; but you also don't need that at all, unless you're trying to unsset a previously set instance of *Tags.

How to save JSON response in dynamodb in GO

I want to save a JSON response in aws-dynamodb, I am using aws-dynamodb-sdk. What I'm currently doing is:
func (e *DB) saveToDynamodb(data map[string]interface{}){
params := &dynamodb.PutItemInput{
Item: map[string]*dynamodb.AttributeValue{
"Key": {
M: data,
},
},
TableName: aws.String("Asset_Data"),
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}
But as you can see data is of map[string]interface{} type while the expected type is map[string]*AttributeValue that's why giving compilation error.
Is there any workaround to save a json response?
The best way to put json into DynamoDB is to use the helper functions.
func ExampleMarshal() (map[string]*dynamodb.AttributeValue, error) {
type Record struct {
Bytes []byte
MyField string
Letters []string
Numbers []int
}
r := Record{
Bytes: []byte{48, 49},
MyField: "MyFieldValue",
Letters: []string{"a", "b", "c", "d"},
Numbers: []int{1, 2, 3},
}
av, err := dynamodbattribute.Marshal(r)
return map[string]*dynamodb.AttributeValue{"object":av}, err
}
You should use type assertion in this case.
Give it a try to this:
func (e *DB) saveToDynamodb(data map[string]interface{}){
params := &dynamodb.PutItemInput{
Item: map[string]*dynamodb.AttributeValue{
"Key": {
M: data.(map[string]*dynamodb.AttributeValue), // assert interface to *dynamodb.AttributeValue
},
},
TableName: aws.String("Asset_Data"),
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}
To put JSON in aws-dynamodb we first need to iterate through each attribute of JSON struct and convert it to dynamodb.AttributeValue in the following manner:
func (e *DB) saveToDynamodb(data map[string]interface{}){
var vv=make(map[string]*dynamodb.AttributeValue)
for k,v:=range data{
x:=(v.(string)) //assert string type
xx:=&(x)
vv[k]=&dynamodb.AttributeValue{S: xx,}
}
//s:=data["asset_id"].(string)
params := &dynamodb.PutItemInput{
Item: vv,
TableName: aws.String("Asset_Data"), // Required
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
// Print the error, cast err to awserr.Error to get the Code and
// Message from an error.
fmt.Println(err.Error())
return
}
// Pretty-print the response data.
fmt.Println(resp)
}