How to save JSON response in dynamodb in GO - json

I want to save a JSON response in aws-dynamodb, I am using aws-dynamodb-sdk. What I'm currently doing is:
func (e *DB) saveToDynamodb(data map[string]interface{}){
params := &dynamodb.PutItemInput{
Item: map[string]*dynamodb.AttributeValue{
"Key": {
M: data,
},
},
TableName: aws.String("Asset_Data"),
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}
But as you can see data is of map[string]interface{} type while the expected type is map[string]*AttributeValue that's why giving compilation error.
Is there any workaround to save a json response?

The best way to put json into DynamoDB is to use the helper functions.
func ExampleMarshal() (map[string]*dynamodb.AttributeValue, error) {
type Record struct {
Bytes []byte
MyField string
Letters []string
Numbers []int
}
r := Record{
Bytes: []byte{48, 49},
MyField: "MyFieldValue",
Letters: []string{"a", "b", "c", "d"},
Numbers: []int{1, 2, 3},
}
av, err := dynamodbattribute.Marshal(r)
return map[string]*dynamodb.AttributeValue{"object":av}, err
}

You should use type assertion in this case.
Give it a try to this:
func (e *DB) saveToDynamodb(data map[string]interface{}){
params := &dynamodb.PutItemInput{
Item: map[string]*dynamodb.AttributeValue{
"Key": {
M: data.(map[string]*dynamodb.AttributeValue), // assert interface to *dynamodb.AttributeValue
},
},
TableName: aws.String("Asset_Data"),
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(resp)
}

To put JSON in aws-dynamodb we first need to iterate through each attribute of JSON struct and convert it to dynamodb.AttributeValue in the following manner:
func (e *DB) saveToDynamodb(data map[string]interface{}){
var vv=make(map[string]*dynamodb.AttributeValue)
for k,v:=range data{
x:=(v.(string)) //assert string type
xx:=&(x)
vv[k]=&dynamodb.AttributeValue{S: xx,}
}
//s:=data["asset_id"].(string)
params := &dynamodb.PutItemInput{
Item: vv,
TableName: aws.String("Asset_Data"), // Required
}
resp, err := e.dynamodb.PutItem(params)
if err != nil {
// Print the error, cast err to awserr.Error to get the Code and
// Message from an error.
fmt.Println(err.Error())
return
}
// Pretty-print the response data.
fmt.Println(resp)
}

Related

Unmarshaling JSON in golang

i'm having a lot of trouble getting my program to work. I want to unmarshal something pretty simple, but it's giving me a lot of issues, unfortunately.
Here is the response that I want to unmarshal:
{"error":[],"result":{"XXBTZUSD":[[1647365820,"39192.0","39192.0","39191.9","39191.9","39191.9","0.18008008",10],[1647365880,"39186.1","39186.1","39172.0","39176.0","39174.4","0.13120077",10]],"last":1647408900}}
I've wrote these structs to help with unmarshalling
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type Trades struct {
Pair []OHLC `json:"XXBTZUSD"`
Last float64 `json:"last"`
}
type OHLC struct {
Time float64
Open string
High string
Low string
Close string
Vwa string
Volume string
Count float64
}
I have a function call that makes the http request and then unmarshals the data. For whatever reason, my code will end before even starting the function call for the http request and subsequent unmarshalling when the Pair type is []OHLC or []*OHLC. If I change the Pair type to interface{}, then it runs. i want to make it work with the OHLC struct instead though. Below is the complete code:
package main
import (
"fmt"
"net/http"
//"strings"
"io/ioutil"
"encoding/json"
)
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type Trades struct {
Pair []OHLC `json:"XXBTZUSD"`
Last float64 `json:"last"`
}
type OHLC struct {
TT float64
Open string
High string
Low string
Close string
Vwap string
Volume string
Count float64
}
/*func main() {
var data = [...]Trade{
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
}
}*/
func main() {
fmt.Println("in main");
getOhlc()
}
func getOhlc() {
fmt.Println("in ohlc func")
resp, err := http.Get("https://api.kraken.com/0/public/OHLC?pair=XXBTZUSD");
if err != nil {
fmt.Errorf("error after request")
return;
}
defer resp.Body.Close();
body, err := ioutil.ReadAll(resp.Body);
if err != nil {
fmt.Errorf("error when reading")
return;
}
var jsonData Resp;
err = json.Unmarshal(body, &jsonData);
if err != nil {
fmt.Errorf("error when unmarshalling")
return
}
if(len(jsonData.Error) > 0) {
fmt.Errorf("error");
return;
}
fmt.Println(jsonData);
}
Any ideas about what might be happening?
"Any ideas about what might be happening?"
The elements in the "XXBTZUSD" JSON array are arrays themselves, i.e. "XXBTZUSD" is an array of arrays. The OHLC type is a struct type. The stdlib will not, by itself, unmarshal a JSON array into a Go struct. Go structs can be used to unmarshal JSON objects. JSON arrays can be unmarshaled into Go slices or arrays.
You would clearly see that that's the issue if you would just print the error from json.Unmarshal:
json: cannot unmarshal array into Go struct field
Trades.result.XXBTZUSD of type main.OHLC
https://go.dev/play/p/D4tjXZVzDI_w
If you want to unmarshal a JSON array into a Go struct you have to have the Go struct type implement a the json.Unmarshaler interface.
func (o *OHLC) UnmarshalJSON(data []byte) error {
// first unmarshal the array into a slice of raw json
raw := []json.RawMessage{}
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
// create a function that unmarshals each raw json element into a field
unmarshalFields := func(raw []json.RawMessage, fields ...interface{}) error {
if len(raw) != len(fields) {
return errors.New("bad number of elements in json array")
}
for i := range raw {
if err := json.Unmarshal([]byte(raw[i]), fields[i]); err != nil {
return err
}
}
return nil
}
// call the function
return unmarshalFields(
raw,
&o.Time,
&o.Open,
&o.High,
&o.Low,
&o.Close,
&o.Vwa,
&o.Volume,
&o.Count,
)
}
https://go.dev/play/p/fkFKLkaNaSU
Your code had some issues:
Remove semicolons from end of lines, it's redundant.
fmt.Errorf return error, and not print it, every time check your error and propagate it.
We can convert array of numbers and string to struct in golang.
for achieving your desired output we need to first convert to intermediate container and then convert to our wanted output:
package main
import (
"errors"
"fmt"
"log"
"net/http"
//"strings"
"encoding/json"
"io/ioutil"
)
type Resp struct {
Error []string `json:"error"`
Result Trades `json:"result"`
}
type IntermediateResp struct {
Error []string `json:"error"`
Result IntermediateTrades `json:"result"`
}
type IntermediateTrades struct {
Pair [][]interface{} `json:"XXBTZUSD"`
Last int `json:"last"`
}
type Trades struct {
Pair []OHLC `json:"result"`
Last int `json:"last"`
}
type OHLC struct {
TT float64
Open string
High string
Low string
Close string
Vwap string
Volume string
Count float64
}
/*func main() {
var data = [...]Trade{
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
Trade{5, "op", "hi", "lo", "cl", "vw", "vo", 2},
}
}*/
func main() {
fmt.Println("in main")
err := getOhlc()
if err != nil {
log.Fatal(err)
}
}
func buildOHLC(l []interface{}) (*OHLC, error) {
if len(l) < 8 {
return nil, errors.New("short list")
}
return &OHLC{
TT: l[0].(float64),
Open: l[1].(string),
High: l[2].(string),
Low: l[3].(string),
Close: l[4].(string),
Vwap: l[5].(string),
Volume: l[6].(string),
Count: l[7].(float64),
}, nil
}
func convert(r IntermediateResp) (*Resp, error) {
result := &Resp{Error: r.Error, Result: Trades{Pair: make([]OHLC, len(r.Result.Pair)), Last: r.Result.Last}}
for i, v := range r.Result.Pair {
ohlc, err := buildOHLC(v)
if err != nil {
return nil, err
}
result.Result.Pair[i] = *ohlc
}
return result, nil
}
func getOhlc() error {
fmt.Println("in ohlc func")
resp, err := http.Get("https://api.kraken.com/0/public/OHLC?pair=XXBTZUSD")
if err != nil {
return fmt.Errorf("error after request, %v", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
fmt.Println(string(body))
if err != nil {
return fmt.Errorf("error when reading %v", err)
}
var jsonData IntermediateResp
err = json.Unmarshal(body, &jsonData)
if err != nil {
return fmt.Errorf("error when unmarshalling %v", err)
}
if len(jsonData.Error) > 0 {
return fmt.Errorf("error")
}
convertedOhlc, err := convert(jsonData)
if err != nil {
return fmt.Errorf("error when convertedOhlc %v", err)
}
fmt.Println(convertedOhlc)
return nil
}
We define IntermediateResp and IntermediateTrades for Unmarshaling json and then convert it to actual Resp.
I think aother way is using custom Unmarshal for Trades struct.

How to list unknown fields after json parsing

Imagine we have following Go structs:
type Config struct {
Name string `json:"name,omitempty"`
Params []Param `json:"params,omitempty"`
}
type Param struct {
Name string `json:"name,omitempty"`
Value string `json:"value,omitempty"`
}
and following json:
{
"name": "parabolic",
"subdir": "pb",
"params": [{
"name": "input",
"value": "in.csv"
}, {
"name": "output",
"value": "out.csv",
"tune": "fine"
}]
}
and we do unmarshalling:
cfg := Config{}
if err := json.Unmarshal([]byte(cfgString), &cfg); err != nil {
log.Fatalf("Error unmarshalling json: %v", err)
}
fmt.Println(cfg)
https://play.golang.org/p/HZgo0jxbQrp
Output would be {parabolic [{input in.csv} {output out.csv}]} which makes sense - unknown fields were ignored.
Question: how to find out which fields were ignored?
I.e. getIgnoredFields(cfg, cfgString) would return ["subdir", "params[1].tune"]
(There is a DisallowUnknownFields option but it's different: this option would result Unmarshal in error while question is how to still parse json without errors and find out which fields were ignored)
Not sure if that is the best way but what I did is:
If current-level type is map:
Check that all map keys are known.
It could be either if keys are struct field names or map keys.
If not known - add to list of unknown fields
Repeat recursively for value that corresponds to each key
If current level type is array:
Run recursively for each element
Code:
// ValidateUnknownFields checks that provided json
// matches provided struct. If that is not the case
// list of unknown fields is returned.
func ValidateUnknownFields(jsn []byte, strct interface{}) ([]string, error) {
var obj interface{}
err := json.Unmarshal(jsn, &obj)
if err != nil {
return nil, fmt.Errorf("error while unmarshaling json: %v", err)
}
return checkUnknownFields("", obj, reflect.ValueOf(strct)), nil
}
func checkUnknownFields(keyPref string, jsn interface{}, strct reflect.Value) []string {
var uf []string
switch concreteVal := jsn.(type) {
case map[string]interface{}:
// Iterate over map and check every value
for field, val := range concreteVal {
fullKey := fmt.Sprintf("%s.%s", keyPref, field)
subStrct := getSubStruct(field, strct)
if !subStrct.IsValid() {
uf = append(uf, fullKey[1:])
} else {
subUf := checkUnknownFields(fullKey, val, subStrct)
uf = append(uf, subUf...)
}
}
case []interface{}:
for i, val := range concreteVal {
fullKey := fmt.Sprintf("%s[%v]", keyPref, i)
subStrct := strct.Index(i)
uf = append(uf, checkUnknownFields(fullKey, val, subStrct)...)
}
}
return uf
}
Full version: https://github.com/yb172/json-unknown/blob/master/validator.go

Parsing Json in variable in golang

how do I parse this json
[
[
[
"Odio los lunes",
"i hate mondays",
null,
null,
1
]
],
null,
"en"
]
to show just Odio los lunes?
Implement unmarshalar to fetch the value required from nested array and parse it into and struct using unmarshal like:
package main
import (
"fmt"
"encoding/json"
)
func(item *Result) UnmarshalJSON(data []byte)error{
var v []interface{}
if err:= json.Unmarshal(data, &v);err!=nil{
fmt.Println(err)
return err
}
item.Data = v[0].(interface{}).([]interface{})[0].([]interface{})[0].(string)
return nil
}
type Result struct {
Data string
}
func main() {
var result Result
jsonString := []byte(`[[["Odio los lunes", "i hate mondays", null, null, 1]], null, "en"]`)
if err := json.Unmarshal(jsonString, &result); err != nil{
fmt.Println(err)
}
//fmt.Printf("%+v\n",result)
value := result.Data
fmt.Println(value)
}
Playground example
A simple approach would be to parse the JSON document as an interface{} ("any type") and grab the target string value by asserting the parsed structure. For example (on the go playground):
func main() {
str := GetTargetString([]byte(jsonstr))
fmt.Println(str) // => "Odio los lunes"
}
func GetTargetString(bs []byte) string {
var doc interface{}
if err := json.Unmarshal(bs, &doc); err != nil {
panic(err)
}
return doc.([]interface{})[0].([]interface{})[0].([]interface{})[0].(string)
}
The "GetTargetString" function will panic if the given byte slice does not contain a valid JSON document or if the structure of the document isn't adequately similar (that is, an array whose first element is an array, whose first element is an array, whose first element is a string).
A more idiomatic (and generally safer) approach would be to inspect the types using the special two-return-value form of the type assertion and return a tuple of (string, error), e.g.:
if err := json.Unmarshal(jsonString, &result); err != nil {
return "", err
}
array0, ok := doc.([]interface{})
if !ok {
return "", fmt.Errorf("JSON document is not an array")
}
array1, ok := array0[0].([]interface{})
if !ok {
return "", fmt.Errorf("first element is not an array")
}
// ...
It is much simpler (and faster) with fastjson:
var p fastjson.Parser
v, err := p.Parse(input)
if err != nil {
log.Fatal(err)
}
fmt.Printf("v[0][0][0]: %s", v.GetStringBytes("0", "0", "0"))

Unmarshalling a single element in an array

Is there a way to unmarshall JSON arrays into single objects in Go?
I have a json response from an endpoint:
{
"results": [
{
"key": "value"
}
]
}
I have a Go struct for the object inside the array:
type Object struct {
Key string `json:"key"`
}
...and a struct for the response object:
type Response struct {
Objects []Object `json:"results"`
}
results is an array of objects, but I know that the endpoint will only ever return an array with 1 object. Is there a way to unmarshall the data and avoid having reference the object by an index? I was hoping I could use something like json:"results[0]" as a field tag.
I'd prefer to be able to:
decoder.Decode(&response)
response.Object.Key
Rather than
decoder.Decode(&response)
response.Objects[0].Key
To does this you need to customize unmarshalling.
An way is create a ResponseCustom like:
//Response json (customized) that match with Unmarshaler interface
type ResponseCustom struct {
Object
}
func (r *ResponseCustom) UnmarshalJSON(b []byte) error{
rsp := &Response{}
err := json.Unmarshal(b, rsp)
if err != nil {
log.Fatalln("error:", err)
} else {
r.Object = rsp.Objects[0]
}
//
return err
}
So you can use ResponseCustom instead of you Response for get Object value.
Look:
func main() {
//
data := []byte(jsondata)
resp := &ResponseCustom{}
//
json.Unmarshal(data, resp)
//
fmt.Println("My Object.value is: " + resp.Object.Key)
}
The result is:
My Object.value is: value
In playground: https://play.golang.org/p/zo7wOSacA4w
Implement unmarshaler interface to convert array of object to object. Fetch the value for key and then unmarshal your json as
package main
import(
"fmt"
"encoding/json"
)
type Object struct {
Key string `json:"key"`
}
func (obj *Object) UnmarshalJSON(data []byte) error {
var v map[string]interface{}
if err := json.Unmarshal(data, &v); err != nil {
fmt.Printf("Error whilde decoding %v\n", err)
return err
}
obj.Key = v["results"].(interface{}).([]interface{})[0].(interface{}).(map[string]interface{})["key"].(interface{}).(string)
return nil
}
func main(){
var obj Object
data := []byte(`{
"results": [
{
"key": "value"
}
]
}`)
if err := json.Unmarshal(data, &obj); err!=nil{
fmt.Println(err)
}
fmt.Printf("%#v", obj)
}
Playground

Parse JSON HTTP response using golang

I am trying to get the value of say "ip" from my following curl output:
{
"type":"example",
"data":{
"name":"abc",
"labels":{
"key":"value"
}
},
"subsets":[
{
"addresses":[
{
"ip":"192.168.103.178"
}
],
"ports":[
{
"port":80
}
]
}
]
}
I have found many examples in the internet to parse json output of curl requests and I have written the following code, but that doesn't seem to return me the value of say "ip"
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
)
type svc struct {
Ip string `json:"ip"`
}
func main() {
url := "http://myurl.com"
testClient := http.Client{
Timeout: time.Second * 2, // Maximum of 2 secs
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
res, getErr := testClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
svc1 := svc{}
jsonErr := json.Unmarshal(body, &svc1)
if jsonErr != nil {
log.Fatal(jsonErr)
}
fmt.Println(svc1.Ip)
}
I would appreciate if anyone could provide me hints on what I need to add to my code to get the value of say "ip".
You can create structs which reflect your json structure and then decode your json.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type Example struct {
Type string `json:"type,omitempty"`
Subsets []Subset `json:"subsets,omitempty"`
}
type Subset struct {
Addresses []Address `json:"addresses,omitempty"`
}
type Address struct {
IP string `json:"IP,omitempty"`
}
func main() {
m := []byte(`{"type":"example","data": {"name": "abc","labels": {"key": "value"}},"subsets": [{"addresses": [{"ip": "192.168.103.178"}],"ports": [{"port": 80}]}]}`)
r := bytes.NewReader(m)
decoder := json.NewDecoder(r)
val := &Example{}
err := decoder.Decode(val)
if err != nil {
log.Fatal(err)
}
// If you want to read a response body
// decoder := json.NewDecoder(res.Body)
// err := decoder.Decode(val)
// Subsets is a slice so you must loop over it
for _, s := range val.Subsets {
// within Subsets, address is also a slice
// then you can access each IP from type Address
for _, a := range s.Addresses {
fmt.Println(a.IP)
}
}
}
The output would be:
192.168.103.178
By decoding this to a struct, you can loop over any slice and not limit yourself to one IP
Example here:
https://play.golang.org/p/sWA9qBWljA
One approach is to unmarshal the JSON to a map, e.g. (assumes jsData contains JSON string)
obj := map[string]interface{}{}
if err := json.Unmarshal([]byte(jsData), &obj); err != nil {
log.Fatal(err)
}
Next, implement a function for searching the value associated with a key from the map recursively, e.g.
func find(obj interface{}, key string) (interface{}, bool) {
//if the argument is not a map, ignore it
mobj, ok := obj.(map[string]interface{})
if !ok {
return nil, false
}
for k, v := range mobj {
//key match, return value
if k == key {
return v, true
}
//if the value is a map, search recursively
if m, ok := v.(map[string]interface{}); ok {
if res, ok := find(m, key); ok {
return res, true
}
}
//if the value is an array, search recursively
//from each element
if va, ok := v.([]interface{}); ok {
for _, a := range va {
if res, ok := find(a, key); ok {
return res,true
}
}
}
}
//element not found
return nil,false
}
Note, that the above function return an interface{}. You need to convert it to appropriate type, e.g. using type switch:
if ip, ok := find(obj, "ip"); ok {
switch v := ip.(type) {
case string:
fmt.Printf("IP is a string -> %s\n", v)
case fmt.Stringer:
fmt.Printf("IP implements stringer interface -> %s\n", v.String())
case int:
default:
fmt.Printf("IP = %v, ok = %v\n", ip, ok)
}
}
A working example can be found at https://play.golang.org/p/O5NUi4J0iR
Typically in these situations you will see people describe all of these sub struct types. If you don't actually need to reuse the definition of any sub structs (like as a type for a function argument), then you don't need to define them. You can just use one definition for the whole response. In addition, in some cases you don't need to define a type at all, you can just do it at the time of declaration:
package main
import "encoding/json"
const s = `
{
"subsets": [
{
"addresses": [
{"ip": "192.168.103.178"}
]
}
]
}
`
func main() {
var svc struct {
Subsets []struct {
Addresses []struct { Ip string }
}
}
json.Unmarshal([]byte(s), &svc)
ip := svc.Subsets[0].Addresses[0].Ip
println(ip == "192.168.103.178")
}
You can write your own decoder or use existing third-party decoders.
For instance, github.com/buger/jsonparser could solve your problem by iterating throw array (two times).
package main
import (
"github.com/buger/jsonparser"
"fmt"
)
var data =[]byte(`{
"type":"example",
"data":{
"name":"abc",
"labels":{
"key":"value"
}
},
"subsets":[
{
"addresses":[
{
"ip":"192.168.103.178"
}
],
"ports":[
{
"port":80
}
]
}
]
}`)
func main() {
jsonparser.ArrayEach(data, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
jsonparser.ArrayEach(value, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
v, _, _, err := jsonparser.Get(value, "ip")
if err != nil {
return
}
fmt.Println("ip: ", string(v[:]))
}, "addresses")
}, "subsets")
}
Output: ip: 192.168.103.178