Convert DynamoDB JSON to AttributeValue, Go Object or Json - json

I am trying to convert simple DynamoDB Object string:
{
"Item": {
"Id": {
"S": "db31"
},
"CreateTime": {
"N": "1647882237618915000"
}
}
to either dynamodb.AttributeValue and then map to a go object (go type structure) or convert to a simple JSON go object.
I think, there are similar answers (1, 2, 3) in Java, but I didn't find a similar implementation in Golang.

You could create a struct type and use json.Unmarshal to unmarshal the JSON string like this:
package main
import (
"encoding/json"
"fmt"
"os"
)
type Record struct {
Item struct {
Id struct {
S string
}
CreateTime struct {
N string
}
}
}
func main() {
str := `{
"Item": {
"Id": {
"S": "db31"
},
"CreateTime": {
"N": "1647882237618915000"
}
}
}`
var record Record
if err := json.Unmarshal([]byte(str), &record); err != nil {
fmt.Fprintf(os.Stderr, "unmarshal failed: %v", err)
os.Exit(1)
}
fmt.Printf("%s %s", record.Item.Id.S, record.Item.CreateTime.N)
}
If you want a different approach, and want to transform the result into a structure that is different than the JSON, you could use a library like gjson.
Here is an example "flattening" the result into a simpler struct:
package main
import (
"fmt"
"github.com/tidwall/gjson"
)
type Record struct {
Id string
CreateTime string
}
func main() {
str := `{
"Item": {
"Id": {
"S": "db31"
},
"CreateTime": {
"N": "1647882237618915000"
}
}
}`
values := gjson.GetMany(str, "Item.Id.S", "Item.CreateTime.N")
record := Record{
Id: values[0].Str,
CreateTime: values[1].Str,
}
fmt.Printf("%s %s", record.Id, record.CreateTime)
}

Related

Trouble mapping json to golang struct

I have a json stream as follows ...
[
{
"page": 1,
"pages": 7,
"per_page": "2000",
"total": 13200
},
[
{
"indicator": {
"id": "SP.POP.TOTL",
"value": "Population, total"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"value": null,
"decimal": "0",
"date": "2019"
},
{
"indicator": {
"id": "SP.POP.TOTL",
"value": "Population, total"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"value": "419790588",
"decimal": "0",
"date": "2018"
},
...
]
]
And I'm trying to decode it ... so I have the following struct ... but I keep getting
"cannot unmarshal array into Go value of type struct { P struct ... "
type Message []struct {
P struct {
Page int
}
V []struct {
Indicator struct {
Id string
Value string
}
Country struct {
Value string
}
Value string
Decimal string
Date string
}
}
My struct looks to match the json ... but obviously not! Any ideas?
Since your JSON array have two different types first unmarshal them into a slice of json.RawMessage which is []byte as underlying type so that we can unmarshal again JSON array data.
So unmarshal data for P and V struct type using index directly (predict) or detect if object(starting with '{') then unmarshal into P and array(starting with '[') then unmarshal into V. Now prepare your Message using those data.
type Message struct {
PageData P
ValData []V
}
type P struct {
Page int
}
type V struct {
Indicator struct {
Id string
Value string
}
Country struct {
Value string
}
Value string
Decimal string
Date string
}
func main() {
var rawdata []json.RawMessage
json.Unmarshal([]byte(jsonData), &rawdata)
var pageData P
json.Unmarshal(rawdata[0], &pageData)
var valData []V
json.Unmarshal(rawdata[1], &valData)
res := Message{pageData, valData}
fmt.Println(res)
}
var jsonData = `[...]` //your json data
Full code in Go Playground
As poWar said, the JSON you actually have is a list of objects whose types do not conform to each other. You must therefore unmarshal into something capable of holding different object types, such as interface{} or—since there is an outer array—[]interface{}.
You can also, if you like, decode into a []json.RawMessage. The underlying json.RawMessage itself has underlying type []byte so that it's basically the undecoded "inner" JSON. In at least some cases this is going to be more work than just decoding directly to []interface{} and checking each resulting interface, but you can, if you wish, decode to struct once you have the JSON separated out. For instance:
func main() {
var x []json.RawMessage
err := json.Unmarshal(input, &x)
if err != nil {
fmt.Printf("err = %v\n", err)
return
}
if len(x) != 2 {
fmt.Println("unexpected input")
return
}
var page struct {
Page int
}
err = json.Unmarshal(x[0], &page)
if err != nil {
fmt.Printf("unable to unmarshal page part: %v\n", err)
return
}
fmt.Printf("page = %d\n", page.Page)
// ...
}
Here on the Go Playground is a more complete example. See also Eklavya's answer.
Looking at your struct, your corresponding JSON should look something like this.
[
{
"P": {"page": 1},
"V": [
{
"Indicator": {"Id": ...},
"Country": {"Value":""},
"Value": "",
...
}
]
},
...
]
The JSON structure you are trying to Unmarshal looks like a list of objects where each object is not of the same type. You can start unmarshalling them into interfaces and defining each interface based on the object being unmarhsalled.
package main
import (
"encoding/json"
"log"
)
type Message []interface{}
func main() {
data := `[{"page":1,"pages":7,"per_page":"2000","total":13200},[{"indicator":{"id":"SP.POP.TOTL","value":"Population, total"},"country":{"id":"1A","value":"Arab World"},"value":null,"decimal":"0","date":"2019"},{"indicator":{"id":"SP.POP.TOTL","value":"Population, total"},"country":{"id":"1A","value":"Arab World"},"value":"419790588","decimal":"0","date":"2018"}]]`
var m Message
if err := json.Unmarshal([]byte(data), &m); err != nil {
log.Fatalf("could not unmarshal")
}
log.Printf("message: %v", m)
}
Output:
message: [map[page:1 pages:7 per_page:2000 total:13200] [map[country:map[id:1A value:Arab World] date:2019 decimal:0 indicator:map[id:SP.POP.TOTL value:Population, total] value:<nil>] map[country:map[id:1A value:Arab World] date:2018 decimal:0 indicator:map[id:SP.POP.TOTL value:Population, total] value:419790588]]]
[Edit]: Ideally you should change your JSON to be structured better for unmarshalling. If you do not have control on it, then your corresponding Go structure is just embedded maps of string to interfaces, which you will have to manually type cast and access.

Converting JSON with array

I have written following code to get array from JSON, and want to retrieve something like
[{"id":"id1","friendly":"friendly1"},{"id":"id2","friendly":"friendly2"}]
But it's empty:
[{"id":"","friendly":""},{"id":"","friendly":""}]
package main
import (
"encoding/json"
"fmt"
)
var input = `[
{
"not needed": "",
"_source": {
"id": "id1",
"friendly": "friendly1"
}
},
{
"_source": {
"id": "id2",
"friendly": "friendly2"
}
}]`
type source struct {
Id string `json:"id"`
Friendly string `json:"friendly"`
}
func main() {
result := make([]source, 0)
sources := []source{}
json.Unmarshal([]byte(input), &sources)
for _, n := range sources {
result = append(result, n)
}
out, _ := json.Marshal(result)
fmt.Println(string(out))
}
Try creating an another struct that have one field called Source of type source. In my example below I called this struct outer. Your input should be an array of outer and your result an array of source.
Something like this:
import (
"encoding/json"
"fmt"
)
var input = `[
{
"not needed": "",
"_source": {
"id": "id1",
"friendly": "friendly1"
}
},
{
"_source": {
"id": "id2",
"friendly": "friendly2"
}
}]`
type outer struct {
Source source `json:"_source"`
}
type source struct {
Id string `json:"id"`
Friendly string `json:"friendly"`
}
func main() {
result := make([]source, 0)
sources := []outer{}
json.Unmarshal([]byte(input), &sources)
for _, n := range sources {
result = append(result, n.Source)
}
out, _ := json.Marshal(result)
fmt.Println(string(out))
}```

Unmarshalling json arrays as json objects

I have to unmarshal a series of Json objects, but one of the objects contain a json array which is not really structured in a good way.
"labels": [
{
"key": "owner",
"value": "harry"
},
{
"key": "group",
"value": "student"
}
]
I am unmarshalling it using this struct -
type StudentDetails struct {
Id string `json:"id"`
Name string `json:"name"`
Labels []Label `json:"labels,omitempty"`
}
type Label struct {
Key string `json:"key"`
Value string `json:"value"`
}
And I have to access it using x.Labels[0].key == "owner" inside a for loop which is very annoying.
I want to be able to do x.Labels.Owner == "harry" instead. How do I go about achieving this? The rest of JSON is unmarshalled fine using the default unmarshal function, so I don't think writing custom function will be good option.
With the constraints you have here, this is about as close as you will get (run in playground):
package main
import (
"encoding/json"
"fmt"
)
func main() {
j := `
{
"id": "42",
"name": "Marvin",
"labels": [
{
"key": "owner",
"value": "harry"
},
{
"key": "group",
"value": "student"
}
]
}`
d := StudentDetails{}
err := json.Unmarshal([]byte(j), &d)
if err != nil {
panic(err)
}
fmt.Println(d.Labels["owner"])
fmt.Println(d.Labels["group"])
}
type StudentDetails struct {
Id string `json:"id"`
Name string `json:"name"`
Labels Labels `json:"labels"`
}
type Labels map[string]string
func (l *Labels) UnmarshalJSON(b []byte) error {
a := []map[string]string{}
err := json.Unmarshal(b, &a)
if err != nil {
return err
}
t := map[string]string{}
for _, m := range a {
t[m["key"]] = m["value"]
}
*l = t
return nil
}
How about to define custom []Label type and add function on it.
For instance
type Labels []Label
func (l Labels) Owner() string {
if len(l) > 1 {
return l[0].Value
}
return ""
}

How to unmarshal JSON in to array of interface and use

I am having difficulty understanding how to correctly unmarshal some JSON data that goes in to an array of type inteface and then use it. I tried to make this example code as simple as possible to illustrate the problem I am having. The code can be found in the playground here: https://play.golang.org/p/U85J_lBJ7Zr
The output looks like:
[map[ObjectType:chair ID:1234 Brand:Blue Inc.] map[ID:5678
Location:Kitchen ObjectType:table]] { } false { } false
Code
package main
import (
"fmt"
"encoding/json"
)
type Chair struct {
ObjectType string
ID string
Brand string
}
type Table struct {
ObjectType string
ID string
Location string
}
type House struct {
Name string
Objects []interface{}
}
func main() {
var h House
data := returnJSONBlob()
err := json.Unmarshal(data, &h)
if err != nil {
fmt.Println(err)
}
fmt.Println(h.Objects)
s1, ok := h.Objects[0].(Table)
fmt.Println(s1, ok)
s2, ok := h.Objects[0].(Chair)
fmt.Println(s2, ok)
}
func returnJSONBlob() []byte {
s := []byte(`
{
"Name": "house1",
"Objects": [
{
"ObjectType": "chair",
"ID": "1234",
"Brand": "Blue Inc."
},
{
"ObjectType": "table",
"ID": "5678",
"Location": "Kitchen"
}
]
}
`)
return s
}
I'm not sure if this is practical, since this is a simplified version of your scenario. However, one way to do this is combine the two object types to a new one, Object, and then unmarshal them directly to Object instead of using interface{}:
package main
import (
"encoding/json"
"fmt"
)
type Object struct {
ObjectType string
ID string
Brand string
Location string
}
type House struct {
Name string
Objects []Object
}
func returnJSONBlob() []byte {
s := []byte(`
{
"Name": "house1",
"Objects": [
{
"ObjectType": "chair",
"ID": "1234",
"Brand": "Blue Inc."
},
{
"ObjectType": "table",
"ID": "5678",
"Location": "Kitchen"
}
]
}
`)
return s
}
func main() {
var h House
data := returnJSONBlob()
err := json.Unmarshal(data, &h)
if err != nil {
fmt.Println(err)
}
fmt.Println(h.Objects[0].Brand)
fmt.Println(h.Objects[1].Location)
}
Prints:
Blue Inc.
Kitchen
Example here: https://play.golang.org/p/91F4UrQlSjJ

How to decode json into structs

I'm trying to decode some json in Go but some fields don't get decoded.
See the code running in browser here:
What am I doing wrong?
I need only the MX records so I didn't define the other fields. As I understand from the godoc you don't need to define the fields you don't use/need.
// You can edit this code!
// Click here and start typing.
package main
import "fmt"
import "encoding/json"
func main() {
body := `
{"response": {
"status": "SUCCESS",
"data": {
"mxRecords": [
{
"value": "us2.mx3.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx1.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx2.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
}
],
"cnameRecords": [
{
"aliasHost": "pop.a.co.uk.",
"canonicalHost": "us2.pop.mailhostbox.com."
},
{
"aliasHost": "webmail.a.co.uk.",
"canonicalHost": "us2.webmail.mailhostbox.com."
},
{
"aliasHost": "smtp.a.co.uk.",
"canonicalHost": "us2.smtp.mailhostbox.com."
},
{
"aliasHost": "imap.a.co.uk.",
"canonicalHost": "us2.imap.mailhostbox.com."
}
],
"dkimTxtRecord": {
"domainname": "20a19._domainkey.a.co.uk",
"value": "\"v=DKIM1; g=*; k=rsa; p=DkfbhO8Oyy0E1WyUWwIDAQAB\"",
"ttl": 1
},
"spfTxtRecord": {
"domainname": "a.co.uk",
"value": "\"v=spf1 redirect=_spf.mailhostbox.com\"",
"ttl": 1
},
"loginUrl": "us2.cp.mailhostbox.com"
}
}}`
type MxRecords struct {
value string
ttl int
priority int
hostName string
}
type Data struct {
mxRecords []MxRecords
}
type Response struct {
Status string `json:"status"`
Data Data `json:"data"`
}
type apiR struct {
Response Response
}
var r apiR
err := json.Unmarshal([]byte(body), &r)
if err != nil {
fmt.Printf("err was %v", err)
}
fmt.Printf("decoded is %v", r)
}
As per the go documentaiton about json.Unmarshal, you can only decode toward exported fields, the main reason being that external packages (such as encoding/json) cannot acces unexported fields.
If your json doesn't follow the go convention for names, you can use the json tag in your fields to change the matching between json key and struct field.
Exemple:
package main
import (
"fmt"
"encoding/json"
)
type T struct {
Foo string `json:"foo"`
priv string `json:"priv"`
}
func main() {
text := []byte(`{"foo":"bar", "priv":"nothing"}`)
var t T
err := json.Unmarshal(text, &t)
if err != nil {
panic(err)
}
fmt.Println(t.Foo) // prints "bar"
fmt.Println(t.priv) // prints "", priv is not exported
}
You must Uppercase struct fields:
type MxRecords struct {
Value string `json:"value"`
Ttl int `json:"ttl"`
Priority int `json:"priority"`
HostName string `json:"hostName"`
}
type Data struct {
MxRecords []MxRecords `json:"mxRecords"`
}
http://play.golang.org/p/EEyiISdoaE
The encoding/json package can only decode into exported struct fields. Your Data.mxRecords member is not exported, so it is ignored when decoding. If you rename it to use a capital letter, the JSON package will notice it.
You will need to do the same thing for all the members of your MxRecords type.