Golang Json unmarshaling with multiple dynamically named fields - json

I have the following Json structure that I need to decode and I'm not sure the proper approach
for unmarshalling it.
var b = []byte(`{
"Card:5417871461137421886": {
"applicationUUID": "5417871461137421886",
"commit": "test_commit",
"lastSeen": "test_lastSeen",
"localUUID": "E4:F5:13:8E:F5:43",
"progress": "3.5",
"externalUUID": "test_externalUUID",
"state": "test_state"
},
"Card:5417871461137421886.Color": {
"applicationUUID": "5417871461137421886",
"color": "blue"
}
}`)
My initial approach has been by somehow creating a structure then unmarshalling to the struct, but due to the dynamic nature I'm unsure how to accomplish this. One approach I've tried is using a parallel structure however I have not succeeded at making that work properly also. I'm kind of at a loss for a good way to solve this problem You can see my sample code here https://play.golang.org/p/mzMArgo8ta2
I can not change the structure of the Json because it is being read from an API. Ultimately I do not want or care about the unique field names, since the application ID is always listed as a value. If there would be a simple way to remove all the Card:Id I figure I could just map it like normal.

One option may be to delay parsing until you know the type; something like the following (playground):
devices := make(map[string]json.RawMessage)
err := json.Unmarshal(b, &devices)
if err != nil {
log.Fatal(err)
}
for key, jsonVal:= range devices {
if len(key) > 5 && key[:5] == "Card:" {
var dev Device
err := json.Unmarshal(jsonVal, &dev)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%#v\n", dev)
} else if len(key) > 6 && key[:6] == "Color:" {
var oth Other
err := json.Unmarshal(jsonVal, &oth)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%#v\n", oth)
}
}
Note: Its not clear what you want to do with the data so I'm just printing the structs for now. Its also not really clear how to identify the type of each element (the example in your question differs from the example in your sample code); there may be some documentation for the API that explains this?.

I would do it like this:
package main
import (
"encoding/json"
"fmt"
)
const s = `
{
"Card:5417871461137421886": {
"applicationUUID": "5417871461137421886",
"commit": "test_commit",
"externalUUID": "test_externalUUID",
"lastSeen": "test_lastSeen",
"localUUID": "E4:F5:13:8E:F5:43",
"progress": "3.5",
"state": "test_state"
},
"Card:5417871461137421886.Color": {
"applicationUUID": "5417871461137421886",
"color": "blue"
}
}
`
func main() {
var m map[string]struct {
ApplicationUUID, Color, Commit, ExternalUUID, LastSeen, LocalUUID string
Progress, State string
}
json.Unmarshal([]byte(s), &m)
fmt.Printf("%+v\n", m)
}
but it seems like a map[string]map[string]string would work as well.

Related

What could be a unit test case for the s3/sqs event when unmarshalling json to struck in Go Lang?

What could be a unit test case for the s3/sqs event when unmarshalling json to struck in Go?
One option that comes to mind is if the "Body" is empty, but not sure what else could it be.
Code example:
type Event struct {
Message string `json:"Message"`
}
func unmarshal(message events.SQSMessage) (s3Event, error) {
var e Event
err := json.Unmarshal([]byte(message.Body), &e)
if err != nil {
return s3Event, err
}
var s3Event events.S3Event
err = json.Unmarshal([]byte(e.Message), &sqsEvent)
if err != nil {
return s3Event, err
}
return s3Event, nil
}
Would appreciate any advice.
Premise: I'm not sure if I've fully understood your question. Furthermore, the code has some issues that cause me hard-time to figure out what's going on. With that being said, I'll try to help you by providing a piece of working software (it's not very useful) together with a piece of software written just to test it. Maybe, this can help you in figuring out your issue.
unmarshal.go file
package unmarshaltest
import (
"encoding/json"
"fmt"
"github.com/aws/aws-lambda-go/events"
)
type Event struct {
Message string `json:"body"`
}
func Unmarshal(message events.SQSMessage) (string, error) {
rawBytes, _ := json.Marshal(message)
var e Event
json.Unmarshal(rawBytes, &e)
if e.Message == "" {
return "", fmt.Errorf("err while unmarshaling")
}
return e.Message, nil
}
One thing I fixed and you should pay attention to is how you name your functions. If you call your function unmarshal with lowercase u it means that this function should not be exported from this package (it's private). So I changed the name with the U as I supposed you wanna test this function.
unmarshal_test.go file
package unmarshaltest
import (
"encoding/json"
"testing"
"github.com/aws/aws-lambda-go/events"
"github.com/stretchr/testify/assert"
)
func TestUnmarshal(t *testing.T) {
t.Run("Unmarshal_NotEmpty_Message", func(t *testing.T) {
var sqsMsg events.SQSMessage
sqsEventRaw := `{ "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78", "receiptHandle": "MessageReceiptHandle", "body": "My own event payload!", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1523232000000", "SenderId": "123456789012", "ApproximateFirstReceiveTimestamp": "1523232000001" }, "messageAttributes": {}, "md5OfBody": "4d1d0024b51659ad8c3725f9ba7e2471", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue", "awsRegion": "us-east-1" }`
json.Unmarshal([]byte(sqsEventRaw), &sqsMsg)
res, err := Unmarshal(sqsMsg)
assert.Equal(t, "My own event payload!", res)
assert.Nil(t, err)
})
t.Run("Unmarshal_Empty_Message", func(t *testing.T) {
res, err := Unmarshal(events.SQSMessage{})
assert.Equal(t, "", res)
assert.NotNil(t, err)
})
}
Here you can find a simple piece of code that is used to test the two possible execution flows that our code might take.
You should also pay attention to how you write the code. It's very hard to swap the invocation to the json.Unmarshal function with a mock as it's hard-coded into the code. If your wish is to be able to mock out this function, you should rely on interfaces in your production code and replace them with mocks during tests.
Let me know if this clarifies something and if you need something else!

Proper json unmarshaling in Go with the empty interface

I'm currently learning golang and (probably as many others before me) I'm trying to properly understand the empty interface.
As an exercise, I'm reading a big json file produced by Postman and trying to access just one field (out of the many available).
Here is a simple representation of the json without the unnecessary fields I don't want to read (but that are still there):
{
"results": [
{
"times": [
1,
2,
3,
4
]
}
]
}
Since the json object is big, I opted out of unmarshaling it with a custom struct, and rather decided to use the empty interface interface{}
After some time, I managed to get some working code, but I'm quite sure this isn't the correct way of doing it.
byteValue, _ := ioutil.ReadAll(jsonFile)
var result map[string]interface{}
err = json.Unmarshal(byteValue, &result)
if err != nil {
log.Fatalln(err)
}
// ESPECIALLY UGLY
r := result["results"].([]interface{})
r1 := r[0].(map[string]interface{})
r2 := r1["times"].([]interface{})
times := make([]float64, len(r2))
for i := range r2 {
times[i] = r2[i].(float64)
}
Is there a better way to navigate through my json object without having to instantiate new variables every time i move deeper and deeper into the object?
Even if the JSON is large, you only have to define the fields you actually care about
You only need to use JSON tags if the keys aren't valid Go
identifiers (keys are idents in this case), even then you can sometimes avoid it by using a map[string]something
Unless you need the sub structs for some function or whatnot, you don't need to define them
Unless you need to reuse the type, you don't even have to define that, you can just define the struct at declaration time
Example:
package main
import (
"encoding/json"
"fmt"
)
const s = `
{
"results": [
{
"times": [1, 2, 3, 4]
}
]
}
`
func main() {
var t struct {
Results []struct {
Times []int
}
}
json.Unmarshal([]byte(s), &t)
fmt.Printf("%+v\n", t) // {Results:[{Times:[1 2 3 4]}]}
}
[...] trying to access just one field (out of the many available).
For this concrete use case I would use a library to query and access to a single value in a known path like:
https://github.com/jmespath/go-jmespath
In the other hand, if you're practicing how to access nested values in a JSON, I would recommend you to give a try to write a recursive function that follows a path in an unknown structure the same way (but simple) like go-jmespath does.
Ok, I challenged myself and spent an hour writing this. It works. Not sure about performance or bugs and it's really limited :)
https://play.golang.org/p/dlIsmG6Lk-p
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
)
func main() {
// I Just added a bit more of data to the structure to be able to test different paths
fileContent := []byte(`
{"results": [
{"times": [
1,
2,
3,
4
]},
{"times2": [
5,
6,
7,
8
]},
{"username": "rosadabril"},
{"age": 42},
{"location": [41.5933262, 1.8376757]}
],
"more_results": {
"nested_1": {
"nested_2":{
"foo": "bar"
}
}
}
}`)
var content map[string]interface{}
if err := json.Unmarshal(fileContent, &content); err != nil {
panic(err)
}
// some paths to test
valuePaths := []string{
"results.times",
"results.times2",
"results.username",
"results.age",
"results.doesnotexist",
"more_results.nested_1.nested_2.foo",
}
for _, p := range valuePaths {
breadcrumbs := strings.Split(p, ".")
value, err := search(breadcrumbs, content)
if err != nil {
fmt.Printf("\nerror searching '%s': %s\n", p, err)
continue
}
fmt.Printf("\nFOUND A VALUE IN: %s\n", p)
fmt.Printf("Type: %T\nValue: %#v\n", value, value)
}
}
// search is our fantastic recursive function! The idea is to search in the structure in a very basic way, for complex querying use jmespath
func search(breadcrumbs []string, content map[string]interface{}) (interface{}, error) {
// we should never hit this point, but better safe than sorry and we could incurr in an out of range error (check line 82)
if len(breadcrumbs) == 0 {
return nil, errors.New("ran out of breadcrumbs :'(")
}
// flag that indicates if we are at the end of our trip and whe should return the value without more checks
lastBreadcrumb := len(breadcrumbs) == 1
// current breadcrumb is always the first element.
currentBreadcrumb := breadcrumbs[0]
if value, found := content[currentBreadcrumb]; found {
if lastBreadcrumb {
return value, nil
}
// if the value is a map[string]interface{}, go down the rabbit hole, recursion!
if aMap, isAMap := value.(map[string]interface{}); isAMap {
// we are calling ourselves popping the first breadcrumb and passing the current map
return search(breadcrumbs[1:], aMap)
}
// if it's an array of interfaces the thing gets complicated :(
if anArray, isArray := value.([]interface{}); isArray {
for _, something := range anArray {
if aMap, isAMap := something.(map[string]interface{}); isAMap && len(breadcrumbs) > 1 {
if v, err := search(breadcrumbs[1:], aMap); err == nil {
return v, nil
}
}
}
}
}
return nil, errors.New("woops, nothing here")
}

Go wrapping json.Number with sql.NullFloat64

I'm working with an API that has json values that can be a number or string and can sometimes be an empty string. For example:
[
{
"Description": "Doorknob",
"Amount": 3.25
},
{
"Description": "Light bulb",
"Amount": "4.70"
},
{
"Description": "Screwdriver",
"Amount": ""
}
]
I've learned I can use json.Number to deal with cases where the value can be a number or a string. But how do I deal with the cases where it's an empty string?
I need to insert these values into a postgres database so I'd like cases where it's an empty string to show as null and I'm thinking sql.NullFloat64 would be best suited to handle it.
Here's a function I'm using that works on the first 2 cases (Doorknob and Light bulb), but fails to unmarshal the last ("Screwdriver").
Here is an unmarshaling function I'm trying to use, but I can't figure out why it's not working:
type NullNumber struct{ sql.NullFloat64 }
func (nn *NullNumber) UnmarshalJSON(data []byte) error {
var x *json.Number
if err := json.Unmarshal(data, &x); err != nil {
return err
}
if len(*x) == 0 {
nn.Valid = false // This doesn't seem to be working. Why?
}
this, err := x.Float64()
if err != nil {
return err
}
nn.Valid = true
nn.Float64 = this
return nil
}
Is this a case where I should be using an open interface for the amount?
Any help is very appreciated.
playground: https://play.golang.org/p/QYQRq94OtV3
You're almost there! json.Unmarshal call errors when a json.Number is given an empty string.
Add this check beforehand:
// edge case: json.Number is given an empty string
if bytes.Equal(data, []byte(`""`)) {
nn.Valid = false
return nil
}
https://play.golang.org/p/ILxC8tjYI_G

How to decode complicated unnamed JSON in Golang

I am currently trying to decode the following JSON structure:
[
{
"2015-08-14 19:29:48-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1a",
"UTC_time": "2015-08-14 23:29:48",
"error": "0"
}
},
{
"2015-08-14 19:37:07-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:37:07",
"error": "0"
}
},
{
"2015-08-14 19:44:16-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:44:16",
"error": "0"
}
}
]
This is to eventually have a slice of Reading structs, formatted as the following:
type reading struct {
Value string `json:"value"`
MTID string `json:"measurement_tag_id"`
UTCTime string `json:"UTC_time"`
Error string `json:"error"`
}
I would then like to add this into an existing structure nested as:
type site struct {
Name string
ID string
Tags []tag
}
type tag struct {
ID string
Readings []reading
}
I've currently been able to create the base structure for sites and tags from a more typical JSON payload with appropriate keys. I have been unsuccessful though in figuring out how to decode the reading JSON. So far the closest I have gotten is via map[string]interface{} chaining, but this feels incredibly clunky and verbose.
Solution so far for reference:
var readingData []interface{}
if err := json.Unmarshal(file, &readingData); err != nil {
panic(err)
}
readings := readingData[0].(map[string]interface{})
firstReading := readings["2015-08-14 19:29:48-04:00"].(map[string]interface{})
fmt.Println(firstReading)
value := firstReading["value"].(string)
error := firstReading["error"].(string)
MTID := firstReading["measurement_tag_id"].(string)
UTCTime := firstReading["UTC_time"].(string)
fmt.Println(value, error, MTID, UTCTime)
While I am not sure if its necessary yet, I would also like to hold on to the arbitrary date keys. My first thought was to create a function that returned a map[string]reading but I am not sure how feasible this is.
Thanks for the help in advance!
You can have your reading type implement the json.Unmarshaler interface.
func (r *reading) UnmarshalJSON(data []byte) error {
type _r reading // same structure, but no methods, avoids infinite calls to this method
m := map[string]_r{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
for _, v := range m {
*r = reading(v)
}
return nil
}
https://play.golang.org/p/7X1oB77XL4
Another way is to use a slice of maps to parse, then copy to slice of readings, eg:
var readingMaps []map[string]reading //slice of maps of string key to reading value
if err := json.Unmarshal([]byte(data), &readingMaps); err != nil {
panic(err)
}
readings := []reading{}
for _, m := range readingMaps {
for _, r := range m {
readings = append(readings, r)
}
}
play.golang.org/p/jXTdmaZz7s

Serialize a map using a specific order

I have a map that uses string for both key and value. I have an array of keys that specifies the order of the values of the map.
I want to serialize that map to a JSON, but keeping the order defined on the array.
There is a sample code here: http://play.golang.org/p/A52GTDY6Wx
I want to serialize it as:
{
"name": "John",
"age": "20"
}
But if I serialize the map directly, the keys are ordered alphabetically:
{
"age": "20",
"name": "John"
}
I can serialize it as an array of maps, thus keeping the order, however that generates a lot of undesired characters:
[
{
"name": "John"
},
{
"age": "20"
}
]
In my real code I need to serialize the results of a database query which is specified in a text file, and I need to maintain the column order. I cannot use structs because the columns are not known at compile time.
EDIT: I don't need to read the JSON later in the specified order. The generated JSON is meant to be read by people, so I just want it to be as humanly readable as possible.
I could use a custom format but JSON suits me perfectly for this.
Thanks!
You need to implement the json.Marshaler interface on a custom type. This has the advantage of playing well within other struct types.
Sorry, you're always going to have to write a little bit of JSON encoding code.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type KeyVal struct {
Key string
Val interface{}
}
// Define an ordered map
type OrderedMap []KeyVal
// Implement the json.Marshaler interface
func (omap OrderedMap) MarshalJSON() ([]byte, error) {
var buf bytes.Buffer
buf.WriteString("{")
for i, kv := range omap {
if i != 0 {
buf.WriteString(",")
}
// marshal key
key, err := json.Marshal(kv.Key)
if err != nil {
return nil, err
}
buf.Write(key)
buf.WriteString(":")
// marshal value
val, err := json.Marshal(kv.Val)
if err != nil {
return nil, err
}
buf.Write(val)
}
buf.WriteString("}")
return buf.Bytes(), nil
}
func main() {
dict := map[string]interface{}{
"orderedMap": OrderedMap{
{"name", "John"},
{"age", 20},
},
}
dump, err := json.Marshal(dict)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", dump)
}
Outputs
{"orderedMap":{"name":"John","age":20}}
For that specific requirement you really don't need to use json.Marshal at all, you can simply implement your own function like this:
type OrderedMap map[string]string
func (om OrderedMap) ToJson(order ...string) string {
buf := &bytes.Buffer{}
buf.Write([]byte{'{', '\n'})
l := len(order)
for i, k := range order {
fmt.Fprintf(buf, "\t\"%s\": \"%v\"", k, om[k])
if i < l-1 {
buf.WriteByte(',')
}
buf.WriteByte('\n')
}
buf.Write([]byte{'}', '\n'})
return buf.String()
}
func main() {
om := OrderedMap{
"age": "20",
"name": "John",
}
fmt.Println(om.ToJson("name", "age"))
}
Probably the easiest solution: https://github.com/iancoleman/orderedmap
Although it might be slow as it's mentioned here
Here is a MapSlice implementation similar to what YAML v2 offers. It can do both Marshal and Unmarshal.
https://github.com/mickep76/mapslice-json