Merge a dynamic data structure in Go - json

Question

I have this incoming payload, that I cannot change.
{
"source": "some random source",
"table": "hosts_table",
"data": [
["address", "id", "services_with_info"],
["0.0.0.1", 1111, [
["service_3", "is very cool", 1],
["service_4", "is very cool", 2]
]
],
["0.0.0.2", 2222, [
["service_3", "is very cool", 3],
["service_4", "is very cool", 4]
]
]
]}
i need to take the first index of "data" and create a new JSON object, that looks like this...
"data": [{
"address": "0.0.0.1",
"id": 1111,
"services_with_info":[
{
"service_name": "service_1",
"service_message": "is very cool",
"service_id": 1
},
{...}
]},
{...}]
and then build an []Host's from it the data structure is 5k "hosts" long. I was able to map this to a struct, but need to get it into this format first. I understand how to unmarshal the JSON, but only if I can convert the payload to the above.

Answer 1

You can use json.Unmarshal for this and parse data with your conditions. I'm just doing it for "data" and you can do same for "services_with_info"
b := []byte(`{
"source": "some random source",
"table": "hosts_table",
"data": [
["address", "id", "services_with_info"],
["0.0.0.1", 1111, [
["service_3", "is very cool", 1],
["service_4", "is very cool", 2]
]
],
["0.0.0.2", 2222, [
["service_3", "is very cool", 3],
["service_4", "is very cool", 4]
]
]
]}`)
var f interface{}
err := json.Unmarshal(b, &f)
if err != nil {
fmt.Println(err)
return
}
m := f.(map[string]interface{})
result := make(map[string]interface{})
results := make(map[string][]map[string]interface{})
for k, v := range m {
if k == "data" {
s := v.([]interface{})
header := make([]interface{}, 3)
for i, u := range s {
if i == 0 {
header = u.([]interface{})
} else {
row := u.([]interface{})
for j, k := range header {
result[k.(string)] = row[j]
}
results["data"] = append(results["data"], result)
}
}
}
}
fmt.Println(results)
here "results" is "data" as required.

Answer 2

I'm not sure if I understood what you wants.
May be some thing like this?
Probably it needs some work, like make slice of pointers to structs instead of slice of structs to prevent allocation and copy, error handling, more custom logic to convert values, anonymize/incapsulate private structs used in the middle of conversion, add json tags to those structures etc.
I create custom Unmarshaller for Data field on IncomingPaylod: parsing expected data, converting it to []MyData and updating Data field with it.
I created custom Unmarshallers for expected_data and expected_services_with_info because we expect it as array of values (3 values: string, int and [array of string, int(?), int]), but I want to convert it to nice structs. If you dont like it, you can delete it, Unmarshal expected data to []interface{} and work with it like []interface{}{string, int, []interface{}{string, int, int} }. Easy to get it wrong, so i like structs more, its easier to read and maintain and refactor (i think there are more fields in you app).
https://play.golang.org/p/xHTvyhecra
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type IncomingPayload struct {
Source string `json:"source"`
Table string `json:"table"`
Data MyDataSlice `json:"data"`
}
type MyDataSlice []MyData
type MyData struct {
Address string `json:"address"`
ID string `json:"id"`
Services_with_info []MyServiceWithInfo `json:"services_with_info"`
}
type MyServiceWithInfo struct {
ServiceName string `json:"service_name"`
ServiceMessage string `json:"service_message"`
ServiceID int `json:"service_id"`
}
type expected_data struct {
IP string
ID int
Info []expected_services_with_info
}
type expected_services_with_info struct {
Name string
Desc string
ID int
}
func (ed *expected_data) UnmarshalJSON(buf []byte) error {
tmp := []interface{}{&ed.IP, &ed.ID, &ed.Info}
// converts ["address", "id", "services_with_info"] into a struct
// will unmarshall "services_with_info" (ed.Info) with *expected_services_with_info.UnmarshalJSON
json.Unmarshal(buf, &tmp)
return nil
}
func (es *expected_services_with_info) UnmarshalJSON(buf []byte) error {
tmp := []interface{}{&es.Name, &es.Desc, &es.ID}
// converts ["service_3", "is very cool", 1] into a struct
json.Unmarshal(buf, &tmp)
return nil
}
func (md *MyDataSlice) UnmarshalJSON(p []byte) error {
var incoming_data_slice []expected_data
json.Unmarshal(p, &incoming_data_slice)
//fmt.Println("incoming", incoming_data_slice)
//transform incoming_data_slice to your needs using your data type
for i := range incoming_data_slice {
my_data := MyData{
Address: incoming_data_slice[i].IP, //copy
ID: strconv.Itoa(incoming_data_slice[i].ID), //some transformation
//nil slice is totally fine, but if you wish you can do
//Data: make(MyDataSlice, len(incoming_data_slice)),
}
//not sure what would be best: "i := range data" or "_, v := range data" (second one makes a copy? and causes allocation)
for j := range incoming_data_slice[i].Info {
tmp := MyServiceWithInfo{
ServiceName: incoming_data_slice[i].Info[j].Name,
ServiceMessage: incoming_data_slice[i].Info[j].Desc,
ServiceID: incoming_data_slice[i].Info[j].ID,
}
my_data.Services_with_info = append(my_data.Services_with_info, tmp)
}
//and populate
*md = append(*md, my_data)
}
return nil
}
func main() {
test_json := `{
"source": "some random source",
"table": "hosts_table",
"data": [
["address", "id", "services_with_info"],
["0.0.0.1", 1111, [
["service_3", "is very cool", 1],
["service_4", "is very cool", 2]
]
],
["0.0.0.2", 2222, [
["service_3", "is very cool", 3],
["service_4", "is very cool", 4]
]
]
]}`
var payload IncomingPayload
json.Unmarshal([]byte(test_json), &payload)
fmt.Println("payload", payload)
buf, _ := json.MarshalIndent(payload, "", "\t")
fmt.Println(string(buf))
}