I am currently trying to decode the following JSON structure:
[
{
"2015-08-14 19:29:48-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1a",
"UTC_time": "2015-08-14 23:29:48",
"error": "0"
}
},
{
"2015-08-14 19:37:07-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:37:07",
"error": "0"
}
},
{
"2015-08-14 19:44:16-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:44:16",
"error": "0"
}
}
]
This is to eventually have a slice of Reading structs, formatted as the following:
type reading struct {
Value string `json:"value"`
MTID string `json:"measurement_tag_id"`
UTCTime string `json:"UTC_time"`
Error string `json:"error"`
}
I would then like to add this into an existing structure nested as:
type site struct {
Name string
ID string
Tags []tag
}
type tag struct {
ID string
Readings []reading
}
I've currently been able to create the base structure for sites and tags from a more typical JSON payload with appropriate keys. I have been unsuccessful though in figuring out how to decode the reading JSON. So far the closest I have gotten is via map[string]interface{} chaining, but this feels incredibly clunky and verbose.
Solution so far for reference:
var readingData []interface{}
if err := json.Unmarshal(file, &readingData); err != nil {
panic(err)
}
readings := readingData[0].(map[string]interface{})
firstReading := readings["2015-08-14 19:29:48-04:00"].(map[string]interface{})
fmt.Println(firstReading)
value := firstReading["value"].(string)
error := firstReading["error"].(string)
MTID := firstReading["measurement_tag_id"].(string)
UTCTime := firstReading["UTC_time"].(string)
fmt.Println(value, error, MTID, UTCTime)
While I am not sure if its necessary yet, I would also like to hold on to the arbitrary date keys. My first thought was to create a function that returned a map[string]reading but I am not sure how feasible this is.
Thanks for the help in advance!
You can have your reading type implement the json.Unmarshaler interface.
func (r *reading) UnmarshalJSON(data []byte) error {
type _r reading // same structure, but no methods, avoids infinite calls to this method
m := map[string]_r{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
for _, v := range m {
*r = reading(v)
}
return nil
}
https://play.golang.org/p/7X1oB77XL4
Another way is to use a slice of maps to parse, then copy to slice of readings, eg:
var readingMaps []map[string]reading //slice of maps of string key to reading value
if err := json.Unmarshal([]byte(data), &readingMaps); err != nil {
panic(err)
}
readings := []reading{}
for _, m := range readingMaps {
for _, r := range m {
readings = append(readings, r)
}
}
play.golang.org/p/jXTdmaZz7s
Related
I need to parse this json
{
"version": "1.1.29-snapshot",
"linux-amd64": {
"url": "https://origin/path",
"size": 7794688,
"sha256": "14b3c3ad05e3a98d30ee7e774646aec7ffa8825a1f6f4d9c01e08bf2d8a08646"
},
"windows-amd64": {
"url": "https://origin/path",
"size": 8102400,
"sha256": "01b8b927388f774bdda4b5394e381beb592d8ef0ceed69324d1d42f6605ab56d"
}
}
Keys like linux-amd64 are dynamic and theirs amount is arbitrary. I tried something like that to describe it and unmarshal. Obviously it doesn't work. Items is always empty.
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
It's similar to this use case, but has no parent key items. I suppose I can use 3rd party library or map[string]interface{} approach, but I'm interested in knowing how to achieve this with explicitly declared types.
The rest of the parsing code is:
func parseUpdateJson(jsonStr []byte) (UpdateInfo, error) {
var allInfo = UpdateInfo{Items: make(map[string]FileInfo)}
var err = json.Unmarshal(jsonStr, &allInfo)
return allInfo, err
}
Look at the link I attached and you will realize that is not that simple as you think. Also I pointed that I interested in typed approach. Ok, how to declare this map[string]FileInfo to get parsed?
You can create a json.Unmarshaller to decode the json into a map, then apply those values to your struct: https://play.golang.org/p/j1JXMpc4Q9u
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
func (i *UpdateInfo) UnmarshalJSON(d []byte) error {
tmp := map[string]json.RawMessage{}
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
err = json.Unmarshal(tmp["version"], &i.Version)
if err != nil {
return err
}
delete(tmp, "version")
i.Items = map[string]FileInfo{}
for k, v := range tmp {
var item FileInfo
err := json.Unmarshal(v, &item)
if err != nil {
return err
}
i.Items[k] = item
}
return nil
}
This answer is adapted from this recipe in my YouTube video on advanced JSON handling in Go.
func (u *UpdateInfo) UnmarshalJSON(d []byte) error {
var x struct {
UpdateInfo
UnmarshalJSON struct{}
}
if err := json.Unmarshal(d, &x); err != nil {
return err
}
var y map[string]json.RawMessage{}
if err := json.Unsmarshal(d, &y); err != nil {
return err
}
delete(y, "version"_ // We don't need this in the map
*u = x.UpdateInfo
u.Items = make(map[string]FileInfo, len(y))
for k, v := range y {
var info FileInfo
if err := json.Unmarshal(v, &info); err != nil {
return err
}
u.Items[k] = info
}
return nil
}
It:
Unmarshals the JSON into the struct directly, to get the struct fields.
It re-unmarshals into a map of map[string]json.RawMessage to get the arbitrary keys. This is necessary since the value of version is not of type FileInfo, and trying to unmarshal directly into map[string]FileInfo will thus error.
It deletes the keys we know we already got in the struct fields.
It then iterates through the map of string to json.RawMessage, and finally unmarshals each value into the FileInfo type, and stores it in the final object.
If you really don't want to unmarshal multiple times, your next best option is to iterate over the JSON tokens in your input by using the json.Decoder type. I've done this in a couple of performance-sensitive bits of code, but it makes your code INCREDIBLY hard to read, and in almost all cases is not worth the effort.
Source server return the data in a Json format of multiple object
interfaces, how can we parse such data ?
I am using a variable of JSON map[string]interface{} type to hold the result from server
The data return from Server.
"data": [
{
"group": "PAA_TEST",
"id": "2018-04-10T09:24:18.000000Z",
"name": "PAA_STATION",
"released": true,
"version": 33
},
{
"group": "PAA_TEST",
"id": "2018-03-19T10:50:21.000000Z",
"name": "PAA_STATION",
"released": false,
"version": 32
}
my fmt.print output outputdata["data"] //where output data is of JSON
map[string]interface{}
[
map[group:PAA_TEST id:2018-04-10T09:24:18.000000Z name:PAA_STATION
released:true version:33]
map[group:PAA_TEST id:2018-03-19T10:50:21.000000Z name:PAA_STATION
released:false version:32]
]
How can we iterate with multiple Map interfaces? For example, if I just want to process the information with released status as true. I am trying various method for indexing but no luck yet.
The best solution is to decode the JSON directly to a Go type that matches the structure of the data. This avoids the type assertions required to dig through a map[string]interface{}.
I'll assume that the common function looks something like this:
func request(path string, ... more arguments) (map[string]interface{}}, error) {
...
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
...
var result map[string]interface{}
err := json.NewDecoder(resp.Body).Decode(&result)
return result, err
}
Change the function to take a pointer to the result as an argument:
func request(pv interface{}, path string, ... more arguments) error {
...
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
...
err := json.NewDecoder(resp.Body).Decode(pv)
return err
}
Call this modified function like this:
var result struct {
Data []struct{
Group, ID, Name string
Released bool
Version int
}
}
err := request(&result, "some/path", ... more arguments )
if err != nil {
// handle error
}
for _, d := range result.Data {
if !d.Released {
continue
}
fmt.Println(d.Group, d.ID, d.Name, d.Version)
... process d
}
The docs (https://github.com/mitchellh/mapstructure) mention the benefits of mapstructure for decoding
{
"type": "person",
"name": "Mitchell"
}
I'm looking for an example that show the decoding process, e.g. to unmarshal either a Person or Pet class
That library only maps maps to structures. It does not do type-specific decoding. Quoting from its doc:
Perhaps we can't populate a specific structure without first reading the "type" field from the JSON. We could always do two passes over the decoding of the JSON (reading the "type" first, and the rest later). However, it is much simpler to just decode this into a map[string]interface{} structure, read the "type" key, then use something like this library to decode it into the proper structure.
All it offers is that you only have to parse the JSON source once, into a map, then you can decide yourself what type you need to map or unmarshal into. Then you can use the already constructed map to fill the type you need.
See this example:
type Person struct {
Name string `json:"name"`
}
type Credentials struct {
User string `json:"user"`
Password string `json:"password"`
}
func main() {
for _, src := range []string{srcPerson, srcCredentials} {
var m map[string]interface{}
if err := json.Unmarshal([]byte(src), &m); err != nil {
panic(err)
}
switch m["type"] {
case "person":
var p Person
if err := mapstructure.Decode(m, &p); err != nil {
panic(err)
}
fmt.Printf("%T %+v\n", p, p)
case "credentials":
var c Credentials
if err := mapstructure.Decode(m, &c); err != nil {
panic(err)
}
fmt.Printf("%T %+v\n", c, c)
}
}
}
const srcPerson = `{
"type": "person",
"name": "Mitchell"
}`
const srcCredentials = `{
"type": "credentials",
"user": "bob",
"password": "secret"
}`
Output:
main.Person {Name:Mitchell}
main.Credentials {User:bob Password:secret}
I have this code:
type Response struct {
ID string `json:"id"`
Tags Tags `json:"tags,omitempty"`
}
type Tags struct {
Geo []string `json:"geo,omitempty"`
Keyword []string `json:"keyword,omitempty"`
Storm []string `json:"storm,omitempty"`
}
func (t *Tags) UnmarshalJSON(b []byte) (err error) {
str := string(b)
if str == "" {
t = &Tags{}
return nil
}
err = json.Unmarshal(b, t)
if err != nil {
return err
}
return nil
}
Now, my JSON response looks like this:
[{
"id": "/cms/v4/assets/en_US",
"doc": [{
"id": "af02b41d-c2c5-48ec-9dbc-ceed693bdbac",
"tags": {
"geo": [
"DMA:US.740:US"
]
}
},
{
"id": "6a90d9ed-7978-4c18-8e36-c01cf4260492",
"tags": ""
},
{
"id": "32cfd045-98ac-408c-b464-c74e02466339",
"tags": {
"storm": [
"HARVEY - AL092017"
],
"keyword": [
"hurrcane",
"wunderground"
]
}
}
]
}]
Preferably, I'd change the JSON response to be done correctly, but I cannot. Unmarshaling continues to error out (goroutine stack exceeds 1000000000-byte limit). Preferably, I'd rather do this using easyjson or ffjson but doubt it is possible. Suggestions?
Your UnmarshalJSON function calls itself recursively, which will cause the stack to explode in size.
func (t *Tags) UnmarshalJSON(b []byte) (err error) {
str := string(b)
if str == "" {
t = &Tags{}
return nil
}
err = json.Unmarshal(b, t) <--- here it calls itself again
if err != nil {
return err
}
return nil
}
If you have a reason to call json.Unmarshal from within a UnmarshalJSON function, it must be on a different type. A common way to do this is to use a local alias:
type tagsAlias Tags
var ta = &tagsAlias
err = json.Unmarshal(b, ta)
if err != nil {
return err
}
*t = Tags(ta)
Also note that t = &Tags{} does nothing in your function; it assigns a new value to t, but that value is lost as soon as the function exits. If you really want to assign to t, you need *t; but you also don't need that at all, unless you're trying to unsset a previously set instance of *Tags.
I have a map that uses string for both key and value. I have an array of keys that specifies the order of the values of the map.
I want to serialize that map to a JSON, but keeping the order defined on the array.
There is a sample code here: http://play.golang.org/p/A52GTDY6Wx
I want to serialize it as:
{
"name": "John",
"age": "20"
}
But if I serialize the map directly, the keys are ordered alphabetically:
{
"age": "20",
"name": "John"
}
I can serialize it as an array of maps, thus keeping the order, however that generates a lot of undesired characters:
[
{
"name": "John"
},
{
"age": "20"
}
]
In my real code I need to serialize the results of a database query which is specified in a text file, and I need to maintain the column order. I cannot use structs because the columns are not known at compile time.
EDIT: I don't need to read the JSON later in the specified order. The generated JSON is meant to be read by people, so I just want it to be as humanly readable as possible.
I could use a custom format but JSON suits me perfectly for this.
Thanks!
You need to implement the json.Marshaler interface on a custom type. This has the advantage of playing well within other struct types.
Sorry, you're always going to have to write a little bit of JSON encoding code.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type KeyVal struct {
Key string
Val interface{}
}
// Define an ordered map
type OrderedMap []KeyVal
// Implement the json.Marshaler interface
func (omap OrderedMap) MarshalJSON() ([]byte, error) {
var buf bytes.Buffer
buf.WriteString("{")
for i, kv := range omap {
if i != 0 {
buf.WriteString(",")
}
// marshal key
key, err := json.Marshal(kv.Key)
if err != nil {
return nil, err
}
buf.Write(key)
buf.WriteString(":")
// marshal value
val, err := json.Marshal(kv.Val)
if err != nil {
return nil, err
}
buf.Write(val)
}
buf.WriteString("}")
return buf.Bytes(), nil
}
func main() {
dict := map[string]interface{}{
"orderedMap": OrderedMap{
{"name", "John"},
{"age", 20},
},
}
dump, err := json.Marshal(dict)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", dump)
}
Outputs
{"orderedMap":{"name":"John","age":20}}
For that specific requirement you really don't need to use json.Marshal at all, you can simply implement your own function like this:
type OrderedMap map[string]string
func (om OrderedMap) ToJson(order ...string) string {
buf := &bytes.Buffer{}
buf.Write([]byte{'{', '\n'})
l := len(order)
for i, k := range order {
fmt.Fprintf(buf, "\t\"%s\": \"%v\"", k, om[k])
if i < l-1 {
buf.WriteByte(',')
}
buf.WriteByte('\n')
}
buf.Write([]byte{'}', '\n'})
return buf.String()
}
func main() {
om := OrderedMap{
"age": "20",
"name": "John",
}
fmt.Println(om.ToJson("name", "age"))
}
Probably the easiest solution: https://github.com/iancoleman/orderedmap
Although it might be slow as it's mentioned here
Here is a MapSlice implementation similar to what YAML v2 offers. It can do both Marshal and Unmarshal.
https://github.com/mickep76/mapslice-json