Golang - How do you decode json array and get the root property - json

I can't figure out how to decode this JSON in Go. The map returns nil. Unmarshal works from memory, but eventually I might need a stream. Also, I need to get Foo, Bar and Baz key names. Not sure about that one.
JSON:
{
"Foo" : {"Message" : "Hello World 1", "Count" : 1},
"Bar" : {"Message" : "Hello World 2", "Count" : 0},
"Baz" : {"Message" : "Hello World 3", "Count" : 1}
}
Code:
package main
import (
"encoding/json"
"fmt"
"os"
)
type Collection struct {
FooBar map[string]Data
}
type Data struct {
Message string `json:"Message"`
Count int `json:"Count"`
}
func main() {
//will be http
file, err := os.Open("stream.json")
if err != nil {
panic(err)
}
decoder := json.NewDecoder(file)
var c Collection
err = decoder.Decode(&c)
if err != nil {
panic(err)
}
for key, value := range c.FooBar {
fmt.Println("Key:", key, "Value:", value)
}
//returns empty map
fmt.Println(c.FooBar)
}

You don't need a top-level struct, decode directly into a map:
err = decoder.Decode(&c.FooBar)
Or, just remove the struct:
type Collection map[string]Data
With your top-level struct, the implied format is:
{
"FooBar": {
"Foo" : {"Message" : "Hello World 1", "Count" : 1},
"Bar" : {"Message" : "Hello World 2", "Count" : 0},
"Baz" : {"Message" : "Hello World 3", "Count" : 1}
}
}

Related

Trouble mapping json to golang struct

I have a json stream as follows ...
[
{
"page": 1,
"pages": 7,
"per_page": "2000",
"total": 13200
},
[
{
"indicator": {
"id": "SP.POP.TOTL",
"value": "Population, total"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"value": null,
"decimal": "0",
"date": "2019"
},
{
"indicator": {
"id": "SP.POP.TOTL",
"value": "Population, total"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"value": "419790588",
"decimal": "0",
"date": "2018"
},
...
]
]
And I'm trying to decode it ... so I have the following struct ... but I keep getting
"cannot unmarshal array into Go value of type struct { P struct ... "
type Message []struct {
P struct {
Page int
}
V []struct {
Indicator struct {
Id string
Value string
}
Country struct {
Value string
}
Value string
Decimal string
Date string
}
}
My struct looks to match the json ... but obviously not! Any ideas?
Since your JSON array have two different types first unmarshal them into a slice of json.RawMessage which is []byte as underlying type so that we can unmarshal again JSON array data.
So unmarshal data for P and V struct type using index directly (predict) or detect if object(starting with '{') then unmarshal into P and array(starting with '[') then unmarshal into V. Now prepare your Message using those data.
type Message struct {
PageData P
ValData []V
}
type P struct {
Page int
}
type V struct {
Indicator struct {
Id string
Value string
}
Country struct {
Value string
}
Value string
Decimal string
Date string
}
func main() {
var rawdata []json.RawMessage
json.Unmarshal([]byte(jsonData), &rawdata)
var pageData P
json.Unmarshal(rawdata[0], &pageData)
var valData []V
json.Unmarshal(rawdata[1], &valData)
res := Message{pageData, valData}
fmt.Println(res)
}
var jsonData = `[...]` //your json data
Full code in Go Playground
As poWar said, the JSON you actually have is a list of objects whose types do not conform to each other. You must therefore unmarshal into something capable of holding different object types, such as interface{} or—since there is an outer array—[]interface{}.
You can also, if you like, decode into a []json.RawMessage. The underlying json.RawMessage itself has underlying type []byte so that it's basically the undecoded "inner" JSON. In at least some cases this is going to be more work than just decoding directly to []interface{} and checking each resulting interface, but you can, if you wish, decode to struct once you have the JSON separated out. For instance:
func main() {
var x []json.RawMessage
err := json.Unmarshal(input, &x)
if err != nil {
fmt.Printf("err = %v\n", err)
return
}
if len(x) != 2 {
fmt.Println("unexpected input")
return
}
var page struct {
Page int
}
err = json.Unmarshal(x[0], &page)
if err != nil {
fmt.Printf("unable to unmarshal page part: %v\n", err)
return
}
fmt.Printf("page = %d\n", page.Page)
// ...
}
Here on the Go Playground is a more complete example. See also Eklavya's answer.
Looking at your struct, your corresponding JSON should look something like this.
[
{
"P": {"page": 1},
"V": [
{
"Indicator": {"Id": ...},
"Country": {"Value":""},
"Value": "",
...
}
]
},
...
]
The JSON structure you are trying to Unmarshal looks like a list of objects where each object is not of the same type. You can start unmarshalling them into interfaces and defining each interface based on the object being unmarhsalled.
package main
import (
"encoding/json"
"log"
)
type Message []interface{}
func main() {
data := `[{"page":1,"pages":7,"per_page":"2000","total":13200},[{"indicator":{"id":"SP.POP.TOTL","value":"Population, total"},"country":{"id":"1A","value":"Arab World"},"value":null,"decimal":"0","date":"2019"},{"indicator":{"id":"SP.POP.TOTL","value":"Population, total"},"country":{"id":"1A","value":"Arab World"},"value":"419790588","decimal":"0","date":"2018"}]]`
var m Message
if err := json.Unmarshal([]byte(data), &m); err != nil {
log.Fatalf("could not unmarshal")
}
log.Printf("message: %v", m)
}
Output:
message: [map[page:1 pages:7 per_page:2000 total:13200] [map[country:map[id:1A value:Arab World] date:2019 decimal:0 indicator:map[id:SP.POP.TOTL value:Population, total] value:<nil>] map[country:map[id:1A value:Arab World] date:2018 decimal:0 indicator:map[id:SP.POP.TOTL value:Population, total] value:419790588]]]
[Edit]: Ideally you should change your JSON to be structured better for unmarshalling. If you do not have control on it, then your corresponding Go structure is just embedded maps of string to interfaces, which you will have to manually type cast and access.

Unable to decode int field from JSON

My goal is to read JSON file. Although the following program reads the file fine, the id field value is incorrect. id field value is 0 for all the objects read.
Example JSON file contents:
{
"users" : [
{
"id" : "1",
"name" : "Matt Hayden"
},
{
"id" : "2",
"name" : "David Warner"
},
]
}
JSON file reader program in Golang
type Schema struct {
Id int `json:"id"`
Name string `json:"name"`
}
func reader() {
fileName := "./input.json"
jsonFile, fileErr := os.Open(fileName)
if fileErr != nil {
_ = errors.Errorf("failed to open file %s : %v", fileName, fileErr)
}
defer jsonFile.Close()
bytesRead, readErr := ioutil.ReadAll(jsonFile)
if readErr != nil {
_ = errors.Errorf("failed to read json file %s : %v", fileName, readErr)
}
var sch Schema
unMarshalErr := json.Unmarshal(bytesRead, &sch)
if unMarshalErr != nil {
_ = errors.Errorf("failed to unmarshal JSON bytes : %v", unMarshalErr)
}
fmt.Printf("%+v", sch)
}
Actual output:
{Users:[{Id:0 Name:Matt Hayden} {Id:0 Name:David Warner}]}
Expected output:
{Users:[{Id:1 Name:Matt Hayden} {Id:2 Name:David Warner}]}
You can add an extra type in your JSON struct tag. Take a look at the Marshal documentation here.
Example: https://play.golang.org/p/VqZu9SrnpW4
package main
import (
"encoding/json"
"fmt"
"reflect"
)
type Response struct {
Users []User `json:"users"`
}
type User struct {
ID int `json:"id,string"`
Name string `json:"name"`
}
func main() {
r := &Response{}
err := json.Unmarshal([]byte(`{
"users" : [
{
"id" : "1",
"name" : "Matt Hayden"
},
{
"id" : "2",
"name" : "David Warner"
}
]
}`), &r)
if err != nil {
panic(err)
}
fmt.Printf("%#v \n", r)
fmt.Printf("Type of ID is %s \n", reflect.TypeOf(r.Users[0].ID)) // int
}

Parsing Json using Golang

I am trying to parse json using go, but I am not sure about the data structure I am building. The output is always an empty object.
data structure :
{
"Catalog": {
"Name": "TypeMime",
"Version": "1.0",
"ObjectType": {
"Area": 2,
"Service": 2,
"Version": 1,
"Number": 117
},
"Items": [
{
"ItemNamespace": "application",
"Name": "binhex-bin",
"Oid": null,
"Uid": 2201,
"DocMandatoryProperties": [],
"DocOptionalProperties": [
"Subsystem",
"Label",
"Description"
],
"DocVersionProperties": [],
"DefaultAction": null,
"Extensions": [
".bin"
],
"Confidentiality": [
"confidentiality/Public"
],
"Converter": [],
"Editor": [
"gedit"
],
"DiffEditor": [
"kompare"
],
"Icon": "",
"IdentificationPattern": [
"^\\S+.bin$"
]
},
{other items}
}
type ObjectType struct {
Area int `json:"Area"`
Service int `json:"Service"`
Version int `json:"Version"`
Number int `json:"Number"`
}
type Catalog struct {
Name string `json:"Name"`
Version string `json:"Version"`
ObjectType ObjectType `json:"ObjectType"`
Items []Item `json:"Items"`
}
type Item struct {
ItemNamespace string `json:"ItemNamespaceItems"`
Name string `json:"Name"`
Oid int `json:"Oid"`
Uid int `json:"Uid"`
DocMandatoryProperties []string `json:"DocMandatoryProperties"`
DocOptionalProperties []string `json:"DocOptionalProperties"`
DocVersionProperties []string `json:"DocVersionProperties"`
DefaultAction string `json:"DefaultAction"`
Extensions []string `json:"Extensions"`
Confidentiality []string `json:"Confidentiality"`
Converter []string `json:"Converter"`
Editor []string `json:"Editor"`
DiffEditor []string `json:"DiffEditor"`
Icon string `json:"Icon"`
IdentificationPattern []string `json:"IdentificationPattern"`
}
var catalog Catalog
// Open our jsonFile
jsonFile, err := os.Open("path.json")
// if we os.Open returns an error then handle it
if err != nil {
fmt.Println(err)
}
fmt.Println("Successfully Opened json")
// defer the closing of our jsonFile so that we can parse it later on
defer jsonFile.Close()
byteValue, err := ioutil.ReadAll(jsonFile)
if err != nil {
fmt.Println("error input file :",err)
}
err = json.Unmarshal(byteValue, &catalog)
if err != nil {
fmt.Println("error input file :",err)
}
fmt.Println("catalog from json : ",catalog)
I am getting this output : { {0 0 0 0} []}
So I am thinking, maybe the structures that I have built are not correct or Go cannot handle "null" values. The JSON file is correctly opened and converted to bytes, I am not getting any errors, just not what I was expecting i.e. the json parse in my Catalog variable.
Any suggestions ?
err = json.Unmarshal(byteValue, &catalog)
You tell go that you're about to parse a Catalog, but then do something else. What you actually do is give it an (unnamed) object that has a "Catalog" key with corresponding value. Go can't find any of catalog keys in this object and that's why you get all empty fields.

Unmarshalling complex json in Go

So I am trying to fetch the analytics of an app by pinging and endpoint. I make the GET request which is successfull (no errors there) but I am unable to decode the JSON
I need to to decode the following json into structs
{
"noResultSearches": {
"results": [
{
"count": 1,
"key": "\"note 9\""
},
{
"count": 1,
"key": "nokia"
}
]
},
"popularSearches": {
"results": [
{
"count": 4,
"key": "6"
},
{
"count": 2,
"key": "\"note 9\""
},
{
"count": 1,
"key": "nokia"
}
]
},
"searchVolume": {
"results": [
{
"count": 7,
"key": 1537401600000,
"key_as_string": "2018/09/20 00:00:00"
}
]
}
}
For which I am using the following structs
type analyticsResults struct {
Count int `json:"count"`
Key string `json:"key"`
}
type analyticsVolumeResults struct {
Count int `json:"count"`
Key int64 `json:"key"`
DateAsStr string `json:"key_as_string"`
}
type analyticsPopularSearches struct {
Results []analyticsResults `json:"results"`
}
type analyticsNoResultSearches struct {
Results []analyticsResults `json:"results"`
}
type analyticsSearchVolume struct {
Results []analyticsVolumeResults `json:"results"`
}
type overviewAnalyticsBody struct {
NoResultSearches analyticsNoResultSearches `json:"noResultSearches"`
PopularSearches analyticsPopularSearches `json:"popularSearches"`
SearchVolume analyticsSearchVolume `json:"searchVolume"`
}
I make a GET request to an endpoint and then use the response body to decode the json but I get an error. Following is a part of the code that stays in my ShowAnalytics function
func ShowAppAnalytics(app string) error {
spinner.StartText("Fetching app analytics")
defer spinner.Stop()
fmt.Println()
req, err := http.NewRequest("GET", "<some-endpoint>", nil)
if err != nil {
return err
}
resp, err := session.SendRequest(req)
if err != nil {
return err
}
spinner.Stop()
var res overviewAnalyticsBody
dec := json.NewDecoder(resp.Body)
err = dec.Decode(&res)
if err != nil {
return err
}
fmt.Println(res)
return nil
}
json: cannot unmarshal array into Go struct field
overviewAnalyticsBody.noResultSearches of type
app.analyticsNoResultSearches
What am I doing wrong here? Why do I get this error?
EDIT: After you edited, your current code works as-is. Check it out here: Go Playground.
Original answer follows.
There is some inconsistency between the code you posted and the error you get.
I tried it on the Go Playground (here's your version), and I get the following error:
json: cannot unmarshal number into Go struct field analyticsVolumeResults.key of type string
We get this error because in the JSON searchVolume.results.key is a number:
"key": 1537401600000,
And you used string in the Go model:
Key string `json:"key"`
If we change it to int64:
Key int64 `json:"key"`
It works, and prints (try it on the Go Playground):
{{[{1 "note 9"} {1 nokia}]} {[{4 6} {2 "note 9"} {1 nokia}]} {[{7 1537401600000 2018/09/20 00:00:00}]}}
If that key may sometimes be a number and sometimes a string, you may also use json.Number in the Go model:
Key json.Number `json:"key"`

Merge a dynamic data structure in Go

I have this incoming payload, that I cannot change.
{
"source": "some random source",
"table": "hosts_table",
"data": [
["address", "id", "services_with_info"],
["0.0.0.1", 1111, [
["service_3", "is very cool", 1],
["service_4", "is very cool", 2]
]
],
["0.0.0.2", 2222, [
["service_3", "is very cool", 3],
["service_4", "is very cool", 4]
]
]
]}
i need to take the first index of "data" and create a new JSON object, that looks like this...
"data": [{
"address": "0.0.0.1",
"id": 1111,
"services_with_info":[
{
"service_name": "service_1",
"service_message": "is very cool",
"service_id": 1
},
{...}
]},
{...}]
and then build an []Host's from it the data structure is 5k "hosts" long. I was able to map this to a struct, but need to get it into this format first. I understand how to unmarshal the JSON, but only if I can convert the payload to the above.
You can use json.Unmarshal for this and parse data with your conditions. I'm just doing it for "data" and you can do same for "services_with_info"
b := []byte(`{
"source": "some random source",
"table": "hosts_table",
"data": [
["address", "id", "services_with_info"],
["0.0.0.1", 1111, [
["service_3", "is very cool", 1],
["service_4", "is very cool", 2]
]
],
["0.0.0.2", 2222, [
["service_3", "is very cool", 3],
["service_4", "is very cool", 4]
]
]
]}`)
var f interface{}
err := json.Unmarshal(b, &f)
if err != nil {
fmt.Println(err)
return
}
m := f.(map[string]interface{})
result := make(map[string]interface{})
results := make(map[string][]map[string]interface{})
for k, v := range m {
if k == "data" {
s := v.([]interface{})
header := make([]interface{}, 3)
for i, u := range s {
if i == 0 {
header = u.([]interface{})
} else {
row := u.([]interface{})
for j, k := range header {
result[k.(string)] = row[j]
}
results["data"] = append(results["data"], result)
}
}
}
}
fmt.Println(results)
here "results" is "data" as required.
I'm not sure if I understood what you wants.
May be some thing like this?
Probably it needs some work, like make slice of pointers to structs instead of slice of structs to prevent allocation and copy, error handling, more custom logic to convert values, anonymize/incapsulate private structs used in the middle of conversion, add json tags to those structures etc.
I create custom Unmarshaller for Data field on IncomingPaylod: parsing expected data, converting it to []MyData and updating Data field with it.
I created custom Unmarshallers for expected_data and expected_services_with_info because we expect it as array of values (3 values: string, int and [array of string, int(?), int]), but I want to convert it to nice structs. If you dont like it, you can delete it, Unmarshal expected data to []interface{} and work with it like []interface{}{string, int, []interface{}{string, int, int} }. Easy to get it wrong, so i like structs more, its easier to read and maintain and refactor (i think there are more fields in you app).
https://play.golang.org/p/xHTvyhecra
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type IncomingPayload struct {
Source string `json:"source"`
Table string `json:"table"`
Data MyDataSlice `json:"data"`
}
type MyDataSlice []MyData
type MyData struct {
Address string `json:"address"`
ID string `json:"id"`
Services_with_info []MyServiceWithInfo `json:"services_with_info"`
}
type MyServiceWithInfo struct {
ServiceName string `json:"service_name"`
ServiceMessage string `json:"service_message"`
ServiceID int `json:"service_id"`
}
type expected_data struct {
IP string
ID int
Info []expected_services_with_info
}
type expected_services_with_info struct {
Name string
Desc string
ID int
}
func (ed *expected_data) UnmarshalJSON(buf []byte) error {
tmp := []interface{}{&ed.IP, &ed.ID, &ed.Info}
// converts ["address", "id", "services_with_info"] into a struct
// will unmarshall "services_with_info" (ed.Info) with *expected_services_with_info.UnmarshalJSON
json.Unmarshal(buf, &tmp)
return nil
}
func (es *expected_services_with_info) UnmarshalJSON(buf []byte) error {
tmp := []interface{}{&es.Name, &es.Desc, &es.ID}
// converts ["service_3", "is very cool", 1] into a struct
json.Unmarshal(buf, &tmp)
return nil
}
func (md *MyDataSlice) UnmarshalJSON(p []byte) error {
var incoming_data_slice []expected_data
json.Unmarshal(p, &incoming_data_slice)
//fmt.Println("incoming", incoming_data_slice)
//transform incoming_data_slice to your needs using your data type
for i := range incoming_data_slice {
my_data := MyData{
Address: incoming_data_slice[i].IP, //copy
ID: strconv.Itoa(incoming_data_slice[i].ID), //some transformation
//nil slice is totally fine, but if you wish you can do
//Data: make(MyDataSlice, len(incoming_data_slice)),
}
//not sure what would be best: "i := range data" or "_, v := range data" (second one makes a copy? and causes allocation)
for j := range incoming_data_slice[i].Info {
tmp := MyServiceWithInfo{
ServiceName: incoming_data_slice[i].Info[j].Name,
ServiceMessage: incoming_data_slice[i].Info[j].Desc,
ServiceID: incoming_data_slice[i].Info[j].ID,
}
my_data.Services_with_info = append(my_data.Services_with_info, tmp)
}
//and populate
*md = append(*md, my_data)
}
return nil
}
func main() {
test_json := `{
"source": "some random source",
"table": "hosts_table",
"data": [
["address", "id", "services_with_info"],
["0.0.0.1", 1111, [
["service_3", "is very cool", 1],
["service_4", "is very cool", 2]
]
],
["0.0.0.2", 2222, [
["service_3", "is very cool", 3],
["service_4", "is very cool", 4]
]
]
]}`
var payload IncomingPayload
json.Unmarshal([]byte(test_json), &payload)
fmt.Println("payload", payload)
buf, _ := json.MarshalIndent(payload, "", "\t")
fmt.Println(string(buf))
}