Golang Json marshalling - json

type Status struct {
slug string `json:"slug"`
}
type Instance struct {
Slug string `json:"slug"`
Stat map[string]*Status `json:"status"`
}
This Json response is recieved for API call:
[ {
"status": {
"stop": false,
"traffic": true,
"label": null,
"dead": true,
"slug": "up",
},
"slug": "instances_raw",
"started_at": "15120736198",
"replacement": null
},
{
"status": {
"stop": false,
"traffic": true,
"label": null,
"dead": true,
"slug": "down",
},
"slug": "instance_raw2",
"started_at": "1512073194",
"replacement": null
}
]
I am trying to marshall json into above struct but running into issue:
instances := make([]Instance, 0)
res := api call return above json
body, _ := ioutil.ReadAll(res.Body)
json.Unmarshal(body, &instances)
fmt.Println("I am struct %s",instances)
It is marshalling into:
I am struct %s [{ map[stop:0xc42018e1b0 dead:0xc42018e1e0 label:<nil> running:0xc42018e220 down:0xc42018e150 traffic:0xc42018e180]}]
Can someone help me figure out why it is not marshalling as I am expecting?
Expected marshalling:
[{instances_raw map[slug:up]} {instances_raw2 map[slug:down]}]

The structs do not match the structure of the data. Perhaps you wanted this:
type Status struct {
Slug string `json:"slug"`
}
type Instance struct {
Slug string `json:"slug"`
Stat Status `json:"status"`
}
Output: [{instances_raw {up}} {instance_raw2 {down}}]
run it on the plaground
or this:
type Instance struct {
Slug string `json:"slug"`
Stat map[string]interface{} `json:"status"`
}
Output: [{instances_raw map[label: dead:true slug:up stop:false traffic:true]} {instance_raw2 map[slug:down stop:false traffic:true label: dead:true]}]
run it on the playground
Always check errors. The example JSON above is not valid and the json.Unmarshal function reports this error.

Related

Parse AWS ec2 DescribeInstanceStatusOutput JSON response using Golang

Hi all I am a beginner trying to learn Go. I am trying to get the status of my EC2 instance using AWS SDK with Golang. I was successfully able to get a JSON response for my instance. I am however, having issues trying to parse the JSON response to get the instance state.
The following is how I get the JSON response:
result, err := ec2Svc.DescribeInstanceStatus(input)
where input is of type ec2.DescribeInstanceStatusInput
The following is the JSON response I get:
{
"InstanceStatuses": [
{
"AvailabilityZone": "ap-southeast-2b",
"Events": null,
"InstanceId": "[VALIDIMAGEID]",
"InstanceState": {
"Code": 16,
"Name": "running"
},
"InstanceStatus": {
"Details": [
{
"ImpairedSince": null,
"Name": "reachability",
"Status": "passed"
}
],
"Status": "ok"
},
"OutpostArn": null,
"SystemStatus": {
"Details": [
{
"ImpairedSince": null,
"Name": "reachability",
"Status": "passed"
}
],
"Status": "ok"
}
}
],
"NextToken": null
}
I then parse the result which is of type DescribeInstanceStatusOutput to String and then pass it to the function parseJson where I attempt to parse the JSON.
result, err := ec2Svc.DescribeInstanceStatus(input)
if err != nil {
fmt.Println("Error", err)
} else {
parseJson(result.String())
//fmt.Printf("%v\n", result)
}
func parseJson(ec2StatusDescription string){
//var x string = ec2StatusDescription.String()
//fmt.Print(ec2StatusDescription)
data := []byte(ec2StatusDescription)
var instanceOutputs InstanceOutput
json.Unmarshal(data, &instanceOutputs)
fmt.Print(instanceOutputs.InstanceStatuses[0].InstanceState.Name)
}
type InstanceOutput struct {
InstanceStatuses [] InstanceStatuses
NextToken *string
}
type InstanceStatuses struct {
AvailabilityZone string
Events *string
InstanceId string
InstanceState InstanceState
InstanceStatus InstanceStatus
OutpostArn *string
SystemStatus SystemStatus
}
type InstanceState struct {
Code int
Name string
}
type InstanceStatus struct {
Details [] InstanceDetails
Status string
}
type InstanceDetails struct {
ImpairedSince *string
Name string
Status string
}
type SystemStatus struct {
Details [] InstanceDetails
Status string
}
I am trying to print the instance state but I keep getting index out of range, and this is because the JSON is not being parsed properly. What am i missing here that's causing the json to not parse correctly? Or am I getting it completely wrong in what I'm doing here? Any help would be highly appreciated.

Why can this data not be properly unmarshalled into my object model?

I have a (non)working example here: https://play.golang.org/p/qaYhKvJ65J3
I'm not sure why the following data:
alertData := `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
Can't be unmarshalled into the following object model:
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
When I do the following:
condition := Condition{}
err := json.Unmarshal([]byte(alertData), &condition)
if err != nil {
panic(err)
}
fmt.Printf("\n\n json object:::: %+v", condition)
I just get: json object:::: {Evaluator:{Params:[] Type:} Operator:{Type:} Query:{Params:[]} Reducer:{Params:[] Type:} Type:}
Ideally I'd be able to parse it into something like type Conditions []struct{ } but I'm not sure if you can define models as lists?
It looks like you are trying to access the "conditions" property nested under the root "Settings" property. As such, you need to define that root-level type and enough fields to tell the unmarshaler how to find your target property. As such, you would just need to create a new "AlertData" type with the necessary "Settings/conditions" fields.
For example (Go Playground):
type AlertData struct {
Settings struct {
Conditions []Condition `json:"conditions"`
}
}
func main() {
alert := AlertData{}
err := json.Unmarshal([]byte(alertData), &alert)
if err != nil {
panic(err)
}
fmt.Printf("OK: conditions=%#v\n", alert.Settings.Conditions)
// OK: conditions=[]main.Condition{main.Condition{Evaluator:struct { Params []int "json:\"params\""; Type string "json:\"type\"" }{Params:[]int{10000}, Type:"gt"}, Operator:struct { Type string "json:\"type\"" }{Type:"and"}, Query:struct { Params []string "json:\"params\"" }{Params:[]string{"C", "5m", "now"}}, Reducer:struct { Params []interface {} "json:\"params\""; Type string "json:\"type\"" }{Params:[]interface {}{}, Type:"avg"}, Type:"query"}}
}
Note that the printed listing includes so much type information because the "Condition" type uses anonymous structs as field types. If you were to extract them into named structs it will be easier to work with the data, e.g.:
type Condition struct {
Evaluator Evaluator `json:"evaluator"`
Operator Operator `json:"operator"`
// ...
}
type Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
}
type Operator struct {
Type string `json:"type"`
}
//...
// OK: conditions=[]main.Condition{
// main.Condition{
// Evaluator:main.Evaluator{Params:[]int{10000}, Type:"gt"},
// Operator:main.Operator{Type:"and"},
// Query:main.Query{Params:[]string{"C", "5m", "now"}},
// Reducer:main.Reducer{Params:[]interface {}{}, Type:"avg"},
// Type:"query",
// },
// }
Go Playground example here...
Maerics explanation is correct, here is an alternative approach which wraps access around struct methods, the data structure is also fully defined. If you're new to Go it's good to get handle on creating the data structures yourself, but here is a handy utility for helping create structs from valid JSON
https://mholt.github.io/json-to-go/
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
type Data struct {
ID int `json:"Id"`
Version int `json:"Version"`
OrgID int `json:"OrgId"`
DashboardID int `json:"DashboardId"`
PanelID int `json:"PanelId"`
Name string `json:"Name"`
Message string `json:"Message"`
Severity string `json:"Severity"`
State string `json:"State"`
Handler int `json:"Handler"`
Silenced bool `json:"Silenced"`
ExecutionError string `json:"ExecutionError"`
Frequency int `json:"Frequency"`
EvalData interface{} `json:"EvalData"`
NewStateDate time.Time `json:"NewStateDate"`
PrevStateDate time.Time `json:"PrevStateDate"`
StateChanges int `json:"StateChanges"`
Created time.Time `json:"Created"`
Updated time.Time `json:"Updated"`
Settings struct {
Conditions []Condition `json:"conditions"`
ExecutionErrorState string `json:"executionErrorState"`
Frequency string `json:"frequency"`
Handler int `json:"handler"`
Name string `json:"name"`
NoDataState string `json:"noDataState"`
Notifications []interface{} `json:"notifications"`
} `json:"Settings"`
}
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
DatasourceID int `json:"datasourceId"`
Model struct {
Hide bool `json:"hide"`
RefCount int `json:"refCount"`
RefID string `json:"refId"`
TextEditor bool `json:"textEditor"`
} `json:"model"`
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
func (d Data) GetFirstCondition() (Condition, error) {
if len(d.Settings.Conditions) > 0 {
return d.Settings.Conditions[0], nil
}
return Condition{}, fmt.Errorf("no conditions found")
}
func (d Data) GetConditionByIndex(index uint) (Condition, error) {
if len(d.Settings.Conditions) == 0 {
return Condition{}, fmt.Errorf("no conditions found")
}
if int(index) > len(d.Settings.Conditions)-1 {
return Condition{}, fmt.Errorf("index out of bounds")
}
return d.Settings.Conditions[index], nil
}
var alertData = `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
func main() {
var res Data
err := json.Unmarshal([]byte(alertData), &res)
if err != nil {
log.Fatal(err)
}
fmt.Println(res.GetFirstCondition())
fmt.Println(res.GetConditionByIndex(0))
// should fail :-)
fmt.Println(res.GetConditionByIndex(1))
}

Dynamic JSON struct, API result golang

I have to make two HTTP API calls in GoLang, the first API call returns this json response:
{
"status": 200,
"msg": "OK",
"result": {
"id": "24",
"folderid": "4248"
}
}
My json struct for first response is setup like this:
type One struct {
Status int `json:"status"`
Msg string `json:"msg"`
Result struct {
ID string `json:"id"`
Folderid string `json:"folderid"`
} `json:"result"`
}
The second call is where the problem is. As you can see the first API call returns a result -> id. This ID should be the name for the beginning of my second struct, but I can't seem how to make it dynamic or put a result as my structure name. This ID (24) will always change based on the first API call. I have no way currently to parse the second call's JSON and setup my struct. On the second API call I want to access the remoteurl/status.
Second call result (I can not parse):
{
"status": 200,
"msg": "OK",
"result": {
24: ** THIS IS DYNAMIC** {
"id": 24,
"remoteurl": "http://proof.ovh.net/files/100Mio.dat",
"status": "new",
"bytes_loaded": null,
"bytes_total": null,
"folderid": "4248",
"added": "2015-02-21 09:20:26",
"last_update": "2015-02-21 09:20:26",
"extid": false,
"url": false
}
}
}
Does anyone know how to setup my struct or go about this. I am a new programmer and go and have worked on this for 4 days. And decided to ask for some help, since I am in school and have normal homework.
Found that using JSON-to-GO helped solve future problems, will create the structs and other necessities based off a JSON content.
{
"status": 200,
"msg": "OK",
"result": {
24: {
"id": 24,
"remoteurl": "http://proof.ovh.net/files/100Mio.dat",
"status": "new",
"bytes_loaded": null,
"bytes_total": null,
"folderid": "4248",
"added": "2015-02-21 09:20:26",
"last_update": "2015-02-21 09:20:26",
"extid": false,
"url": false
}
}
}
Is not value JSON. You MUST mean the JSON i posted below, if you want to check yourself, copy your version of the JSON into any JSON validator;
https://jsonlint.com/
https://jsoneditoronline.org/
https://jsonformatter.curiousconcept.com/
Also view the thread linked below.. if the API truly is returning what you claim it returns then the API has a bug in it
Why JSON allows only string to be a key?
{
"status": 200,
"msg": "OK",
"result": {
"24": {
"id": 24,
"remoteurl": "http://proof.ovh.net/files/100Mio.dat",
"status": "new",
"bytes_loaded": null,
"bytes_total": null,
"folderid": "4248",
"added": "2015-02-21 09:20:26",
"last_update": "2015-02-21 09:20:26",
"extid": false,
"url": false
}
}
}
Here is some example code that uses a map to a struct what solves the dynamic response of the second response
package main
import (
"encoding/json"
"fmt"
"log"
)
var res1 = `{
"status": 200,
"msg": "OK",
"result": {
"id": "24",
"folderid": "4248"
}
}`
var res2 = `{
"status": 200,
"msg": "OK",
"result": {
"24": {
"id": 24,
"remoteurl": "http://proof.ovh.net/files/100Mio.dat",
"status": "new",
"bytes_loaded": null,
"bytes_total": null,
"folderid": "4248",
"added": "2015-02-21 09:20:26",
"last_update": "2015-02-21 09:20:26",
"extid": false,
"url": false
}
}
}
`
type One struct {
Status int `json:"status"`
Msg string `json:"msg"`
Result struct {
ID string `json:"id"`
Folderid string `json:"folderid"`
} `json:"result"`
}
type Two struct {
Status int `json:"status"`
Msg string `json:"msg"`
Result map[string]innerData `json:"result"`
}
type innerData struct {
ID int `json:"id"`
Remoteurl string `json:"remoteurl"`
Status string `json:"status"`
BytesLoaded interface{} `json:"bytes_loaded"`
BytesTotal interface{} `json:"bytes_total"`
Folderid string `json:"folderid"`
Added string `json:"added"`
LastUpdate string `json:"last_update"`
Extid bool `json:"extid"`
URL bool `json:"url"`
}
func main() {
var one One
err := json.Unmarshal([]byte(res1), &one)
if err != nil {
log.Fatal(err)
}
var two Two
err = json.Unmarshal([]byte(res2), &two)
if err != nil {
log.Fatal(err)
}
//pretty print both strutures
b, _ := json.MarshalIndent(one, "", " ")
fmt.Printf("%s \n\n", b)
b, _ = json.MarshalIndent(two, "", " ")
fmt.Printf("%s \n\n", b)
// access data from two with id from one
if dat, ok := two.Result[one.Result.ID]; ok {
b, _ = json.MarshalIndent(dat, "", " ")
fmt.Printf("inner data\n%s\n", b)
}
}

MGO return bson field instead of json field

The bson name is used when performing pipe in mgo.
Struct :
type Training struct {
Id bson.ObjectId `json:"id" bson:"_id"`
Name string `json:"name" bson:"name"`
Description string `json:"description"`
Level *TrainingLevel `json:"level"`
Preworks []bson.ObjectId `json:"preworks"`
PrePostTests []bson.ObjectId `json:"preposttests" bson:"preposttests"`
TrainingEvaluations []bson.ObjectId `json:"training_evaluations" bson:"training_evaluations"`
TrainerEvaluations []bson.ObjectId `json:"trainer_evaluations" bson:"trainer_evaluations"`
AppCompanyId bson.ObjectId `json:"app_company_id" bson:"app_company_id"`
Company *Company `json:"company"`
}
Function :
func (this *TrainingClass) GetAllTraining() (interface{}, error) {
if !this.tokenInfo.IsAllowed(this.c) {
return nil, tlib.NewTError(common.Error_NoAccess, "You don't have the right!")
}
sess, db := GetDB()
defer sess.Close()
pipeline := []bson.M{
{"$match": bson.M{
"app_company_id": this.tokenInfo.AppCompanyId}},
{"$lookup": bson.M{
"from": "trainingbatch",
"localField": "_id",
"foreignField": "training._id",
"as": "trainingbatches"}},
}
resp := []bson.M{}
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)
return bson.M{"data": resp}, nil
}
Json result :
{
"data": [
{
"_id": "5995a749dbcfbe4e8cc31378",
"app_company_id": "58b24756e65bd121f6b1a923",
"description": "Description First Training",
"name": "First Training",
"trainingbatches": [
{
"_id": "5995a74adbcfbe4e8cc31379",
"app_company_id": "58b24756e65bd121f6b1a923",
"company": {
"_id": "58b24756e65bd121f6b1a923",
"address": "",
"app_company_id": "58b24756e65bd121f6b1a923",
"fullname": "",
"name": "Tandem",
"phone": ""
},
}
]
}
]
}
As you can see field _id is generated instead of id. That's not happen if I use find or findId. Is there any way to keep using json field no matter what's the query?
The way you're reading the result, it has no idea what the JSON field names are. In order for it to use those tags, it must actually deserialize into the struct where the tags have been specified. When you do:
resp := []bson.M{}
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)
You're explicitly telling mgo to return BSON results. The object you pass in (a slice of bson.M) has no json tags on it. In order to control the serialization to JSON, you must pass a struct with the JSON tags specified to All:
resp := []Training
db.C(common.C_TRAINING).Pipe(pipeline).All(&resp)

How to decode json into structs

I'm trying to decode some json in Go but some fields don't get decoded.
See the code running in browser here:
What am I doing wrong?
I need only the MX records so I didn't define the other fields. As I understand from the godoc you don't need to define the fields you don't use/need.
// You can edit this code!
// Click here and start typing.
package main
import "fmt"
import "encoding/json"
func main() {
body := `
{"response": {
"status": "SUCCESS",
"data": {
"mxRecords": [
{
"value": "us2.mx3.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx1.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx2.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
}
],
"cnameRecords": [
{
"aliasHost": "pop.a.co.uk.",
"canonicalHost": "us2.pop.mailhostbox.com."
},
{
"aliasHost": "webmail.a.co.uk.",
"canonicalHost": "us2.webmail.mailhostbox.com."
},
{
"aliasHost": "smtp.a.co.uk.",
"canonicalHost": "us2.smtp.mailhostbox.com."
},
{
"aliasHost": "imap.a.co.uk.",
"canonicalHost": "us2.imap.mailhostbox.com."
}
],
"dkimTxtRecord": {
"domainname": "20a19._domainkey.a.co.uk",
"value": "\"v=DKIM1; g=*; k=rsa; p=DkfbhO8Oyy0E1WyUWwIDAQAB\"",
"ttl": 1
},
"spfTxtRecord": {
"domainname": "a.co.uk",
"value": "\"v=spf1 redirect=_spf.mailhostbox.com\"",
"ttl": 1
},
"loginUrl": "us2.cp.mailhostbox.com"
}
}}`
type MxRecords struct {
value string
ttl int
priority int
hostName string
}
type Data struct {
mxRecords []MxRecords
}
type Response struct {
Status string `json:"status"`
Data Data `json:"data"`
}
type apiR struct {
Response Response
}
var r apiR
err := json.Unmarshal([]byte(body), &r)
if err != nil {
fmt.Printf("err was %v", err)
}
fmt.Printf("decoded is %v", r)
}
As per the go documentaiton about json.Unmarshal, you can only decode toward exported fields, the main reason being that external packages (such as encoding/json) cannot acces unexported fields.
If your json doesn't follow the go convention for names, you can use the json tag in your fields to change the matching between json key and struct field.
Exemple:
package main
import (
"fmt"
"encoding/json"
)
type T struct {
Foo string `json:"foo"`
priv string `json:"priv"`
}
func main() {
text := []byte(`{"foo":"bar", "priv":"nothing"}`)
var t T
err := json.Unmarshal(text, &t)
if err != nil {
panic(err)
}
fmt.Println(t.Foo) // prints "bar"
fmt.Println(t.priv) // prints "", priv is not exported
}
You must Uppercase struct fields:
type MxRecords struct {
Value string `json:"value"`
Ttl int `json:"ttl"`
Priority int `json:"priority"`
HostName string `json:"hostName"`
}
type Data struct {
MxRecords []MxRecords `json:"mxRecords"`
}
http://play.golang.org/p/EEyiISdoaE
The encoding/json package can only decode into exported struct fields. Your Data.mxRecords member is not exported, so it is ignored when decoding. If you rename it to use a capital letter, the JSON package will notice it.
You will need to do the same thing for all the members of your MxRecords type.