Unmarshal JSON object published over MQTT in Go - json

I am receiving sensor data over MQTT. I want to check if the temperature is over 20 degrees, and if it is, send a message.
var f MQTT.MessageHandler = func(client MQTT.Client, msg MQTT.Message) {
type Data struct {
Sensor string `json:"sensor"`
Temp []int `json: "temperature"`
Hum []int `json: "humidity"`
}
var sensorData []Data
message := ""
err := json.Unmarshal(msg.Payload(), &sensorData)
if err != nil {
panic(err)
}else {
//access certain element in the Data struct
for _, i := range sensorData {
if i.Temp[2] > 20 { //where the temperature is stored
fmt.Println("Temperature too high")
message = "Temperature too high"
}else{
fmt.Println("Perfect temperature")
message = "Perfect temperature"
}
}
}
// Publish further instructions to "sensor/instruction"
token := client.Publish("sensor/instruction", 0, false, message)
token.Wait()
}
Currently I am publishing two JSON objects to be received. I think part of the problem is distinguishing between the two objects. This is the error I am getting panic: json: cannot unmarshal object into Go value of type []main.Data. These are the JSON objects I am publishing to a topic:
{'sensor': 'DHT22', 'temperature': [22.7, 22.7, 22.8], 'humidity': [51.9, 52.0, 52.0]}
{'actuator': 'arduinoLED', 'blink': ['false', 'false', 'false']}
The msg.Payload() gives
{"sensor": "DHT22", "temperature": [22.9, 22.9, 22.9], "humidity": [50.9, 50.9, 50.9]}
{"actuator": "arduinoLED", "blink": ["false", "false", "false"]
These objects are published constantly in a loop. I want to unmarshal the first object and access a specific element.

Slice vs single object
You are declaring a slice:
var sensorData []Data
But then your payload is not an array but rather only one object:
{'sensor': 'DHT22', 'temperature': [22.7, 22.7, 22.8], 'humidity': [51.9, 52.0, 52.0]}
So that error message is telling you it cannot unmarshal a single object as a slice.
Try changing that declaration to:
var sensorData Data
Then, instead of iterating over it, you need to just check that one instance of the data.
Additional mismatches between the payload and the struct type
After that, you'll probably get another error since the temperature and humidity array seem to contain decimal numbers:
`{'sensor': 'DHT22', 'temperature': [22.7, 22.7, 22.8], 'humidity': [51.9, 52.0, 52.0]`
But you are declaring those as int slices in your code:
Temp []int `json: "temperature"`
Hum []int `json: "humidity"`
So you'll need to change those to be []float64
Differentiating the two different payloads
About differentiating the two object types, it'd be better if you try to do that, but even if you don't, Go's json library will ignore problems if field names do not match, so what will happen is that when de-serializing your actuator payloads into Data, all fields will have their default values, but no error will be received.
This check will probably throw a panic, cause the array will be empty in that case:
if i.Temp[2] > 20
One "hacky" way of solving this issue would be to only process the data if the sensor field is not a blank string.
Assuming you always receive a sensor name in the sensor messages, the only case when that will result in an empty string is if you just processed one of the other messages.

There are two main reasons for your error One is you have float value for temperature and humidity but you are passing slice of int
type Data struct {
Sensor string `json:"sensor"`
Temp []int `json: "temperature"` // this should be []float64
Hum []int `json: "humidity"` // this should be []float64
}
Other is there are two objects in msg.Payload which is not a slice of Data struct. Working code.
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
Sensor string `json:"sensor"`
Temp []float64 `json:"temperature"`
Hum []float64 `json:"humidity"`
}
func main() {
bytes := []byte(`{
"sensor": "DHT22",
"temperature": [22.7, 22.7, 22.8],
"humidity": [51.9, 52.0, 52.0]
}`)
var sensorData Data
if err := json.Unmarshal(bytes, &sensorData); err != nil {
fmt.Println(err)
}
fmt.Println(sensorData)
}
Check working code on Go playground

Related

Unmarshal Inconsistent JSON in Go

I'm working with JSON that returns three different object types 'items','categories' and 'modifiers'. An example of the JSON can be viewed here. I created models for the three types of objects. But when I unmarshal I have selected one of the types to unmarshal the entire JSON to.(I know this cant be the correct way...) I then try to parse out the different items depending on what their type is identified as in the json field 'Type' and then append that object to a slice of the proper type. I am having errors because I don't know how to unmarshal JSON that has different types in it that have different fields.
What is the proper method to unmarshal JSON that contains different objects, each with their own respective fields?
Is the solution to create a "super model" which contains all possible fields and then unmarshal to that?
I'm still fairly new and would appreciate any advice. Thanks!
If you implement json.Unmarshaler, you can define a struct that parses each item type into it's relevant struct.
Example:
// Dynamic represents an item of any type.
type Dynamic struct {
Value interface{}
}
// UnmarshalJSON is called by the json package when we ask it to
// parse something into Dynamic.
func (d *Dynamic) UnmarshalJSON(data []byte) error {
// Parse only the "type" field first.
var meta struct {
Type string
}
if err := json.Unmarshal(data, &meta); err != nil {
return err
}
// Determine which struct to unmarshal into according to "type".
switch meta.Type {
case "product":
d.Value = &Product{}
case "post":
d.Value = &Post{}
default:
return fmt.Errorf("%q is an invalid item type", meta.Type)
}
return json.Unmarshal(data, d.Value)
}
// Product and Post are structs representing two different item types.
type Product struct {
Name string
Price int
}
type Post struct {
Title string
Content string
}
Usage:
func main() {
// Parse a JSON item into Dynamic.
input := `{
"type": "product",
"name": "iPhone",
"price": 1000
}`
var dynamic Dynamic
if err := json.Unmarshal([]byte(input), &dynamic); err != nil {
log.Fatal(err)
}
// Type switch on dynamic.Value to get the parsed struct.
// See https://tour.golang.org/methods/16
switch dynamic.Value.(type) {
case *Product:
log.Println("got a product:", dynamic.Value)
case *Post:
log.Println("got a product:", dynamic.Value)
}
}
Output:
2009/11/10 23:00:00 got a product: &{iPhone 1000}
Try it in the Go Playground.
Tip: if you have a list of dynamic objects, just parse into a slice of Dynamic:
var items []Dynamic
json.Unmarshal(`[{...}, {...}]`, &items)
Example output:
[&{iPhone 1000} &{A Good Post Lorem ipsum...}]
I think https://github.com/mitchellh/mapstructure also fits into your use case.

Unmarshalling Json data into map in Go

I am dealing with an api that returns json data such as:
{
"bpi": {
"2018-06-01": 128.2597,
"2018-06-02": 127.3648
},
"disclaimer": "something here.",
"time": {
"updated": "Sep 6, 2013 00:03:00 UTC",
"updatedISO": "2013-09-06T00:03:00+00:00"
}
However the price data that has the accompanying dates can return a dynamic date range (ie could be anything from 1 data pair to 1000).
I'm trying to take only the date and price pairs and put them into a map for later consumption, but I'm not finding a straight forward way of doing it. When I put this into a json-to-go auto struct generator it will create a statically and named struct for the pricing.
This is my best attempt at handling the data dynamically. I am passing an empty interface from the response body of the http get, specifically:
var unstructuredJSON interface{}
json.Unmarshal(body, &unstructuredJSON)
and passing the unstructuredJSON to the function:
func buildPriceMap(unstructuredJSON interface{}, priceMap map[string]float64) {
jsonBody := unstructuredJSON.(map[string]interface{})
for k, v := range jsonBody {
switch vv := v.(type) {
case string:
// Do Nothing
case float64:
priceMap[k] = vv
case interface{}:
buildPriceMap(vv, priceMap)
default:
log.Fatal("Json unknown data handling unmarshal error: ", k, vv)
}
}
Is there a better way to do this?
Assuming that you know the top level keys, e.g. bpi, disclaimer, time etc and that the "dynamic data pairs" that you are talking about are part of the bpi field, and that the key and value types of each of the members of bpi are always string: decimal number you do something like....
type APIResp struct {
BPI map[string]float64 `json:"bpi"`
Disclaimer string
// other fields
}
Now each of your pairs will be typed correctly and contained in the APIResp.BPI map. Unmarshal as you are doing already;
var r APIResp
err := json.Unmarshal(body, &r)
// TODO: check err

Parsing JSON in Go Lang: Attribute without name

I'm having some trouble parsing a JSON file from an API to Go, this is the JSON I want to parse:
{"method":"stats.provider.ex",
"result":{
"addr":"17a212wdrvEXWuipCV5gcfxdALfMdhMoqh",
"current":[{
"algo":3, // algorithm number (3 = X11)
"name":"X11", // algorithm name
"suffix":"MH", // speed suffix (kH, MH, GH, TH,...)
"profitability":"0.00045845", // current profitability in BTC/suffix/Day
"data":[{ // speed object can contain following fields:
// a (accepted), rt (rejected target), rs (rejected stale),
// rd (rejected duplicate) and ro (rejected other)
// if fields are not present, speed is 0
"a":"23.09", // accepted speed (in MH/s for X11)
"rs":"0.54", // rejected speed - stale
},
"0.0001234" // balance (unpaid)
]},
... // other algorithms here
],
"past":[{
"algo":3,
"data":[
[4863234, // timestamp; multiply with 300 to get UNIX timestamp
{"a":"28.6"}, // speed object
"0" // balance (unpaid)
],[4863235,{"a":"27.4"},"0.00000345"],
... // next entries with inc. timestamps
]},
... // other algorithms here
],
"payments":[{
"amount":"0.00431400",
"fee":"0.00023000",
"TXID":"txidhere",
"time":1453538732, // UNIX timestamp
"type":0 // payment type (0 for standard NiceHash payment)
},
... // other payments here
]
}
}
You can find more info about the API in this link: https://www.nicehash.com/doc-api
The problem I'm experiencing is in the data attribute:
"data":[{ // speed object can contain following fields:
// a (accepted), rt (rejected target), rs (rejected stale),
// rd (rejected duplicate) and ro (rejected other)
// if fields are not present, speed is 0
"a":"23.09", // accepted speed (in MH/s for X11)
"rs":"0.54", // rejected speed - stale
},
"0.0001234" // balance (unpaid)
]},
Because of the balance (unpaid) line, since it doesn't have a name I don't know how to do the struct in go.
It seems that this "data" object can be described by the following struct types (assuming its shape doesn't vary from your examples):
type Data struct {
Timestamp *int64
Speed *Speed
Balance *float64
}
type Speed struct {
Accepted *float64 `json:"a,string,omitempty"`
RejectedTarget *float64 `json:"rt,string,omitempty"`
RejectedStale *float64 `json:"rs,string,omitempty"`
RejectedDuplicate *float64 `json:"rd,string,omitempty"`
RejectedOther *float64 `json:"ro,string,omitempty"`
}
The "Speed" struct has JSON tags since that object is well-suited for the default JSON un/marshaler.
The "Data" struct, however, should implement a custom json.UnmarshalJSON so that it can handle the odd choice of a JSON array with varying types to serialize its fields. Note that my sample implementation below uses the json.RawMessage type to simplify things a bit by allowing the JSON unmarshaler to ensure proper JSON array syntax and store the bytes of each element separately so we can unmarshal them according to their respective types and shapes:
// Parse valid JSON arrays as "Data" by assuming one of the following shapes:
// 1: [int64, Speed, string(float64)]
// 2: [Speed, string(float64)]
func (d *Data) UnmarshalJSON(bs []byte) error {
// Ensure that the bytes contains a valid JSON array.
msgs := []json.RawMessage{}
err := json.Unmarshal(bs, &msgs)
if err != nil {
return err
}
// Parse the initial message as "Timestamp" int64, if necessary.
idx := 0
if len(msgs) == 3 {
ts, err := strconv.ParseInt(string(msgs[idx]), 10, 64)
if err != nil {
return err
}
d.Timestamp = &ts
idx++
}
// Parse the mandatory "Speed" struct per usual.
d.Speed = &Speed{}
err = json.Unmarshal(msgs[idx], &d.Speed)
idx++
if err != nil {
return err
}
// Parse the mandatory "Balance" item after trimming quotes.
balance, err := strconv.ParseFloat(string(msgs[idx][1:len(msgs[idx])-1]), 64)
if err != nil {
return err
}
d.Balance = &balance
return nil
}
As such, you can parse valid, properly shaped JSON arrays as "Data" objects like so:
jsonstr := `[
[4863234, {"a":"28.6"}, "0" ],
[{"a":"23.09","rs":"0.54"},"0.0001234"]
]`
datas := []Data{}
err := json.Unmarshal([]byte(jsonstr), &datas)
if err != nil {
panic(err)
}
// datas[0] = Data{Timestamp:4863234,Speed{Accepted:28.6},Balance:0}
// datas[1] = Data{Speed{Accepted:23.09,RejectedStale:0.54},Balance:0.0001234}
Of course, you would also need to implement json.MarshalJSON if you want to serialize "Data" objects into JSON.
The data field in your JSON object has an array […] as its value, and
in your example that array has two elements: an object and a string apparently containing a floating-point number.
As you can see, this is an array of geterogenous types,
hence in Go, you have two options:
Create a custom type for the elements of that array, and have an
that type implement the encoding/json.Unmarshaler interface.
Then, in that method, you can go creative about interpreting what
kind of data you're about to unmarshal, and act accordingly.
Basically, you'd peek into the input data using Decoder.Token and then
unmarshal the whole input byte slice into a value of an appropriate type
Have the value for that data field to be unmarshaled into a
slice of type []interface{} and then inspect the individual elements
by a type switch
or a series of "comma ok" type asserts.
In this case, an object will be unmarshaled into a map of type
map[string]interface{}, and that string will be unmarshaled
to a value of type string.
Basically these two approaches can be classified as "detect type as you go"
vs "unmarshal everything into data structures of the most generic types
and deal with the real typing afterwards".
Here's also a third approach.
First, it may well turn out that the types of objects in the array
which is the value of that data field are implicit from their positions
in the array. You may act accordingly by unmarshaling the value of data
into an object of your custom type implementing json.Unmarshaler, which
knows which is the real type of each data element it processes.
Second, from that
{
// speed object can contain following fields:
// a (accepted), rt (rejected target), rs (rejected stale),
// rd (rejected duplicate) and ro (rejected other)
// if fields are not present, speed is 0
"a":"23.09", // accepted speed (in MH/s for X11)
"rs":"0.54", // rejected speed - stale
}
I'd say that this "object" really can have different combinations of fields,
so to me, this looks like a candidate to be unmarshaled into
into map[string]string or map[string]float,
and not into some struct-typed object.

Converting from JSON to XML

I have seen lots of posts on here about converting from XML to JSON, and I've recently wrote a program to do so, but I was also curious how you would go about converting from JSON to XML?
Sample JSON:
"version":"0.1",
"termsofService":"http://www.wunderground.com/weather/api/d/terms.html",
"features": {
"conditions": 1
}
}
, "current_observation": {
"image": {
"url":"http://icons.wxug.com/graphics/wu2/logo_130x80.png",
"title":"Weather Underground",
"link":"http://www.wunderground.com"
},
"display_location": {
"full":"Kearney, MO",
"city":"Kearney",
"state":"MO",
"state_name":"Missouri",
I'm not sure if it'd be any use to you, but i'll post my JSON to XML program.
package main
import (
"fmt"
"net/url"
"encoding/xml"
"net/http"
"log"
"io/ioutil"
"encoding/json"
)
type reportType struct{
Version xml.CharData `xml:"version"`
TermsOfService xml.CharData `xml:"termsofService"
`
Features xml.CharData `xml:"features>feature"`
Full string `xml:"current_observation>display_location>full"`
StateName string `xml:"current_observation>display_location>state_name"`
WindGust string `xml:"current_observation>observation_location>full"`
Problem myErrorType `xml:"error"`
}
type myErrorType struct{
TypeOfError xml.CharData `xml:"type"`
Desciption xml.CharData `xml:"description"`
}
type reportTypeJson struct{
Version string `json:"version"`;
TermsOfService string `json:"termsofService"`;
Features map[string]string `json:"features"`;
CurrentObservation map[string]map[string]string `json:"current_observation"`
}
func main() {
fmt.Println("data is from WeatherUnderground.")
fmt.Println("https://www.wunderground.com/")
var state, city string
str1 := "What is your state?"
str2 := "What is your city?"
fmt.Println(str1)
fmt.Scanf("%s", &state)
fmt.Println(str2)
fmt.Scanf("%s", &city)
baseURL := "http://api.wunderground.com/api/";
apiKey := "Nunna"
var query string
//set up the query
query = baseURL+apiKey +
"/conditions/q/"+
url.QueryEscape(state)+ "/"+
url.QueryEscape(city)+ ".xml"
fmt.Println("The escaped query: "+query)
response, err := http.Get(query)
doErr(err, "After the GET")
var body []byte
body, err = ioutil.ReadAll(response.Body)
doErr(err, "After Readall")
fmt.Println(body);
fmt.Printf("The body: %s\n",body)
//Unmarshalling
var report reportType
xml.Unmarshal(body, &report)
fmt.Printf("The Report: %s\n", report)
fmt.Printf("The description is [%s]\n",report.Problem.Desciption)
//Now marshal the data out in JSON
var data []byte
var output reportTypeJson
output.Version = string(report.Version);
output.TermsOfService = string(report.TermsOfService)
output.Features= map[string]string{"feature":string(report.Features)} // allocate a map, add the 'features' value to it and assign it to output.Features
output.CurrentObservation = map[string]map[string]string {
"display_location": map[string]string {
"full": report.Full,
"state_name": report.StateName,
},"observation_location":map[string]string {"full": report.WindGust},
}
data,err = json.MarshalIndent(output,""," ")
doErr(err, "From marshalIndent")
fmt.Printf("JSON output nicely formatted: \n%s\n",data)
}
func doErr( err error, message string){
if err != nil{
log.Panicf("ERROR: %s %s \n", message, err.Error())
}
}
//OUTPUT:
//JSON output nicely formatted:
//{
// "version": "0.1",
// "termsofService": "http://www.wunderground.com/weather/api/d/terms.html",
// "features": {
// "feature": "conditions"
// },
// "current_observation": {
// "display_location": {
// "full": "Kearney, MO",
// "state_name": "Missouri"
// },
// "observation_location": {
// "full": "HOMESTEAD HILLS, Kearney, Missouri"
// }
// }
//}
This is the same process as going in the other direction. Define structs/objects to model the input (be it json in this case), create method to assign all the values from the struct modeling the input to the one you're using for output, then marshal the output to get a string. So to give a practice example using one of the more conceptually difficult fields from you type, having unmarshalled some json into an instance of reportTypeJson I can assign to reportType used to model xml like so;
report.StateName = jsonReport.CurrentObservation["display_location"]["state_name"]
The biggest difference here is the struct representing your xml is flat (like it has Something>InnerSomething>InnerInnerSomething to represent it's nested values while the struct in golang has no nesting) where as with the json, your structures in golang tend to have the same amount of nesting (like having a map[string]map[string][string] meaning items are nested 3 levels inside of the main struct). You can observe this by the amount of indirection when you access fields, like in the example above, there is one level of indirection to access CurrentObservation but that is a map of maps, so then I index into it with the display_location key which yields a map[string]string, since I'm looking for the statename value, I have to index into that with state_name to access that value.
Note that in an actual program, a number of checks would be required because these operations are unsafe. For example, if the json read did not contain a display_location object then jsonReport.CurrentObservation["display_location"] would return nil and the attempt to access ["state_name" would result in a panic.
Also, another side note, in your program I would recommend adding two functions, one called NewJsonReport(report *reportType) *reportTypeJson, err and one called NewXmlReport(report *reportTypeJson) *reportType, err in which you initialize/allocate the a new instance of the return type and return it as to avoid code duplication and make your programs main more readable. Doing this type of assignment in more than one location in a program is sloppy coding which will add a lot of maintenance cost and is likely to result in bugs down the line (like if something changes on one of the models or the input you have to fix every reference to it throughout the program rather than just updating the functions I mentioned above).

Json decode/unmarshal in golang

I have the following json:
{"results":
[{"columns":["room_id","player_name","player_ip"],
"types":["integer","text","text"],
"values":[[1,"alice","127.0.0.1"]
[1,"bob","127.0.0.1"]],
"time":0.00018839100000000002}]}
where values can have any number of elements [] inside them. When i try to unmarshal the json into a struct, the "values" tag does not get parsed properly
struct:
type queryResults struct {
Results []struct {
Columns []string `json:"columns"`
Types []string `json:"types"`
Values []struct {
Room_id int
Player_name string
Player_ip string
} `json:"values"`
Time float64 `json:"time"`
} `json:"results"`
}
Code:
//jsonString is the string input to Unmarshal
resultjson := queryResults{}
json.Unmarshal([]byte(jsonString), &resultjson)
fmt.Printf("%+v",resultjson)
Current Output:
{Results:
[{Columns:[room_id player_name player_ip]
Types:[integer text text]
Values:[{room_id:0 player_name: player_ip:}
{room_id:0 player_name: player_ip:}]
Time:0.00018839100000000002}]}
Expected Output:
{Results:
[{Columns:[room_id player_name player_ip]
Types:[integer text text]
Values:[{room_id:1 player_name:alice player_ip:127.0.0.1}
{room_id:1 player_name:bob player_ip:127.0.0.1}]
Time:0.00018839100000000002}]}
Json arrays should be unmarshalled into Go slices or arrays. Looks like you are trying to unmarshal the arrays inside values to a struct
"values": [[1,"alice","127.0.0.1"], [1,"bob","127.0.0.1"]]
Above array of arrays should be unmarshalled into a Go slice of slices.
try,
type queryResults struct {
Results []struct {
Columns []string `json:"columns"`
Types []string `json:"types"`
Values [][]interface{} `json:"values"`
Time float64 `json:"time"`
} `json:"results"`
}
in Go Playground
And don't ignore errors. If you tried,
err := json.Unmarshal([]byte(jsonString), &resultjson)
if(err != nil){
fmt.Println(err)
}
you could have seen the error.
The problem is that your struct definition expects "values" to contain an array of objects, but your actual json contains an array of arrays. If you check the result of json.Unmarshal() you'll see it reports an error about this. try on golang playground
error json: cannot unmarshal array into Go value of type struct { Room_id int; Player_name string; Player_ip string }
As far as i know you can't map this directly into the struct, you'd need to read it into an array then post process it into your final type. Here's an example that successfully parses the json [the post parsing conversion is left as an exercise for the reader]
{Results:[{Columns:[room_id player_name player_ip]
Types:[integer text text]
Values:[[1 alice 127.0.0.1] [1 bob 127.0.0.1]]
Time:0.00018839100000000002}]}