How to use FieldByName to extract nested struct value - json

Here is the struct:
type UP struct {
Rxinfo []struct {
Gatewayid string `json:"gatewayID"`
Uplinkid string `json:"uplinkID"`
Name string `json:"name"`
Time time.Time `json:"time"`
Rssi int `json:"rssi"`
Lorasnr float64 `json:"loRaSNR"`
Location struct {
Latitude int `json:"latitude"`
Longitude int `json:"longitude"`
Altitude int `json:"altitude"`
} `json:"location"`
} `json:"rxInfo"`
Adr bool `json:"adr"`
Fcnt int `json:"fCnt"`
Data string `json:"data"`
}
When I code like this:
up := UP{}
if err := json.Unmarshal(msg.Payload(), &up); err != nil {
fmt.Printf("Message could not be parsed (%s): %s", msg.Payload(), err)
}
val := reflect.ValueOf(up).FieldByName("Name")
fmt.Printf("%v",val)
It returns <invalid reflect.Value>.

The reason why the logs return an <invalid reflect.Value> error is because the field Name is located inside a slice of structs. If you want to get the value of a field from a slice of struct, you have to specify which index of the slice you want to get value from.
For example, your payload looks like this:
{
"rx_info":[
{
"gateway_id":"1",
"up_link_id":"2",
"name":"sunstrider",
"time":"2021-05-25T19:37:00Z",
"rssi":-21,
"lo_ra_snr":0.342,
"location":{
"latitiude":413,
"longitude":124,
"altitude":41
}
},
{
"gateway_id":"2",
"up_link_id":"4",
"name":"sunstrider 2",
"time":"2021-06-25T19:37:00Z",
"rssi":-41,
"lo_ra_snr":0.562,
"location":{
"latitiude":213,
"longitude":321,
"altitude":443
}
}
],
"address":true,
"fcnt":53,
"data":"this is the data"
}
Notice how the field Name is inside an object that's inside an array. If your payload looks like this, then your code should look like this:
package main
import (
"encoding/json"
"fmt"
"reflect"
"time"
)
type UP struct {
RxInfo []RxInfo `json:"rx_info"`
Address bool `json:"address"`
FCNT int `json:"fcnt"`
Data string `json:"data"`
}
type RxInfo struct {
GatewayID string `json:"gateway_id"`
UplinkID string `json:"uplink_id"`
Name string `json:"name"`
Time time.Time `json:"time"`
RSSI int `json:"rssi"`
LoRaSNR float64 `json:"lo_ra_snr"`
Location Location `json:"location"`
}
type Location struct {
Latitude int `json:"latitude"`
Longitude int `json:"longitude"`
Altitude int `json:"altitude"`
}
func main() {
up := UP{}
if err := json.Unmarshal([]byte(payload), &up); err != nil {
fmt.Printf("Message could not be parsed (%s): %s", payload, err)
}
// because the "Name" field is inside an array, we have to
// bind the payload array to an array of a predefined struct
valSlice := reflect.ValueOf(up).FieldByName("RxInfo").Interface().([]RxInfo)
// getting the value from the zeroth index
fmt.Println(valSlice[0].Name)
// getting the value from the first index
fmt.Println(valSlice[1].Name)
// since your goal is to get the value of a field, I suggest you
// get the value using the structure of the struct using
// "dot" operator rather than of using the FieldByName()
valString := up.RxInfo[0].Name
fmt.Println(valString)
}
Also, a couple of tips to keep your code clean:
You should always access the value of the payload using the structure of your struct. Only use reflect.ValueOf(variable).FieldByName("name_of_field") when you don't know the complete structure of your payload.
If you have a deeply nested struct, split them up into multiple tiny structs (like my example above) to make it easier to read and maintain.
Standardize your naming convention. This applies when you're naming variables, structs, struct fields, struct tags, etc. I personally use camel-case for field names and snake-case for json struct tags.

Related

Retrieve element from nested JSON string

This is the Go code that I have:
func main(){
s := string(`{"Id": "ABC123",
"Name": "Hello",
"RelatedItems":[
{"RId":"TEST123","RName":"TEST1","RChildren":"Ch1"},
{"RId":"TEST234","RName":"TEST2","RChildren":"Ch2"}]
}`)
var result map[string]interface{}
json.Unmarshal([]byte(s), &result)
fmt.Println("Id:", result["Id"])
Rlist := result["RelatedItems"].([]map[string]interface{})
for key, pist := range pist {
fmt.Println("Key: ", key)
fmt.Println("RID:", pist["RId"])
}
}
The struct is down below
type Model struct {
Id string `json:"Id"`
Name string `json:"ModelName"`
RelatedItems []RelatedItems `json:"RelatedItems"`
}
type RelatedItems struct {
RId string `json:"PCId"`
RName string `json:"PCName"`
RChildren string `json:"string"`
}
How would I get an output that would let me choose a particular field from the above?
eg:
Output
Id: ABC123
key:0
RID:TEST123
key:1
RID:TEST234
I am seeing this error
panic: interface conversion: interface {} is nil, not []map[string]interface {}
Based on the posted content,
I'm clear that you are facing issues retrieving data from the nested JSON string.
I've taken your piece of code and tried to compile and reproduce the issue.
After observing, I have a few suggestions based on the way the code has been written.
When the datatype present in the s is known to be similar to the type Model, the result could have been declared as type Model.
That results in var result Model instead of map[string]interface{}.
When the data that's gonna be decoded from the interface{} is not known, the usage of switch would come into rescue without crashing the code.
Something similar to:
switch dataType := result["RelatedItems"].(type){
case interface{}:
// Handle interface{}
case []map[string]interface{}:
// Handle []map[string]interface{}
default:
fmt.Println("Unexpected-Datatype", dataType)
// Handle Accordingly
When we try to Unmarshal, we make sure to look into the json tags that are provided for the fields of a structure. If the data encoded is not having the tags we provided, the data will not be decoded accordingly.
Hence, the result of decoding the data from s into result would result in {ABC123 [{ } { }]} as the tags of the fields Name, RId, RName, RChildren are given as ModelName, PCId, PCName, string respectively.
By the above suggestions and refining the tags given, the piece of code would be as following which would definitely retrieve data from nested JSON structures.
s := string(`{"Id": "ABC123",
"Name": "Hello",
"RelatedItems":[
{"RId":"TEST123","RName":"TEST1","RChildren":"Ch1"},
{"RId":"TEST234","RName":"TEST2","RChildren":"Ch2"}]
}`)
var result Model
json.Unmarshal([]byte(s), &result)
fmt.Println(result)
type Model struct {
Id string `json:"Id"`
Name string `json:"Name"`
RelatedItems []RelatedItems `json:"RelatedItems"`
}
type RelatedItems struct {
RId string `json:"RId"`
RName string `json:"RName"`
RChildren string `json:"RChildren"`
}
This results in the output: {ABC123 Hello [{TEST123 TEST1 Ch1} {TEST234 TEST2 Ch2}]}
Why would you unmarshal to a map anyway and go through type checks?
type Model struct {
Id string `json:"Id"`
Name string `json:"ModelName"`
RelatedItems []RelatedItems `json:"RelatedItems"`
}
type RelatedItems struct {
RId string `json:"PCId"`
RName string `json:"PCName"`
RChildren string `json:"string"`
}
s := `{"Id": "ABC123",
"Name": "Hello",
"RelatedItems":[
{"RId":"TEST123","RName":"TEST1","RChildren":"Ch1"},
{"RId":"TEST234","RName":"TEST2","RChildren":"Ch2"}]
}`
var result Model
if err := json.Unmarshal([]byte(s), &result); err != nil {
log.Fatal(err.Error())
}
fmt.Println("Id: ", result.Id)
for index, ri := range result.RelatedItems {
fmt.Printf("Key: %d\n", index)
fmt.Printf("RID: %s\n", ri.RId)
}

String from struct pointer

As a student of Go I encountered this problem.
My ultimate goal of doing this is to convert *blockchain into a valid JSON string.
My structs are:
type Blockchain struct{
blocks []Block `json:"blocks"`
difficulty int `json:"difficulty"`
}
type Block struct{
index int `json:"index"`
timestamp string `json:"timestamp"`
data string `json:"data"`
previousHash string `json:"previousHash"`
hash string `json:"hash"`
nonce int `json:"nonce"`
}
I have the following code:
var s = fmt.Sprintf("%#v", *blockchain)
print(s)
Which gives me the following:
main.Blockchain{blocks:[]main.Block{main.Block{index:1, timestamp:"2019-04-06 12:50:54", data:"Genesis block", previousHash:"", hash:"eca16d7bdd20a91f471fc3231fa5de7d892fb540789673d64f29a7b93719b74b", nonce:0}, main.Block{index:2, timestamp:"2019-04-06 12:50:54", data:"d.duck", previousHash:"eca16d7bdd20a91f471fc3231fa5de7d892fb540789673d64f29a7b93719b74b", hash:"2096ccfa6fdd8305f0e31c2e6858173a21764be4c8e1d3d50c9c31193bf06a2a", nonce:0}, main.Block{index:3, timestamp:"2019-04-06 12:50:54", data:"dumbo", previousHash:"2096ccfa6fdd8305f0e31c2e6858173a21764be4c8e1d3d50c9c31193bf06a2a", hash:"d76d4a002c6dde01009e3122aa1ccfb455e1d453ac83e2a0eb123c6080943cdb", nonce:0}}, difficulty:4}
Obviously invalid JSON.
Any suggestions?
I also tried the following:
var json, err = json.Marshal(*blockchain)
if err != nil{
panic(err.Error())
}
var s = fmt.Sprintf("%#v", json)
print(s)
It gave me the following:
[]byte{0x7b, 0x7d}
The fields on structs need to be exported (start with capital letter). This is required because JSON marshalling uses reflection, and that requires the field to be exported (to be visible).
Also json.Marshal first return value is the JSON.
package main
import (
"encoding/json"
"fmt"
"log"
)
type Blockchain struct {
Blocks []Block `json:"blocks"`
Difficulty int `json:"difficulty"`
}
type Block struct {
Index int `json:"index"`
Timestamp string `json:"timestamp"`
Data string `json:"data"`
PreviousHash string `json:"previousHash"`
Hash string `json:"hash"`
Nonce int `json:"nonce"`
}
func main() {
bc := &Blockchain{
Blocks: []Block{
Block{},
},
Difficulty: 1,
}
v, err := json.Marshal(bc)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(v))
}
This prints:
{"blocks":[{"index":0,"timestamp":"","data":"","previousHash":"","hash":"","nonce":0}],"difficulty":1}

Unmarshall should return error when having wrong json input structure

I have struct A and B. When a JSON string A is unmarshalled to struct A then it is valid, however if the JSON string A is unmarshalled to struct B it is still successful (which should not).
Is there any way to detect that wrong JSON input has been wrongly converted to a wrong struct type?
Please look at the code below: play
package main
import (
"encoding/json"
"fmt"
)
type A struct {
Name string `json:"name"`
Age int `json:"age"`
}
type B struct {
Alamat string `json:"alamat"`
Umur int `json:"umur"`
}
func main() {
var structA A
var structAA A
valA := "{\"name\":\"budi\",\"age\":10}"
valB := "{\"alamat\":\"jakarta\",\"umur\":120}"
//correct case
err := json.Unmarshal([]byte(valA), &structA)
if err != nil {
fmt.Println("fail unmarshal")
}
fmt.Println(structA.Name)
fmt.Println(structA.Age)
//unmarshalled successfully but with wrong json
err = json.Unmarshal([]byte(valB), &structAA)
if err != nil {
fmt.Println("fail unmarshal")
}
fmt.Println(structAA.Name)
fmt.Println(structAA.Age)
}
Referring the docs, json.Unmarshal(data []byte, v interface{}), by default, does not work in the way you suppose:
By default, object keys which don't have a corresponding struct field are ignored (see Decoder.DisallowUnknownFields for an alternative).
So, JSON properties alamat and umur in your valB are ignored as they are not matched in struct A and name and age are set to their type default.

How to navigate a nested json with unknown structure in Go?

I am trying to parse and get selected data from a deep nested json data in Go Lang. I'm having issues navigating through the structure and accessing the data. The data is too deep and complex to be parsed with a-priori known structures in Go.
Here is the URL of the file:
-https://www.data.gouv.fr/api/1/datasets/?format=csv&page=0&page_size=20
I did some parsing with map interfaces and using a json string:
resultdata := map[string]interface {}
json.Unmarshal([]byte(inputbytestring), &resultdata) //Inputstring is the string containing the JSON data of the above URL
The problem:
How can turn resultdata into a map (of strings), so I can use methods available for maps?
The JSON data is nested and has several levels. How is it possible to access the lower level JSON fields? is it possible to unmarshal the data recursively?
Once you have data as a map[string]interface{}, you can use type assertions to get to the lower levels of data.
There's a good explanation here of how to do this at https://blog.golang.org/json-and-go
Here's an example to get you most of the way:
https://play.golang.org/p/P8cGP1mTDmD
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
jsonData := `{
"string": "string_value",
"number": 123.45,
"js_array": ["a", "b", "c"],
"integer": 678,
"subtype": {
"number_array": [1, 2, 3]
}
}`
m := map[string]interface{}{}
err := json.Unmarshal([]byte(jsonData), &m)
if err != nil {
log.Fatal(err)
}
for key, value := range m {
switch v := value.(type) {
case int:
fmt.Printf("Key: %s, Integer: %d\n", key, v)
case float64:
fmt.Printf("Key: %s, Float: %v\n", key, v)
case string:
fmt.Printf("Key: %s, String: %s\n", key, v)
case map[string]interface{}:
fmt.Printf("Key: %s, Subtype: %+v\n", key, v)
case []interface{}:
//TODO: Read through each item in the interface and work out what type it is.
fmt.Printf("Key: %s, []interface: %v\n", key, v)
default:
fmt.Printf("Key: %s, unhandled type: %+v\n", key, v)
}
}
}
Alternatively https://mholt.github.io/json-to-go/ does a decent job of turning examples of JSON data into Go structs that can be used for marshalling.
Putting the example in, I get something that isn't too bad.
type AutoGenerated struct {
Data []struct {
Acronym interface{} `json:"acronym"`
Badges []interface{} `json:"badges"`
CreatedAt string `json:"created_at"`
Deleted interface{} `json:"deleted"`
Description string `json:"description"`
Extras struct {
} `json:"extras"`
Frequency string `json:"frequency"`
FrequencyDate interface{} `json:"frequency_date"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
LastUpdate string `json:"last_update"`
License string `json:"license"`
Metrics struct {
Discussions int `json:"discussions"`
Followers int `json:"followers"`
Issues int `json:"issues"`
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Reuses int `json:"reuses"`
Views int `json:"views"`
} `json:"metrics"`
Organization struct {
Acronym string `json:"acronym"`
Class string `json:"class"`
ID string `json:"id"`
Logo string `json:"logo"`
LogoThumbnail string `json:"logo_thumbnail"`
Name string `json:"name"`
Page string `json:"page"`
Slug string `json:"slug"`
URI string `json:"uri"`
} `json:"organization"`
Owner interface{} `json:"owner"`
Page string `json:"page"`
Private bool `json:"private"`
Resources []struct {
Checksum struct {
Type string `json:"type"`
Value string `json:"value"`
} `json:"checksum"`
CreatedAt string `json:"created_at"`
Description interface{} `json:"description"`
Extras struct {
} `json:"extras"`
Filesize int `json:"filesize"`
Filetype string `json:"filetype"`
Format string `json:"format"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
Latest string `json:"latest"`
Metrics struct {
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Views int `json:"views"`
} `json:"metrics"`
Mime string `json:"mime"`
PreviewURL string `json:"preview_url"`
Published string `json:"published"`
Title string `json:"title"`
Type string `json:"type"`
URL string `json:"url"`
} `json:"resources"`
Slug string `json:"slug"`
Spatial interface{} `json:"spatial"`
Tags []interface{} `json:"tags"`
TemporalCoverage interface{} `json:"temporal_coverage"`
Title string `json:"title"`
URI string `json:"uri"`
} `json:"data"`
Facets struct {
Format [][]interface{} `json:"format"`
} `json:"facets"`
NextPage string `json:"next_page"`
Page int `json:"page"`
PageSize int `json:"page_size"`
PreviousPage interface{} `json:"previous_page"`
Total int `json:"total"`
}
If you want inline decode of nested data for quick uses follow the below method:
myJsonData := `{
"code": "string_code",
"data": {
"id": 123,
"user": {
"username": "my_username",
"age": 30,
"posts": [ "post1", "post2"]
}
}
}`
Let's say you have the above nested and unknown JSON data that want to be read and parsed, first read that intomap[string]interface{}:
m := map[string]interface{}{}
err := json.Unmarshal([]byte(myJsonData), &m)
if err != nil {
log.Fatal(err)
}
Now if you want to access code
fmt.Println(m["code"])
For id in nested data block of JSON:
fmt.Println(m["data"].(map[string]interface{})["id"].(float64))
For username in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["username"].(string))
For age in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["age"].(float64))
For post1 in the third level nested posts block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["posts"].([]interface{})[0].(string))
Please check the example in playground

convert interface{} directly to int in Golang, where interface stores a number as string

I got a map[string]interface{} because decoding to JSON; with normal data, interface most be only a number but in type string, like this :
var a interface{}
a="3"
And then all data will be stored into a struct.
type someStruct struct {
ID string
Number1 int
Number2 int
Number3 int
Number4 int
}
So I need to convert interface to int, but can´t do it easily and efficiently because only the code would be https://play.golang.org/p/oktbvTUbk93, pretty annoying and code does not seem to be readable if you take in mind the fact that I should handle all of possible errors
I would want to convert it directly to int, I´ve been searching for solutions but any of this convertions works as I want https://play.golang.org/p/Dw67U6kZwHC
In case you wondering why I don´t just decode it into struct directly, it´s because it´s dynamic data, the actual decoding occurs like this :
type dataIn struct {
Code int `json:"code"`
ID string `json:"id"`
Data interface{} `json:"data"`
}
And then I handle Data according to code and id, and they all are different data structures, so I can´t directly handle it with JSON
Create a helper function which does parsing and validating for you, in one place:
func parseInt(i interface{}) (int, error) {
s, ok := i.(string)
if !ok {
return 0, errors.New("not string")
}
return strconv.Atoi(s)
}
And you can use this where needed. Here's a complete example, in which I also used another helper function, which takes care of the error handling:
m := map[string]interface{}{
"number1": "1",
"number2": "2",
"number3": "3",
"number4": "4",
"ID": "asdsa",
"Title": "asdas",
}
getInt := func(key string) int {
n, err := parseInt(m[key])
if err != nil {
panic(err) // Decide what you wanna do with error
}
return n
}
// converting to struct
data := element{
ID: m["ID"].(string),
Title: m["Title"].(string),
Number1: getInt("number1"),
Number2: getInt("number2"),
Number3: getInt("number3"),
Number4: getInt("number4"),
}
fmt.Printf("%+v\n", data)
Output of the above (try it on the Go Playground):
{ID:asdsa Title:asdas Number1:1 Number2:2 Number3:3 Number4:4}
Also note that the open source package github.com/icza/dyno should help you handle dynamic objects at ease. (Disclosure: I'm the author.) It has a dyno.GetInteger() function for example, which is capable of extracting an int64 value out of multiple types (such as integers, floats, strings etc.).
Sounds like what you need is a custom unmarshal json method on your main struct. First, unmarshal into your main struct to get your code and id, then use those in a switch statement to determine which struct to use for the rest of the data and unmarshal into that, stirring that struct in your Data field.
I still didn't get the part where you stated the struct is generated dynamically. Anyways, you can have a struct method attached, that does the int conversion. If the Data field, which is of type interface{} is always gonna hold integers, you an try this:
type DataIn struct {
Code int `json:"code"`
ID string `json:"id"`
Data interface{} `json:"data"`
}
func (s DataIn) toInt() int {
switch t := s.Data.(type)
case int:
i, _ := strings.Atoi(fmt.Sprintf("%v",s.Data))
return i
}
// using it
sampleData := someStruct{
Number1: datain.toInt(),
}