String from struct pointer - json

As a student of Go I encountered this problem.
My ultimate goal of doing this is to convert *blockchain into a valid JSON string.
My structs are:
type Blockchain struct{
blocks []Block `json:"blocks"`
difficulty int `json:"difficulty"`
}
type Block struct{
index int `json:"index"`
timestamp string `json:"timestamp"`
data string `json:"data"`
previousHash string `json:"previousHash"`
hash string `json:"hash"`
nonce int `json:"nonce"`
}
I have the following code:
var s = fmt.Sprintf("%#v", *blockchain)
print(s)
Which gives me the following:
main.Blockchain{blocks:[]main.Block{main.Block{index:1, timestamp:"2019-04-06 12:50:54", data:"Genesis block", previousHash:"", hash:"eca16d7bdd20a91f471fc3231fa5de7d892fb540789673d64f29a7b93719b74b", nonce:0}, main.Block{index:2, timestamp:"2019-04-06 12:50:54", data:"d.duck", previousHash:"eca16d7bdd20a91f471fc3231fa5de7d892fb540789673d64f29a7b93719b74b", hash:"2096ccfa6fdd8305f0e31c2e6858173a21764be4c8e1d3d50c9c31193bf06a2a", nonce:0}, main.Block{index:3, timestamp:"2019-04-06 12:50:54", data:"dumbo", previousHash:"2096ccfa6fdd8305f0e31c2e6858173a21764be4c8e1d3d50c9c31193bf06a2a", hash:"d76d4a002c6dde01009e3122aa1ccfb455e1d453ac83e2a0eb123c6080943cdb", nonce:0}}, difficulty:4}
Obviously invalid JSON.
Any suggestions?
I also tried the following:
var json, err = json.Marshal(*blockchain)
if err != nil{
panic(err.Error())
}
var s = fmt.Sprintf("%#v", json)
print(s)
It gave me the following:
[]byte{0x7b, 0x7d}

The fields on structs need to be exported (start with capital letter). This is required because JSON marshalling uses reflection, and that requires the field to be exported (to be visible).
Also json.Marshal first return value is the JSON.
package main
import (
"encoding/json"
"fmt"
"log"
)
type Blockchain struct {
Blocks []Block `json:"blocks"`
Difficulty int `json:"difficulty"`
}
type Block struct {
Index int `json:"index"`
Timestamp string `json:"timestamp"`
Data string `json:"data"`
PreviousHash string `json:"previousHash"`
Hash string `json:"hash"`
Nonce int `json:"nonce"`
}
func main() {
bc := &Blockchain{
Blocks: []Block{
Block{},
},
Difficulty: 1,
}
v, err := json.Marshal(bc)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(v))
}
This prints:
{"blocks":[{"index":0,"timestamp":"","data":"","previousHash":"","hash":"","nonce":0}],"difficulty":1}

Related

How to use FieldByName to extract nested struct value

Here is the struct:
type UP struct {
Rxinfo []struct {
Gatewayid string `json:"gatewayID"`
Uplinkid string `json:"uplinkID"`
Name string `json:"name"`
Time time.Time `json:"time"`
Rssi int `json:"rssi"`
Lorasnr float64 `json:"loRaSNR"`
Location struct {
Latitude int `json:"latitude"`
Longitude int `json:"longitude"`
Altitude int `json:"altitude"`
} `json:"location"`
} `json:"rxInfo"`
Adr bool `json:"adr"`
Fcnt int `json:"fCnt"`
Data string `json:"data"`
}
When I code like this:
up := UP{}
if err := json.Unmarshal(msg.Payload(), &up); err != nil {
fmt.Printf("Message could not be parsed (%s): %s", msg.Payload(), err)
}
val := reflect.ValueOf(up).FieldByName("Name")
fmt.Printf("%v",val)
It returns <invalid reflect.Value>.
The reason why the logs return an <invalid reflect.Value> error is because the field Name is located inside a slice of structs. If you want to get the value of a field from a slice of struct, you have to specify which index of the slice you want to get value from.
For example, your payload looks like this:
{
"rx_info":[
{
"gateway_id":"1",
"up_link_id":"2",
"name":"sunstrider",
"time":"2021-05-25T19:37:00Z",
"rssi":-21,
"lo_ra_snr":0.342,
"location":{
"latitiude":413,
"longitude":124,
"altitude":41
}
},
{
"gateway_id":"2",
"up_link_id":"4",
"name":"sunstrider 2",
"time":"2021-06-25T19:37:00Z",
"rssi":-41,
"lo_ra_snr":0.562,
"location":{
"latitiude":213,
"longitude":321,
"altitude":443
}
}
],
"address":true,
"fcnt":53,
"data":"this is the data"
}
Notice how the field Name is inside an object that's inside an array. If your payload looks like this, then your code should look like this:
package main
import (
"encoding/json"
"fmt"
"reflect"
"time"
)
type UP struct {
RxInfo []RxInfo `json:"rx_info"`
Address bool `json:"address"`
FCNT int `json:"fcnt"`
Data string `json:"data"`
}
type RxInfo struct {
GatewayID string `json:"gateway_id"`
UplinkID string `json:"uplink_id"`
Name string `json:"name"`
Time time.Time `json:"time"`
RSSI int `json:"rssi"`
LoRaSNR float64 `json:"lo_ra_snr"`
Location Location `json:"location"`
}
type Location struct {
Latitude int `json:"latitude"`
Longitude int `json:"longitude"`
Altitude int `json:"altitude"`
}
func main() {
up := UP{}
if err := json.Unmarshal([]byte(payload), &up); err != nil {
fmt.Printf("Message could not be parsed (%s): %s", payload, err)
}
// because the "Name" field is inside an array, we have to
// bind the payload array to an array of a predefined struct
valSlice := reflect.ValueOf(up).FieldByName("RxInfo").Interface().([]RxInfo)
// getting the value from the zeroth index
fmt.Println(valSlice[0].Name)
// getting the value from the first index
fmt.Println(valSlice[1].Name)
// since your goal is to get the value of a field, I suggest you
// get the value using the structure of the struct using
// "dot" operator rather than of using the FieldByName()
valString := up.RxInfo[0].Name
fmt.Println(valString)
}
Also, a couple of tips to keep your code clean:
You should always access the value of the payload using the structure of your struct. Only use reflect.ValueOf(variable).FieldByName("name_of_field") when you don't know the complete structure of your payload.
If you have a deeply nested struct, split them up into multiple tiny structs (like my example above) to make it easier to read and maintain.
Standardize your naming convention. This applies when you're naming variables, structs, struct fields, struct tags, etc. I personally use camel-case for field names and snake-case for json struct tags.

Unmarshall should return error when having wrong json input structure

I have struct A and B. When a JSON string A is unmarshalled to struct A then it is valid, however if the JSON string A is unmarshalled to struct B it is still successful (which should not).
Is there any way to detect that wrong JSON input has been wrongly converted to a wrong struct type?
Please look at the code below: play
package main
import (
"encoding/json"
"fmt"
)
type A struct {
Name string `json:"name"`
Age int `json:"age"`
}
type B struct {
Alamat string `json:"alamat"`
Umur int `json:"umur"`
}
func main() {
var structA A
var structAA A
valA := "{\"name\":\"budi\",\"age\":10}"
valB := "{\"alamat\":\"jakarta\",\"umur\":120}"
//correct case
err := json.Unmarshal([]byte(valA), &structA)
if err != nil {
fmt.Println("fail unmarshal")
}
fmt.Println(structA.Name)
fmt.Println(structA.Age)
//unmarshalled successfully but with wrong json
err = json.Unmarshal([]byte(valB), &structAA)
if err != nil {
fmt.Println("fail unmarshal")
}
fmt.Println(structAA.Name)
fmt.Println(structAA.Age)
}
Referring the docs, json.Unmarshal(data []byte, v interface{}), by default, does not work in the way you suppose:
By default, object keys which don't have a corresponding struct field are ignored (see Decoder.DisallowUnknownFields for an alternative).
So, JSON properties alamat and umur in your valB are ignored as they are not matched in struct A and name and age are set to their type default.

How to navigate a nested json with unknown structure in Go?

I am trying to parse and get selected data from a deep nested json data in Go Lang. I'm having issues navigating through the structure and accessing the data. The data is too deep and complex to be parsed with a-priori known structures in Go.
Here is the URL of the file:
-https://www.data.gouv.fr/api/1/datasets/?format=csv&page=0&page_size=20
I did some parsing with map interfaces and using a json string:
resultdata := map[string]interface {}
json.Unmarshal([]byte(inputbytestring), &resultdata) //Inputstring is the string containing the JSON data of the above URL
The problem:
How can turn resultdata into a map (of strings), so I can use methods available for maps?
The JSON data is nested and has several levels. How is it possible to access the lower level JSON fields? is it possible to unmarshal the data recursively?
Once you have data as a map[string]interface{}, you can use type assertions to get to the lower levels of data.
There's a good explanation here of how to do this at https://blog.golang.org/json-and-go
Here's an example to get you most of the way:
https://play.golang.org/p/P8cGP1mTDmD
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
jsonData := `{
"string": "string_value",
"number": 123.45,
"js_array": ["a", "b", "c"],
"integer": 678,
"subtype": {
"number_array": [1, 2, 3]
}
}`
m := map[string]interface{}{}
err := json.Unmarshal([]byte(jsonData), &m)
if err != nil {
log.Fatal(err)
}
for key, value := range m {
switch v := value.(type) {
case int:
fmt.Printf("Key: %s, Integer: %d\n", key, v)
case float64:
fmt.Printf("Key: %s, Float: %v\n", key, v)
case string:
fmt.Printf("Key: %s, String: %s\n", key, v)
case map[string]interface{}:
fmt.Printf("Key: %s, Subtype: %+v\n", key, v)
case []interface{}:
//TODO: Read through each item in the interface and work out what type it is.
fmt.Printf("Key: %s, []interface: %v\n", key, v)
default:
fmt.Printf("Key: %s, unhandled type: %+v\n", key, v)
}
}
}
Alternatively https://mholt.github.io/json-to-go/ does a decent job of turning examples of JSON data into Go structs that can be used for marshalling.
Putting the example in, I get something that isn't too bad.
type AutoGenerated struct {
Data []struct {
Acronym interface{} `json:"acronym"`
Badges []interface{} `json:"badges"`
CreatedAt string `json:"created_at"`
Deleted interface{} `json:"deleted"`
Description string `json:"description"`
Extras struct {
} `json:"extras"`
Frequency string `json:"frequency"`
FrequencyDate interface{} `json:"frequency_date"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
LastUpdate string `json:"last_update"`
License string `json:"license"`
Metrics struct {
Discussions int `json:"discussions"`
Followers int `json:"followers"`
Issues int `json:"issues"`
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Reuses int `json:"reuses"`
Views int `json:"views"`
} `json:"metrics"`
Organization struct {
Acronym string `json:"acronym"`
Class string `json:"class"`
ID string `json:"id"`
Logo string `json:"logo"`
LogoThumbnail string `json:"logo_thumbnail"`
Name string `json:"name"`
Page string `json:"page"`
Slug string `json:"slug"`
URI string `json:"uri"`
} `json:"organization"`
Owner interface{} `json:"owner"`
Page string `json:"page"`
Private bool `json:"private"`
Resources []struct {
Checksum struct {
Type string `json:"type"`
Value string `json:"value"`
} `json:"checksum"`
CreatedAt string `json:"created_at"`
Description interface{} `json:"description"`
Extras struct {
} `json:"extras"`
Filesize int `json:"filesize"`
Filetype string `json:"filetype"`
Format string `json:"format"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
Latest string `json:"latest"`
Metrics struct {
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Views int `json:"views"`
} `json:"metrics"`
Mime string `json:"mime"`
PreviewURL string `json:"preview_url"`
Published string `json:"published"`
Title string `json:"title"`
Type string `json:"type"`
URL string `json:"url"`
} `json:"resources"`
Slug string `json:"slug"`
Spatial interface{} `json:"spatial"`
Tags []interface{} `json:"tags"`
TemporalCoverage interface{} `json:"temporal_coverage"`
Title string `json:"title"`
URI string `json:"uri"`
} `json:"data"`
Facets struct {
Format [][]interface{} `json:"format"`
} `json:"facets"`
NextPage string `json:"next_page"`
Page int `json:"page"`
PageSize int `json:"page_size"`
PreviousPage interface{} `json:"previous_page"`
Total int `json:"total"`
}
If you want inline decode of nested data for quick uses follow the below method:
myJsonData := `{
"code": "string_code",
"data": {
"id": 123,
"user": {
"username": "my_username",
"age": 30,
"posts": [ "post1", "post2"]
}
}
}`
Let's say you have the above nested and unknown JSON data that want to be read and parsed, first read that intomap[string]interface{}:
m := map[string]interface{}{}
err := json.Unmarshal([]byte(myJsonData), &m)
if err != nil {
log.Fatal(err)
}
Now if you want to access code
fmt.Println(m["code"])
For id in nested data block of JSON:
fmt.Println(m["data"].(map[string]interface{})["id"].(float64))
For username in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["username"].(string))
For age in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["age"].(float64))
For post1 in the third level nested posts block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["posts"].([]interface{})[0].(string))
Please check the example in playground

How do I unmarshal a string value from a JSON document into a variable of type int?

My struct:
type User struct {
FirstName string `json:"firstname, omitempty" validate:"required"`
LastName string `json:"lastname, omitempty" validate:"required"`
NumberofDays int `json:"numberofdays, string" validate:"min=0,max=100"`
}
Value for NumberofDays is passed as string from the server but I want to check if it is within range and store as int.
Ex: user := &User{"Michael","Msk","3"}
I'm getting 'cannot unmarshal string into Go value of type int'.
I'm not sure how to typecast to int and do the validation
Remove the space after the comma in the struct tags, e.g. json:"numberofdays, string" should be json:"numberofdays,string". With the space, the json package ignores the string part, hence the error message you're getting. Without the space, the error goes away and behavior is as expected.
Demo comparing the two here: https://play.golang.org/p/AUImnw_PIS
Documentation of json package struct tags: https://golang.org/pkg/encoding/json/#Marshal
You can use a custom type to unmarshal a string
to an integer value using whatever set of parsing rules
you want this mapping to use:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type User struct {
FirstName string `json:"firstname,omitempty" validate:"required"`
LastName string `json:"lastname,omitempty" validate:"required"`
NumberofDays StrInt `json:"numberofdays" validate:"min=0,max=100"`
}
type StrInt int
func (si *StrInt) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
n, err := strconv.ParseInt(s, 10, 0)
if err != nil {
return err
}
*si = StrInt(n)
return nil
}
const data = `{
"FirstName": "John",
"LastName": "Doe",
"NumberOfDays": "42"
}`
func main() {
var u User
err := json.Unmarshal([]byte(data), &u)
if err != nil {
panic(err)
}
fmt.Println(u)
}
Playground.
The idea is as follows:
Have a custom type which is mostly int but allows defining custom methods on it.
Have that type satisfy the encoding/json.Unmarshaler interface by implementing the UnmarshalJSON() method on a pointer to that type.
In the method, first unmarshal the value as its native type—string—and
then parse it as an integer.
This approach might appear to be weird, but if you'll think of it
a bit more, there's no single "true" string representation of an integer:
there are different bases and different conventions at representing
integers even in a particular base.
In your unmarshaling method you implement the policy of the text representation
of your integers.
The json package defines a custom type json.Number for these cases.

Create a struct from a byte array

I use the json.Marshal interface to accept a map[string]interface{} and convert it to a []byte (is this a byte array?)
data, _ := json.Marshal(value)
log.Printf("%s\n", data)
I get this output
{"email_address":"joe#me.com","street_address":"123 Anywhere Anytown","name":"joe","output":"Hello World","status":1}
The underlying bytes pertain to the struct of the below declaration
type Person struct {
Name string `json:"name"`
StreetAddress string `json:"street_address"`
Output string `json:"output"`
Status float64 `json:"status"`
EmailAddress string `json:"email_address",omitempty"`
}
I'd like to take data and generate a variable of type Person struct
How do I do that?
You use json.Unmarshal:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
Name string `json:"name"`
StreetAddress string `json:"street_address"`
Output string `json:"output"`
Status float64 `json:"status"`
EmailAddress string `json:"email_address",omitempty"`
}
func main() {
data := []byte(`{"email_address":"joe#me.com","street_address":"123 Anywhere Anytown","name":"joe","output":"Hello World","status":1}`)
var p Person
if err := json.Unmarshal(data, &p); err != nil {
panic(err)
}
fmt.Printf("%#v\n", p)
}
Output:
main.Person{Name:"joe", StreetAddress:"123 Anywhere Anytown", Output:"Hello World", Status:1, EmailAddress:"joe#me.com"}