Create a struct from a byte array - json

I use the json.Marshal interface to accept a map[string]interface{} and convert it to a []byte (is this a byte array?)
data, _ := json.Marshal(value)
log.Printf("%s\n", data)
I get this output
{"email_address":"joe#me.com","street_address":"123 Anywhere Anytown","name":"joe","output":"Hello World","status":1}
The underlying bytes pertain to the struct of the below declaration
type Person struct {
Name string `json:"name"`
StreetAddress string `json:"street_address"`
Output string `json:"output"`
Status float64 `json:"status"`
EmailAddress string `json:"email_address",omitempty"`
}
I'd like to take data and generate a variable of type Person struct
How do I do that?

You use json.Unmarshal:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
Name string `json:"name"`
StreetAddress string `json:"street_address"`
Output string `json:"output"`
Status float64 `json:"status"`
EmailAddress string `json:"email_address",omitempty"`
}
func main() {
data := []byte(`{"email_address":"joe#me.com","street_address":"123 Anywhere Anytown","name":"joe","output":"Hello World","status":1}`)
var p Person
if err := json.Unmarshal(data, &p); err != nil {
panic(err)
}
fmt.Printf("%#v\n", p)
}
Output:
main.Person{Name:"joe", StreetAddress:"123 Anywhere Anytown", Output:"Hello World", Status:1, EmailAddress:"joe#me.com"}

Related

Where is the golang code to auto parse time.Time from json strings? [duplicate]

I have the following JSON
{"student_number":1234567, "name":"John Doe", "subjects":"Chemistry-Maths-History-Geography"}
I would like to unmarshal it in a struct, where one item (the subjects) are split on '-' into a []string.
type Student struct {
StudentNumber int `json:"student_number"`
Name string `json:"name"`
Subjects []string
}
I have attempted several different ways of achieving this with custom Unmarshalling using strings.Split(), but have not succeeded so far.
Is there any way to achieve this in the unmarshalling process? Or will I need to simply unmarshal as is and make the conversion afterward?
Easiest would be to define your own string slice type and implement json.Unmarshaler on that:
type strslice []string
func (ss *strslice) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*ss = strings.Split(s, "-")
return nil
}
And use this in your struct:
type Student struct {
StudentNumber int `json:"student_number"`
Name string `json:"name"`
Subjects strslice `json:"subjects"`
}
And then it'll work:
func main() {
var s Student
err := json.Unmarshal([]byte(src), &s)
fmt.Println(s, err)
}
const src = `{"student_number":1234567, "name":"John Doe", "subjects":"Chemistry-Maths-History-Geography"}`
Output (try it on the Go Playground):
{1234567 John Doe [Chemistry Maths History Geography]} <nil>

Unmarshalling nested JSON objects with dates in Golang

I'm a noob with Golang. I managed to get some things done with lots of effort.
I'm dealing with JSON files containing dates in a nested way.
I came across some workaround to unmarshal dates from JSON data into time.Time but I'm having a hard time dealing with nested ones.
The following code (obtained here in StackOverflow) is easy to understand since creates a user-defined function to parse the time objects first to a string and then to time.Time with time.Parse.
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
const dateFormat = "2006-01-02"
const data = `{
"name": "Gopher",
"join_date": "2007-09-20"
}`
type User struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
}
func (u *User) UnmarshalJSON(p []byte) error {
var aux struct {
Name string `json:"name"`
JoinDate string `json:"join_date"`
}
err := json.Unmarshal(p, &aux)
if err != nil {
return err
}
t, err := time.Parse(dateFormat, aux.JoinDate)
if err != nil {
return err
}
u.Name = aux.Name
u.JoinDate = t
return nil
}
func main() {
var u User
err := json.Unmarshal([]byte(data), &u)
if err != nil {
log.Fatal(err)
}
fmt.Println(u.JoinDate.Format(time.RFC3339))
}
So far, so good.
Now I would like to extend it in order to handle the nested date fields in the JSON, like the example below:
[{
"name": "Gopher",
"join_date": "2007-09-20",
"cashflow": [
{"date": "2021-02-25",
"amount": 100},
{"date": "2021-03-25",
"amount": 105}
]
}]
The struct that I would like to get is:
type Record []struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
Cashflow []struct {
Date time.Time `json:"date"`
Amount int `json:"amount"`
} `json:"cashflow"`
}
Thanks for the help.
To solve this using the patterns you've already got, you can write a separate unmarshalling function for the inner struct. You can do that by hoisting the inner struct to its own named struct, and then writing the function.
type CashflowRec struct {
Date time.Time `json:"date"`
Amount int `json:"amount"`
}
type Record struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
Cashflow []CashflowRec `json:"cashflow"`
}
You've already shown how to write the unmarshalling function for CashflowRec, it looks almost the same as your User function. The unmarshalling function for Record will make use of that when it calls
func (u *Record) UnmarshalJSON(p []byte) error {
var aux struct {
Name string `json:"name"`
JoinDate string `json:"join_date"`
Cashflow []CashflowRec `json:"cashflow"`
}
err := json.Unmarshal(p, &aux)
Working example: https://go.dev/play/p/1X7BJ4NETM0
aside 1 Something amusing I learned while looking at this: because you've provided your own unmarshalling function, you don't actually need the json tags in your original structs. Those are hints for the unmarshaller that the json package provides. You should probably still leave them in, in case you have to marshal the struct later. Here's it working without those tags: https://go.dev/play/p/G2VWopO_A3t
aside 2 You might find it simpler not to use time.Time, but instead create a new type of your own, and then give that type its own unmarshaller. This gives you the interesting choice for writing only that one unmarshaller, but whether or not this is a win depends on what else you do with the struct later on. Working example that still uses your nested anonymous structs: https://go.dev/play/p/bJUcaw3_r41
type dateType time.Time
type Record struct {
Name string `json:"name"`
JoinDate dateType `json:"join_date"`
Cashflow []struct {
Date dateType `json:"date"`
Amount int `json:"amount"`
} `json:"cashflow"`
}
func (c *dateType) UnmarshalJSON(p []byte) error {
var s string
if err := json.Unmarshal(p, &s); err != nil {
return err
}
t, err := time.Parse(dateFormat, s)
if err != nil {
return err
}
*c = dateType(t)
return nil
}

String from struct pointer

As a student of Go I encountered this problem.
My ultimate goal of doing this is to convert *blockchain into a valid JSON string.
My structs are:
type Blockchain struct{
blocks []Block `json:"blocks"`
difficulty int `json:"difficulty"`
}
type Block struct{
index int `json:"index"`
timestamp string `json:"timestamp"`
data string `json:"data"`
previousHash string `json:"previousHash"`
hash string `json:"hash"`
nonce int `json:"nonce"`
}
I have the following code:
var s = fmt.Sprintf("%#v", *blockchain)
print(s)
Which gives me the following:
main.Blockchain{blocks:[]main.Block{main.Block{index:1, timestamp:"2019-04-06 12:50:54", data:"Genesis block", previousHash:"", hash:"eca16d7bdd20a91f471fc3231fa5de7d892fb540789673d64f29a7b93719b74b", nonce:0}, main.Block{index:2, timestamp:"2019-04-06 12:50:54", data:"d.duck", previousHash:"eca16d7bdd20a91f471fc3231fa5de7d892fb540789673d64f29a7b93719b74b", hash:"2096ccfa6fdd8305f0e31c2e6858173a21764be4c8e1d3d50c9c31193bf06a2a", nonce:0}, main.Block{index:3, timestamp:"2019-04-06 12:50:54", data:"dumbo", previousHash:"2096ccfa6fdd8305f0e31c2e6858173a21764be4c8e1d3d50c9c31193bf06a2a", hash:"d76d4a002c6dde01009e3122aa1ccfb455e1d453ac83e2a0eb123c6080943cdb", nonce:0}}, difficulty:4}
Obviously invalid JSON.
Any suggestions?
I also tried the following:
var json, err = json.Marshal(*blockchain)
if err != nil{
panic(err.Error())
}
var s = fmt.Sprintf("%#v", json)
print(s)
It gave me the following:
[]byte{0x7b, 0x7d}
The fields on structs need to be exported (start with capital letter). This is required because JSON marshalling uses reflection, and that requires the field to be exported (to be visible).
Also json.Marshal first return value is the JSON.
package main
import (
"encoding/json"
"fmt"
"log"
)
type Blockchain struct {
Blocks []Block `json:"blocks"`
Difficulty int `json:"difficulty"`
}
type Block struct {
Index int `json:"index"`
Timestamp string `json:"timestamp"`
Data string `json:"data"`
PreviousHash string `json:"previousHash"`
Hash string `json:"hash"`
Nonce int `json:"nonce"`
}
func main() {
bc := &Blockchain{
Blocks: []Block{
Block{},
},
Difficulty: 1,
}
v, err := json.Marshal(bc)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(v))
}
This prints:
{"blocks":[{"index":0,"timestamp":"","data":"","previousHash":"","hash":"","nonce":0}],"difficulty":1}

How to navigate a nested json with unknown structure in Go?

I am trying to parse and get selected data from a deep nested json data in Go Lang. I'm having issues navigating through the structure and accessing the data. The data is too deep and complex to be parsed with a-priori known structures in Go.
Here is the URL of the file:
-https://www.data.gouv.fr/api/1/datasets/?format=csv&page=0&page_size=20
I did some parsing with map interfaces and using a json string:
resultdata := map[string]interface {}
json.Unmarshal([]byte(inputbytestring), &resultdata) //Inputstring is the string containing the JSON data of the above URL
The problem:
How can turn resultdata into a map (of strings), so I can use methods available for maps?
The JSON data is nested and has several levels. How is it possible to access the lower level JSON fields? is it possible to unmarshal the data recursively?
Once you have data as a map[string]interface{}, you can use type assertions to get to the lower levels of data.
There's a good explanation here of how to do this at https://blog.golang.org/json-and-go
Here's an example to get you most of the way:
https://play.golang.org/p/P8cGP1mTDmD
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
jsonData := `{
"string": "string_value",
"number": 123.45,
"js_array": ["a", "b", "c"],
"integer": 678,
"subtype": {
"number_array": [1, 2, 3]
}
}`
m := map[string]interface{}{}
err := json.Unmarshal([]byte(jsonData), &m)
if err != nil {
log.Fatal(err)
}
for key, value := range m {
switch v := value.(type) {
case int:
fmt.Printf("Key: %s, Integer: %d\n", key, v)
case float64:
fmt.Printf("Key: %s, Float: %v\n", key, v)
case string:
fmt.Printf("Key: %s, String: %s\n", key, v)
case map[string]interface{}:
fmt.Printf("Key: %s, Subtype: %+v\n", key, v)
case []interface{}:
//TODO: Read through each item in the interface and work out what type it is.
fmt.Printf("Key: %s, []interface: %v\n", key, v)
default:
fmt.Printf("Key: %s, unhandled type: %+v\n", key, v)
}
}
}
Alternatively https://mholt.github.io/json-to-go/ does a decent job of turning examples of JSON data into Go structs that can be used for marshalling.
Putting the example in, I get something that isn't too bad.
type AutoGenerated struct {
Data []struct {
Acronym interface{} `json:"acronym"`
Badges []interface{} `json:"badges"`
CreatedAt string `json:"created_at"`
Deleted interface{} `json:"deleted"`
Description string `json:"description"`
Extras struct {
} `json:"extras"`
Frequency string `json:"frequency"`
FrequencyDate interface{} `json:"frequency_date"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
LastUpdate string `json:"last_update"`
License string `json:"license"`
Metrics struct {
Discussions int `json:"discussions"`
Followers int `json:"followers"`
Issues int `json:"issues"`
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Reuses int `json:"reuses"`
Views int `json:"views"`
} `json:"metrics"`
Organization struct {
Acronym string `json:"acronym"`
Class string `json:"class"`
ID string `json:"id"`
Logo string `json:"logo"`
LogoThumbnail string `json:"logo_thumbnail"`
Name string `json:"name"`
Page string `json:"page"`
Slug string `json:"slug"`
URI string `json:"uri"`
} `json:"organization"`
Owner interface{} `json:"owner"`
Page string `json:"page"`
Private bool `json:"private"`
Resources []struct {
Checksum struct {
Type string `json:"type"`
Value string `json:"value"`
} `json:"checksum"`
CreatedAt string `json:"created_at"`
Description interface{} `json:"description"`
Extras struct {
} `json:"extras"`
Filesize int `json:"filesize"`
Filetype string `json:"filetype"`
Format string `json:"format"`
ID string `json:"id"`
LastModified string `json:"last_modified"`
Latest string `json:"latest"`
Metrics struct {
NbHits int `json:"nb_hits"`
NbUniqVisitors int `json:"nb_uniq_visitors"`
NbVisits int `json:"nb_visits"`
Views int `json:"views"`
} `json:"metrics"`
Mime string `json:"mime"`
PreviewURL string `json:"preview_url"`
Published string `json:"published"`
Title string `json:"title"`
Type string `json:"type"`
URL string `json:"url"`
} `json:"resources"`
Slug string `json:"slug"`
Spatial interface{} `json:"spatial"`
Tags []interface{} `json:"tags"`
TemporalCoverage interface{} `json:"temporal_coverage"`
Title string `json:"title"`
URI string `json:"uri"`
} `json:"data"`
Facets struct {
Format [][]interface{} `json:"format"`
} `json:"facets"`
NextPage string `json:"next_page"`
Page int `json:"page"`
PageSize int `json:"page_size"`
PreviousPage interface{} `json:"previous_page"`
Total int `json:"total"`
}
If you want inline decode of nested data for quick uses follow the below method:
myJsonData := `{
"code": "string_code",
"data": {
"id": 123,
"user": {
"username": "my_username",
"age": 30,
"posts": [ "post1", "post2"]
}
}
}`
Let's say you have the above nested and unknown JSON data that want to be read and parsed, first read that intomap[string]interface{}:
m := map[string]interface{}{}
err := json.Unmarshal([]byte(myJsonData), &m)
if err != nil {
log.Fatal(err)
}
Now if you want to access code
fmt.Println(m["code"])
For id in nested data block of JSON:
fmt.Println(m["data"].(map[string]interface{})["id"].(float64))
For username in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["username"].(string))
For age in the second level nested user block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["age"].(float64))
For post1 in the third level nested posts block of JSON:
fmt.Println(m["data"].(map[string]interface{})["user"].(map[string]interface{})["posts"].([]interface{})[0].(string))
Please check the example in playground

Json Decode cannot parse timestamp in json into Go struct

I am trying to obtain an HTTP request body which is a json object and decode it into a Go struct I have defined.
Two of the fields of the struct is of time.Time type. While having only one such typed field everything works correctly.
If I have more than one time.Time typed fields in the go struct I cannot decode it and get the error:
2014/11/01 01:07:04 parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """
The problem is in the decoding lines. Despite my debugging efforts I could have reached a meaningful result. That issue seems strange which is actually should not be.
What am I missing here?
func register(w http.ResponseWriter, r *http.Request){
//Read Request Body JSON Into Go Types
var requestBody = []byte(`{"username":"qwewwwqweqwe","password":"can","usertype":"student","firstname":"",‌​"midname":null,"surname":null,"signuptimestamp":null,"userstatus":null,"phone":nu‌​ll,"email":null,"address":null,"city":null,"country":null,"language":null,"lastlo‌​gintimestamp":null}`)
type RegisterStructure struct {
Id int `json:"id"`
Timestamp time.Time `json:"timestamp,omitemty"`
SignupTimestamp time.Time `json:"signuptimestamp,omitempty"`
Username string `json:"username"`
Password string `json:"password"`
UserType string `json:"usertype"`
FirstName string `json:"firstname"`
Midname string `json:"midname"`
Surname string `json:"surname"`
UserStatus string `json:"userstatus"`
Phone string `json:"phone"`
Email string `json:"email"`
Address string `json:"address"`
City string `json:"city"`
Country string `json:"country"`
Language string `json:"language"`
//LastLoginTimestamp time.Time `json:"lastlogintimestamp,omitempty"`
}
var registerInstance RegisterStructure
var now = time.Now()
fmt.Printf("now is %v", now)
fmt.Println()
fmt.Printf("1 registerInstance after inited here is %v", registerInstance)
fmt.Println()
registerInstance = RegisterStructure{Timestamp: now, SignupTimestamp: now,}
fmt.Printf("registerInstance after set to var now here is %v", registerInstance)
fmt.Println()
dec := json.NewDecoder(bytes.NewReader(requestBody))
err = dec.Decode(&registerInstance)
if err != nil {
fmt.Printf("error happens here.")
log.Fatal(err)
}
Ok. Here is a reproducible example that demonstrates the error you're seeing.
package main
import (
"encoding/json"
"fmt"
"bytes"
"time"
)
type RegisterStructure struct {
SignupTimestamp time.Time `json:"signuptimestamp,omitempty"`
}
func main() {
requestBody := []byte(`{"signuptimestamp" : null}`)
dec := json.NewDecoder(bytes.NewReader(requestBody))
registerInstance := RegisterStructure{}
err := dec.Decode(&registerInstance)
if err != nil {
fmt.Println(err)
}
}
The error you're seeing has nothing to do with having multiple timestamps. This is why showing inputs is critical for debugging situations like this, and why you should go back and change your question to include a sample requestBody as part of the question content. Otherwise, it becomes very difficult to guess what you're doing.
What's happening is that null is not handled by the JSON unmarshaller for time.Time. The documentation for time.Time's unmarshaller says: UnmarshalJSON implements the json.Unmarshaler interface. The time is expected to be a quoted string in RFC 3339 format.
null is not such a value: the decoder in this case will not try to construct a "zero" value for the timestamp.
What you want to do is change the type of SignupTimestamp from time.Time to *time.Time, so that the null value can be explicitly represented as a nil pointer.
Here is an example to demonstrate:
package main
import (
"bytes"
"encoding/json"
"fmt"
"time"
)
type RegisterStructure struct {
SignupTimestamp *time.Time `json:"signuptimestamp,omitempty"`
}
func main() {
requestBodies := []string{
`{"signuptimestamp" : "1985-04-12T23:20:50.52Z"}`,
`{"signuptimestamp" : null}`,
}
for _, requestBody := range requestBodies {
dec := json.NewDecoder(bytes.NewReader([]byte(requestBody)))
registerInstance := RegisterStructure{}
err := dec.Decode(&registerInstance)
if err != nil {
fmt.Println(err)
}
fmt.Printf("%#v\n", registerInstance)
}
}