Json Decode cannot parse timestamp in json into Go struct - json

I am trying to obtain an HTTP request body which is a json object and decode it into a Go struct I have defined.
Two of the fields of the struct is of time.Time type. While having only one such typed field everything works correctly.
If I have more than one time.Time typed fields in the go struct I cannot decode it and get the error:
2014/11/01 01:07:04 parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """
The problem is in the decoding lines. Despite my debugging efforts I could have reached a meaningful result. That issue seems strange which is actually should not be.
What am I missing here?
func register(w http.ResponseWriter, r *http.Request){
//Read Request Body JSON Into Go Types
var requestBody = []byte(`{"username":"qwewwwqweqwe","password":"can","usertype":"student","firstname":"",‌​"midname":null,"surname":null,"signuptimestamp":null,"userstatus":null,"phone":nu‌​ll,"email":null,"address":null,"city":null,"country":null,"language":null,"lastlo‌​gintimestamp":null}`)
type RegisterStructure struct {
Id int `json:"id"`
Timestamp time.Time `json:"timestamp,omitemty"`
SignupTimestamp time.Time `json:"signuptimestamp,omitempty"`
Username string `json:"username"`
Password string `json:"password"`
UserType string `json:"usertype"`
FirstName string `json:"firstname"`
Midname string `json:"midname"`
Surname string `json:"surname"`
UserStatus string `json:"userstatus"`
Phone string `json:"phone"`
Email string `json:"email"`
Address string `json:"address"`
City string `json:"city"`
Country string `json:"country"`
Language string `json:"language"`
//LastLoginTimestamp time.Time `json:"lastlogintimestamp,omitempty"`
}
var registerInstance RegisterStructure
var now = time.Now()
fmt.Printf("now is %v", now)
fmt.Println()
fmt.Printf("1 registerInstance after inited here is %v", registerInstance)
fmt.Println()
registerInstance = RegisterStructure{Timestamp: now, SignupTimestamp: now,}
fmt.Printf("registerInstance after set to var now here is %v", registerInstance)
fmt.Println()
dec := json.NewDecoder(bytes.NewReader(requestBody))
err = dec.Decode(&registerInstance)
if err != nil {
fmt.Printf("error happens here.")
log.Fatal(err)
}

Ok. Here is a reproducible example that demonstrates the error you're seeing.
package main
import (
"encoding/json"
"fmt"
"bytes"
"time"
)
type RegisterStructure struct {
SignupTimestamp time.Time `json:"signuptimestamp,omitempty"`
}
func main() {
requestBody := []byte(`{"signuptimestamp" : null}`)
dec := json.NewDecoder(bytes.NewReader(requestBody))
registerInstance := RegisterStructure{}
err := dec.Decode(&registerInstance)
if err != nil {
fmt.Println(err)
}
}
The error you're seeing has nothing to do with having multiple timestamps. This is why showing inputs is critical for debugging situations like this, and why you should go back and change your question to include a sample requestBody as part of the question content. Otherwise, it becomes very difficult to guess what you're doing.
What's happening is that null is not handled by the JSON unmarshaller for time.Time. The documentation for time.Time's unmarshaller says: UnmarshalJSON implements the json.Unmarshaler interface. The time is expected to be a quoted string in RFC 3339 format.
null is not such a value: the decoder in this case will not try to construct a "zero" value for the timestamp.
What you want to do is change the type of SignupTimestamp from time.Time to *time.Time, so that the null value can be explicitly represented as a nil pointer.
Here is an example to demonstrate:
package main
import (
"bytes"
"encoding/json"
"fmt"
"time"
)
type RegisterStructure struct {
SignupTimestamp *time.Time `json:"signuptimestamp,omitempty"`
}
func main() {
requestBodies := []string{
`{"signuptimestamp" : "1985-04-12T23:20:50.52Z"}`,
`{"signuptimestamp" : null}`,
}
for _, requestBody := range requestBodies {
dec := json.NewDecoder(bytes.NewReader([]byte(requestBody)))
registerInstance := RegisterStructure{}
err := dec.Decode(&registerInstance)
if err != nil {
fmt.Println(err)
}
fmt.Printf("%#v\n", registerInstance)
}
}

Related

Unmarshalling Dynamic JSON Data With Overlapping Fields in Golang

Sorry If i'm posting a question that has already been answered, but I can't seem to find any similar situations on here. I have a websocket client that receives dynamic json data with overlapping fields. The fact that the fields overlap has has made Unmarshalling very difficult for me.
I have structs for the data types I receive, but I need a way to check the json data before I unmarshal it to a specific struct. I was hoping that an interface could act as a temporary holder and I would then be able to match the interface to the specific struct I want to unmarshal to, but that doesn't seem possible, or I just don't know how to go about it. Here are a few examples of the data types I'm receiving and structs to go along with it in case that helps.
response 1: {"connectionID":17973829270596587247,"event":"systemStatus","status":"online","version":"1.9.0"}
response 2: {"channelID":328,"channelName":"ohlc-5","event":"subscriptionStatus","pair":"XBT/USD","status":"subscribed","subscription":{"interval":5,"name":"ohlc"}}
response 3: [328,["1649576721.042916","1649577000.000000","42641.50000","42641.50000","42641.50000","42641.50000","42641.50000","0.00335101",2],"ohlc-5","XBT/USD"]
response 4: {"event":"heartbeat"}
structs below
import (
"time"
"encoding/json"
)
type ConnStatus struct {
ConnectionID uint64 `json:"connectionID"`
Event string `json:"event"`
Status string `json:"status"`
Version string `json:"version"`
}
type HeartBeat struct {
Event string `json:"event"`
}
type OHLCsuccess struct {
ChannelID int `json:"channelID"`
ChannelName string `json:"channelName"`
Event string `json:"event"`
Pair string `json:"pair"`
Status string `json:"status"`
Subscription OHLC `json:"subscription"`
}
type OHLC struct {
Interval int `json:"interval"`
Name string `json:"name"`
}
type OHLCUpdates struct {
ChannelID int
OHLCArray OHLCNewTrade
ChannelName string
Pair string
}
type OHLCNewTrade struct {
StartTime UnixTime
EndTime UnixTime
Open float64
High float64
Low float64
Close float64
VWAP float64
Volume float64
Count int
}
type UnixTime struct {
time.Time
}
func (u *UnixTime) UnmarshalJSON(d []byte) error {
var ts int64
err := json.Unmarshal(d, &ts)
if err != nil {
return err
}
u.Time = time.Unix(ts, 0).UTC()
return nil
}
Any idea(s) on how to go about this? Thanks in advance for the help!
Are you in control of the different responses? If so, wow about adding a "type" field to the top level?
See "How to put everything at the top level" section on https://eagain.net/articles/go-dynamic-json/ for more info.
E.g. (untested):
func UnmarshalJSON(d []byte) error {
var jsonValue map[string]interface{}
err := json.Unmarshal(d, &jsonValue)
if err != nil {
return err
}
switch jsonValue["type"] {
case 1:
// unmarshal into struct type 1
case 2:
// unmarshal into struct type 2
default:
// throw err
}
// or if you don't have access to type:
if jsonValue["connectionID"] != nil {
// unmarshal into struct type 1
}
return nil
}
Alternatively you could try to (strictly) unmarshal into each struct, until you don't get an error, e.g. something like:
func DetermineStruct(d []byte) int {
var connStatus *ConnStatus
reader := bytes.NewReader(d)
decoder := json.NewDecoder(reader)
decoder.DisallowUnknownFields()
err := decoder.Decode(connStatus)
if err == nil {
panic(err)
}
err = json.Unmarshal(d, &connStatus)
if err == nil {
return 1
}
var ohlcSuccess OHLCsuccess
err = json.Unmarshal(d, &ohlcSuccess)
if err == nil {
return 2
}
}

Unmarshalling nested JSON objects with dates in Golang

I'm a noob with Golang. I managed to get some things done with lots of effort.
I'm dealing with JSON files containing dates in a nested way.
I came across some workaround to unmarshal dates from JSON data into time.Time but I'm having a hard time dealing with nested ones.
The following code (obtained here in StackOverflow) is easy to understand since creates a user-defined function to parse the time objects first to a string and then to time.Time with time.Parse.
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
const dateFormat = "2006-01-02"
const data = `{
"name": "Gopher",
"join_date": "2007-09-20"
}`
type User struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
}
func (u *User) UnmarshalJSON(p []byte) error {
var aux struct {
Name string `json:"name"`
JoinDate string `json:"join_date"`
}
err := json.Unmarshal(p, &aux)
if err != nil {
return err
}
t, err := time.Parse(dateFormat, aux.JoinDate)
if err != nil {
return err
}
u.Name = aux.Name
u.JoinDate = t
return nil
}
func main() {
var u User
err := json.Unmarshal([]byte(data), &u)
if err != nil {
log.Fatal(err)
}
fmt.Println(u.JoinDate.Format(time.RFC3339))
}
So far, so good.
Now I would like to extend it in order to handle the nested date fields in the JSON, like the example below:
[{
"name": "Gopher",
"join_date": "2007-09-20",
"cashflow": [
{"date": "2021-02-25",
"amount": 100},
{"date": "2021-03-25",
"amount": 105}
]
}]
The struct that I would like to get is:
type Record []struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
Cashflow []struct {
Date time.Time `json:"date"`
Amount int `json:"amount"`
} `json:"cashflow"`
}
Thanks for the help.
To solve this using the patterns you've already got, you can write a separate unmarshalling function for the inner struct. You can do that by hoisting the inner struct to its own named struct, and then writing the function.
type CashflowRec struct {
Date time.Time `json:"date"`
Amount int `json:"amount"`
}
type Record struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
Cashflow []CashflowRec `json:"cashflow"`
}
You've already shown how to write the unmarshalling function for CashflowRec, it looks almost the same as your User function. The unmarshalling function for Record will make use of that when it calls
func (u *Record) UnmarshalJSON(p []byte) error {
var aux struct {
Name string `json:"name"`
JoinDate string `json:"join_date"`
Cashflow []CashflowRec `json:"cashflow"`
}
err := json.Unmarshal(p, &aux)
Working example: https://go.dev/play/p/1X7BJ4NETM0
aside 1 Something amusing I learned while looking at this: because you've provided your own unmarshalling function, you don't actually need the json tags in your original structs. Those are hints for the unmarshaller that the json package provides. You should probably still leave them in, in case you have to marshal the struct later. Here's it working without those tags: https://go.dev/play/p/G2VWopO_A3t
aside 2 You might find it simpler not to use time.Time, but instead create a new type of your own, and then give that type its own unmarshaller. This gives you the interesting choice for writing only that one unmarshaller, but whether or not this is a win depends on what else you do with the struct later on. Working example that still uses your nested anonymous structs: https://go.dev/play/p/bJUcaw3_r41
type dateType time.Time
type Record struct {
Name string `json:"name"`
JoinDate dateType `json:"join_date"`
Cashflow []struct {
Date dateType `json:"date"`
Amount int `json:"amount"`
} `json:"cashflow"`
}
func (c *dateType) UnmarshalJSON(p []byte) error {
var s string
if err := json.Unmarshal(p, &s); err != nil {
return err
}
t, err := time.Parse(dateFormat, s)
if err != nil {
return err
}
*c = dateType(t)
return nil
}

String from struct pointer

As a student of Go I encountered this problem.
My ultimate goal of doing this is to convert *blockchain into a valid JSON string.
My structs are:
type Blockchain struct{
blocks []Block `json:"blocks"`
difficulty int `json:"difficulty"`
}
type Block struct{
index int `json:"index"`
timestamp string `json:"timestamp"`
data string `json:"data"`
previousHash string `json:"previousHash"`
hash string `json:"hash"`
nonce int `json:"nonce"`
}
I have the following code:
var s = fmt.Sprintf("%#v", *blockchain)
print(s)
Which gives me the following:
main.Blockchain{blocks:[]main.Block{main.Block{index:1, timestamp:"2019-04-06 12:50:54", data:"Genesis block", previousHash:"", hash:"eca16d7bdd20a91f471fc3231fa5de7d892fb540789673d64f29a7b93719b74b", nonce:0}, main.Block{index:2, timestamp:"2019-04-06 12:50:54", data:"d.duck", previousHash:"eca16d7bdd20a91f471fc3231fa5de7d892fb540789673d64f29a7b93719b74b", hash:"2096ccfa6fdd8305f0e31c2e6858173a21764be4c8e1d3d50c9c31193bf06a2a", nonce:0}, main.Block{index:3, timestamp:"2019-04-06 12:50:54", data:"dumbo", previousHash:"2096ccfa6fdd8305f0e31c2e6858173a21764be4c8e1d3d50c9c31193bf06a2a", hash:"d76d4a002c6dde01009e3122aa1ccfb455e1d453ac83e2a0eb123c6080943cdb", nonce:0}}, difficulty:4}
Obviously invalid JSON.
Any suggestions?
I also tried the following:
var json, err = json.Marshal(*blockchain)
if err != nil{
panic(err.Error())
}
var s = fmt.Sprintf("%#v", json)
print(s)
It gave me the following:
[]byte{0x7b, 0x7d}
The fields on structs need to be exported (start with capital letter). This is required because JSON marshalling uses reflection, and that requires the field to be exported (to be visible).
Also json.Marshal first return value is the JSON.
package main
import (
"encoding/json"
"fmt"
"log"
)
type Blockchain struct {
Blocks []Block `json:"blocks"`
Difficulty int `json:"difficulty"`
}
type Block struct {
Index int `json:"index"`
Timestamp string `json:"timestamp"`
Data string `json:"data"`
PreviousHash string `json:"previousHash"`
Hash string `json:"hash"`
Nonce int `json:"nonce"`
}
func main() {
bc := &Blockchain{
Blocks: []Block{
Block{},
},
Difficulty: 1,
}
v, err := json.Marshal(bc)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(v))
}
This prints:
{"blocks":[{"index":0,"timestamp":"","data":"","previousHash":"","hash":"","nonce":0}],"difficulty":1}

Why doesn't JSON parsing fail with completely different type passed to Decode()?

I have the following data structures which I'd like to parse from an API:
type OrderBook struct {
Pair string `json:"pair"`
UpdateTime int64 `json:"update_time"`
}
type depthResponse struct {
Result OrderBook `json:"result"`
// doesn't matter here
//Cmd string `json:"-"`
}
and when I parse the following:
data := `{"error":{"code":"3016","msg":"交易对错误"},"cmd":"depth"}`
It doesn't fail. Why?
Full source code (playground)
package main
import (
"encoding/json"
"fmt"
"log"
"strings"
)
type OrderBook struct {
Pair string `json:"pair"`
UpdateTime int64 `json:"update_time"`
}
type depthResponse struct {
Result OrderBook `json:"result"`
}
func main() {
data := `{"error":{"code":"3016","msg":"交易对错误"},"cmd":"depth"}`
r := strings.NewReader(data)
var resp depthResponse
if err := json.NewDecoder(r).Decode(&resp); err != nil {
log.Fatalf("We should end up here: %v", err)
}
fmt.Printf("%+v\n", resp)
}
That's the expected behaviour of Decode (as documented in the Unmarshal function):
https://golang.org/pkg/encoding/json/#Unmarshal
By default, object keys which don't have a corresponding struct field are ignored.
You can however use the DisallowUnknownFields() function (as described in the docs as well) to have it fail if the input JSON has fields not contained in the destination struct.
dec := json.NewDecoder(r)
dec.DisallowUnknownFields()
In that case, you'll get an error as you expect.
Modified playground here: https://play.golang.org/p/A0f6dxTXV34

Create a struct from a byte array

I use the json.Marshal interface to accept a map[string]interface{} and convert it to a []byte (is this a byte array?)
data, _ := json.Marshal(value)
log.Printf("%s\n", data)
I get this output
{"email_address":"joe#me.com","street_address":"123 Anywhere Anytown","name":"joe","output":"Hello World","status":1}
The underlying bytes pertain to the struct of the below declaration
type Person struct {
Name string `json:"name"`
StreetAddress string `json:"street_address"`
Output string `json:"output"`
Status float64 `json:"status"`
EmailAddress string `json:"email_address",omitempty"`
}
I'd like to take data and generate a variable of type Person struct
How do I do that?
You use json.Unmarshal:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
Name string `json:"name"`
StreetAddress string `json:"street_address"`
Output string `json:"output"`
Status float64 `json:"status"`
EmailAddress string `json:"email_address",omitempty"`
}
func main() {
data := []byte(`{"email_address":"joe#me.com","street_address":"123 Anywhere Anytown","name":"joe","output":"Hello World","status":1}`)
var p Person
if err := json.Unmarshal(data, &p); err != nil {
panic(err)
}
fmt.Printf("%#v\n", p)
}
Output:
main.Person{Name:"joe", StreetAddress:"123 Anywhere Anytown", Output:"Hello World", Status:1, EmailAddress:"joe#me.com"}