I'm a noob with Golang. I managed to get some things done with lots of effort.
I'm dealing with JSON files containing dates in a nested way.
I came across some workaround to unmarshal dates from JSON data into time.Time but I'm having a hard time dealing with nested ones.
The following code (obtained here in StackOverflow) is easy to understand since creates a user-defined function to parse the time objects first to a string and then to time.Time with time.Parse.
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
const dateFormat = "2006-01-02"
const data = `{
"name": "Gopher",
"join_date": "2007-09-20"
}`
type User struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
}
func (u *User) UnmarshalJSON(p []byte) error {
var aux struct {
Name string `json:"name"`
JoinDate string `json:"join_date"`
}
err := json.Unmarshal(p, &aux)
if err != nil {
return err
}
t, err := time.Parse(dateFormat, aux.JoinDate)
if err != nil {
return err
}
u.Name = aux.Name
u.JoinDate = t
return nil
}
func main() {
var u User
err := json.Unmarshal([]byte(data), &u)
if err != nil {
log.Fatal(err)
}
fmt.Println(u.JoinDate.Format(time.RFC3339))
}
So far, so good.
Now I would like to extend it in order to handle the nested date fields in the JSON, like the example below:
[{
"name": "Gopher",
"join_date": "2007-09-20",
"cashflow": [
{"date": "2021-02-25",
"amount": 100},
{"date": "2021-03-25",
"amount": 105}
]
}]
The struct that I would like to get is:
type Record []struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
Cashflow []struct {
Date time.Time `json:"date"`
Amount int `json:"amount"`
} `json:"cashflow"`
}
Thanks for the help.
To solve this using the patterns you've already got, you can write a separate unmarshalling function for the inner struct. You can do that by hoisting the inner struct to its own named struct, and then writing the function.
type CashflowRec struct {
Date time.Time `json:"date"`
Amount int `json:"amount"`
}
type Record struct {
Name string `json:"name"`
JoinDate time.Time `json:"join_date"`
Cashflow []CashflowRec `json:"cashflow"`
}
You've already shown how to write the unmarshalling function for CashflowRec, it looks almost the same as your User function. The unmarshalling function for Record will make use of that when it calls
func (u *Record) UnmarshalJSON(p []byte) error {
var aux struct {
Name string `json:"name"`
JoinDate string `json:"join_date"`
Cashflow []CashflowRec `json:"cashflow"`
}
err := json.Unmarshal(p, &aux)
Working example: https://go.dev/play/p/1X7BJ4NETM0
aside 1 Something amusing I learned while looking at this: because you've provided your own unmarshalling function, you don't actually need the json tags in your original structs. Those are hints for the unmarshaller that the json package provides. You should probably still leave them in, in case you have to marshal the struct later. Here's it working without those tags: https://go.dev/play/p/G2VWopO_A3t
aside 2 You might find it simpler not to use time.Time, but instead create a new type of your own, and then give that type its own unmarshaller. This gives you the interesting choice for writing only that one unmarshaller, but whether or not this is a win depends on what else you do with the struct later on. Working example that still uses your nested anonymous structs: https://go.dev/play/p/bJUcaw3_r41
type dateType time.Time
type Record struct {
Name string `json:"name"`
JoinDate dateType `json:"join_date"`
Cashflow []struct {
Date dateType `json:"date"`
Amount int `json:"amount"`
} `json:"cashflow"`
}
func (c *dateType) UnmarshalJSON(p []byte) error {
var s string
if err := json.Unmarshal(p, &s); err != nil {
return err
}
t, err := time.Parse(dateFormat, s)
if err != nil {
return err
}
*c = dateType(t)
return nil
}
Related
I have the following JSON
{"student_number":1234567, "name":"John Doe", "subjects":"Chemistry-Maths-History-Geography"}
I would like to unmarshal it in a struct, where one item (the subjects) are split on '-' into a []string.
type Student struct {
StudentNumber int `json:"student_number"`
Name string `json:"name"`
Subjects []string
}
I have attempted several different ways of achieving this with custom Unmarshalling using strings.Split(), but have not succeeded so far.
Is there any way to achieve this in the unmarshalling process? Or will I need to simply unmarshal as is and make the conversion afterward?
Easiest would be to define your own string slice type and implement json.Unmarshaler on that:
type strslice []string
func (ss *strslice) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*ss = strings.Split(s, "-")
return nil
}
And use this in your struct:
type Student struct {
StudentNumber int `json:"student_number"`
Name string `json:"name"`
Subjects strslice `json:"subjects"`
}
And then it'll work:
func main() {
var s Student
err := json.Unmarshal([]byte(src), &s)
fmt.Println(s, err)
}
const src = `{"student_number":1234567, "name":"John Doe", "subjects":"Chemistry-Maths-History-Geography"}`
Output (try it on the Go Playground):
{1234567 John Doe [Chemistry Maths History Geography]} <nil>
I have a struct that I can't change and an array of these structs in a separate JSON file.
I could have parsed data from JSON file easily, but there are mismatched types in same fields:
(main.go)
import "time"
type SomeType struct {
name string `json: "name"`
time time.Time `json: "someTime"`
}
(someData.json)
[
{
"name": "some name",
"someTime": "15:20:00"
},
{
"name": "some other name",
"someTime": "23:15:00"
}
]
If "time" field was a type of a string, I would simply use json.Unmarshal and parse all of the data from json into []SomeType, but since types mismatch, I can't find a way to do it correctly.
You should add the UnmarshalJSON method to your struct
type SomeType struct {
Name string `json:"name"`
Time time.Time `json:"someTime"`
}
func (st *SomeType) UnmarshalJSON(data []byte) error {
type parseType struct {
Name string `json:"name"`
Time string `json:"someTime"`
}
var res parseType
if err := json.Unmarshal(data, &res); err != nil {
return err
}
parsed, err := time.Parse("15:04:05", res.Time)
if err != nil {
return err
}
now := time.Now()
st.Name = res.Name
st.Time = time.Date(now.Year(), now.Month(), now.Day(), parsed.Hour(), parsed.Minute(), parsed.Second(), 0, now.Location())
return nil
}
GoPlay
I think you should define a custom time type, with a custom unmarshaller as follows:
type CustomTime struct {
time.Time
}
func (t *CustomTime) UnmarshalJSON(b []byte) error {
formattedTime, err := time.Parse(`"15:04:05"`, string(b))
t.Time = formattedTime
return err
}
And have another struct, which is basically the exact struct you have, instead it uses your custom time type rather than the original struct which uses time.Time:
type SameTypeWithDifferentTime struct {
Name string `json:"name"`
SomeTime CustomTime `json:"someTime"`
}
func (s SameTypeWithDifferentTime) ToOriginal() SomeType {
return SomeType {
Name: s.Name,
SomeTime: s.SomeTime.Time,
}
}
The rest of the process is pretty straight forward, you just deserialize your json to the new type, and use ToOriginal() receiver function to convert it to your original struct. Another point to mention here is that your not exporting your struct fields.
Sorry If i'm posting a question that has already been answered, but I can't seem to find any similar situations on here. I have a websocket client that receives dynamic json data with overlapping fields. The fact that the fields overlap has has made Unmarshalling very difficult for me.
I have structs for the data types I receive, but I need a way to check the json data before I unmarshal it to a specific struct. I was hoping that an interface could act as a temporary holder and I would then be able to match the interface to the specific struct I want to unmarshal to, but that doesn't seem possible, or I just don't know how to go about it. Here are a few examples of the data types I'm receiving and structs to go along with it in case that helps.
response 1: {"connectionID":17973829270596587247,"event":"systemStatus","status":"online","version":"1.9.0"}
response 2: {"channelID":328,"channelName":"ohlc-5","event":"subscriptionStatus","pair":"XBT/USD","status":"subscribed","subscription":{"interval":5,"name":"ohlc"}}
response 3: [328,["1649576721.042916","1649577000.000000","42641.50000","42641.50000","42641.50000","42641.50000","42641.50000","0.00335101",2],"ohlc-5","XBT/USD"]
response 4: {"event":"heartbeat"}
structs below
import (
"time"
"encoding/json"
)
type ConnStatus struct {
ConnectionID uint64 `json:"connectionID"`
Event string `json:"event"`
Status string `json:"status"`
Version string `json:"version"`
}
type HeartBeat struct {
Event string `json:"event"`
}
type OHLCsuccess struct {
ChannelID int `json:"channelID"`
ChannelName string `json:"channelName"`
Event string `json:"event"`
Pair string `json:"pair"`
Status string `json:"status"`
Subscription OHLC `json:"subscription"`
}
type OHLC struct {
Interval int `json:"interval"`
Name string `json:"name"`
}
type OHLCUpdates struct {
ChannelID int
OHLCArray OHLCNewTrade
ChannelName string
Pair string
}
type OHLCNewTrade struct {
StartTime UnixTime
EndTime UnixTime
Open float64
High float64
Low float64
Close float64
VWAP float64
Volume float64
Count int
}
type UnixTime struct {
time.Time
}
func (u *UnixTime) UnmarshalJSON(d []byte) error {
var ts int64
err := json.Unmarshal(d, &ts)
if err != nil {
return err
}
u.Time = time.Unix(ts, 0).UTC()
return nil
}
Any idea(s) on how to go about this? Thanks in advance for the help!
Are you in control of the different responses? If so, wow about adding a "type" field to the top level?
See "How to put everything at the top level" section on https://eagain.net/articles/go-dynamic-json/ for more info.
E.g. (untested):
func UnmarshalJSON(d []byte) error {
var jsonValue map[string]interface{}
err := json.Unmarshal(d, &jsonValue)
if err != nil {
return err
}
switch jsonValue["type"] {
case 1:
// unmarshal into struct type 1
case 2:
// unmarshal into struct type 2
default:
// throw err
}
// or if you don't have access to type:
if jsonValue["connectionID"] != nil {
// unmarshal into struct type 1
}
return nil
}
Alternatively you could try to (strictly) unmarshal into each struct, until you don't get an error, e.g. something like:
func DetermineStruct(d []byte) int {
var connStatus *ConnStatus
reader := bytes.NewReader(d)
decoder := json.NewDecoder(reader)
decoder.DisallowUnknownFields()
err := decoder.Decode(connStatus)
if err == nil {
panic(err)
}
err = json.Unmarshal(d, &connStatus)
if err == nil {
return 1
}
var ohlcSuccess OHLCsuccess
err = json.Unmarshal(d, &ohlcSuccess)
if err == nil {
return 2
}
}
I have a question in Go especially with gin-gionic and gorm.
Let's say I have model like this
// Classroom struct.
type Classroom struct {
gorm.Model
Name string `json:"name"`
Code string `json:"code"`
StartedAt time.Time `json:"started_at"`
}
I want to create data of Classroom Model with this JSON
{
"name": "Math",
"code": "math-mr-robie",
"started_at": "2020-10-10 10:00:00"
}
But when I bind the JSON data, I got this following error
parsing time ""2020-10-10 10:00:00"" as ""2006-01-02T15:04:05Z07:00"": cannot parse " 10:00:00"" as "T"
I know that error appear because of the format that I sent was not the exact format of time.Time?
Is it possible to set default format of time.Time?
How to do that?
Because I've try to add .Format in after time.Time but error occurs.
// Classroom struct.
type Classroom struct {
gorm.Model
Name string `json:"name"`
Code string `json:"code"`
StartedAt time.Time.Format("2006-01-02 15:04:05") `json:"started_at"`
}
I resolve this issue by creating new struct JSONData that contain time inside it.
// JSONData struct.
type JSONData struct {
Time time.Time
}
After I red Customize Data Types in Gorm and see some examples here then I add some methods
// Scan JSONDate.
func (j *JSONDate) Scan(value interface{}) (err error) {
nullTime := &sql.NullTime{}
err = nullTime.Scan(value)
*j = JSONDate{nullTime.Time}
return
}
// Value JSONDate.
func (j JSONDate) Value() (driver.Value, error) {
y, m, d := time.Time(j.Time).Date()
return time.Date(y, m, d, 0, 0, 0, 0, time.Time(j.Time).Location()), nil
}
// GormDataType gorm common data type
func (j JSONDate) GormDataType() string {
return "timestamp"
}
For the gin things. Another resource #Eklavya given. So I add another methods.
// UnmarshalJSON JSONDate.
func (j *JSONDate) UnmarshalJSON(b []byte) error {
s := strings.Trim(string(b), "\"")
t, err := time.Parse(helpers.YMDHIS, s)
if err != nil {
return err
}
*j = JSONDate{
Time: t,
}
return nil
}
// MarshalJSON JSONDate.
func (j JSONDate) MarshalJSON() ([]byte, error) {
return []byte("\"" + j.Time.Format(helpers.YMDHIS) + "\""), nil
}
// Format method.
func (j JSONDate) Format(s string) string {
t := time.Time(j.Time)
return t.Format(helpers.YMDHIS)
}
And it's works!
I came across this question with the same issue and found that if you are looking for a specific time format like the ISOStrings sent from the browser you can have something like this;
type Classroom struct {
gorm.Model
Name string `json:"name"`
Code string `json:"code"`
StartedAt time.Time `json:"started_at" time_format:"RFC3339"`
}
With the time_format I did not need to define marshal functions to handle the formatting. However, if you need to do a completely custom format for your date and time then I believe you will need to define these functions.
I need to unmarshal json object which may have the following formats:
Format1:
{
"contactType": 2,
"value": "0123456789"
}
Format2:
{
"contactType": "MobileNumber",
"value": "0123456789"
}
The structure I'm using for unmarshalling is:-
type Contact struct {
ContactType int `json:"contactType"`
Value string `json:"value"`
}
But this works only for format 1. I don't want to change the datatype of ContactType but I want to accommodate the 2nd format as well. I heard about json.RawMarshal and tried using it.
type Contact struct {
ContactType int
Value string `json:"value"`
Type json.RawMessage `json:"contactType"`
}
type StringContact struct {
Type string `json:"contactType"`
}
type IntContact struct {
Type int `json:"contactType"`
}
This gets the unmarshalling done, but I'm unable to set the ContactType variable which depends on the type of json.RawMessage. How do I model my structure so that this problem gets solved?
You will need to do the unmarshalling yourself. There is a very good article that shows how to use the json.RawMessage right and a number of other solutions to this very problem, Like using interfaces, RawMessage, implemention your own unmarshal and decode functions etc.
You will find the article here: JSON decoding in GO by Attila Oláh
Note: Attila has made a few errors on his code examples.
I taken the liberty to put together (using some of the code from Attila) a working example using RawMessage to delay the unmarshaling so we can do it on our own version of the Decode func.
Link to GOLANG Playground
package main
import (
"fmt"
"encoding/json"
"io"
)
type Record struct {
AuthorRaw json.RawMessage `json:"author"`
Title string `json:"title"`
URL string `json:"url"`
Author Author
}
type Author struct {
ID uint64 `json:"id"`
Email string `json:"email"`
}
func Decode(r io.Reader) (x *Record, err error) {
x = new(Record)
if err = json.NewDecoder(r).Decode(x); err != nil {
return
}
if err = json.Unmarshal(x.AuthorRaw, &x.Author); err == nil {
return
}
var s string
if err = json.Unmarshal(x.AuthorRaw, &s); err == nil {
x.Author.Email = s
return
}
var n uint64
if err = json.Unmarshal(x.AuthorRaw, &n); err == nil {
x.Author.ID = n
}
return
}
func main() {
byt_1 := []byte(`{"author": 2,"title": "some things","url": "https://stackoverflow.com"}`)
byt_2 := []byte(`{"author": "Mad Scientist","title": "some things","url": "https://stackoverflow.com"}`)
var dat Record
if err := json.Unmarshal(byt_1, &dat); err != nil {
panic(err)
}
fmt.Printf("%#s\r\n", dat)
if err := json.Unmarshal(byt_2, &dat); err != nil {
panic(err)
}
fmt.Printf("%#s\r\n", dat)
}
Hope this helps.