I have made a go playground reproducing my issue. I have some test that fail because the json Marshal method seems to lose precision during the conversion of a time.Time object.
Running the below code:
package main
import (
"fmt"
"encoding/json"
"time"
)
type myStruct struct{
Now time.Time `json:"time"`
}
func main() {
expect := &myStruct{Now: time.Now()}
fmt.Println(expect.Now.String())
byteData, err := json.Marshal(expect)
if err != nil {
panic(err.Error())
}
actual := &myStruct{}
err = json.Unmarshal(byteData, &actual)
if err != nil {
panic(err.Error())
}
fmt.Println(actual.Now.String())
}
Outputs:
2009-11-10 23:00:00 +0000 UTC m=+0.000000001
2009-11-10 23:00:00 +0000 UTC
The issue is that when I run any test involving the conversion of a time.Time object, I cannot use any standard method to compare the results (testify, reflect...).
Is there a way to either, preserve precision when marshalling a time.Time object, or 'round' the time value to a lower precision ?
use expect := &myStruct{Now: time.Now().Local()} or
expect := &myStruct{Now: time.Now().UTC()}. This will remove your precision part and you can test it successfully.
One can also use Time.Round:
Round returns t stripped of any monotonic clock reading but otherwise
unchanged.
So:
time.Now().Round(0)
will strip away the monotonic offset leaving the rest of the time components intact (timezone etc.)
Related
Background
I am learning Go and I'm trying to do some JSON unmarshaling of a datetime.
I have some JSON produced by a program I wrote in C, I am outputting what I thought was a valid ISO8601 / RFC3339 timezone offset. I'm using strftime with the following format string:
%Y-%m-%dT%H:%M:%S.%f%z
(Note that %f is not supported by strftime natively, I have a wrapper that replaces it with the nanoseconds).
This will then produce the following result:
2016-08-08T21:35:14.052975+0200
Unmarshaling this in Go however will not work:
https://play.golang.org/p/vzOXbzAwdW
package main
import (
"fmt"
"time"
)
func main() {
t, err := time.Parse(time.RFC3339Nano, "2016-08-08T21:35:14.052975+0200")
if err != nil {
panic(err)
}
fmt.Println(t)
}
Output:
panic: parsing time "2016-08-08T21:35:14.052975+0200" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "+0200" as "Z07:00"
(Working example: https://play.golang.org/p/5xcM0aHsSw)
This is because RFC3339 expects the timezone offset to be in the format 02:00 with a :, but strftime outputs it as 0200.
So I need to fix this in my C program to output the correct format.
%z The +hhmm or -hhmm numeric timezone (that is, the hour and
minute offset from UTC). (SU)
Question
However, now I have a bunch of JSON files with this incorrect format:
2016-08-08T21:35:14.052975+0200
instead of the correct (with the : in the timezone offset):
2016-08-08T21:35:14.052975+02:00
but I still want to be able to unmarshal it correctly in my Go program. Preferably two different JSON files with only this difference should parse in the exact same way.
Regarding marshaling back to JSON, the correct format should be used.
This is how I have defined it in my struct:
Time time.Time `json:"time"`
So the question is, what is the "Go" way of doing this?
Also in my code example I am using RFC3339Nano. How would I specify that in the metadata for the struct as well? As I have it now with just json:"time" will that ignore the nano seconds?
You can define your own time field type that supports both formats:
type MyTime struct {
time.Time
}
func (self *MyTime) UnmarshalJSON(b []byte) (err error) {
s := string(b)
// Get rid of the quotes "" around the value.
// A second option would be to include them
// in the date format string instead, like so below:
// time.Parse(`"`+time.RFC3339Nano+`"`, s)
s = s[1:len(s)-1]
t, err := time.Parse(time.RFC3339Nano, s)
if err != nil {
t, err = time.Parse("2006-01-02T15:04:05.999999999Z0700", s)
}
self.Time = t
return
}
type Test struct {
Time MyTime `json:"time"`
}
Try on Go Playground
In the example above we take the predefined format time.RFC3339Nano, which is defined like this:
RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
and remove the :
"2006-01-02T15:04:05.999999999Z0700"
This time format used by time.Parse is described here:
https://golang.org/pkg/time/#pkg-constants
Also see the documentation for time.Parse
https://golang.org/pkg/time/#Parse
P.S. The fact that the year 2006 is used in the time format strings is probably because the first version of Golang was released that year.
You can try https://play.golang.org/p/IsUpuTKENg
package main
import (
"fmt"
"time"
)
func main() {
t, err := time.Parse("2006-01-02T15:04:05.999999999Z0700", "2016-08-08T21:35:14.052975-0200")
if err != nil {
panic(err)
}
fmt.Println(t)
}
I'm trying to make a simple tool which parses JSON-formatted lines in a file and performs an INSERT operation into a database.
I have a struct which looks like this:
type DataBlob struct {
....
Datetime time.Time `json:"datetime, string"`
....
}
And parsing code which looks like this:
scanner := bufio.NewScanner(file)
// Loop through all lines in the file
for scanner.Scan() {
var t DataBlob
// Decode the line, parse the JSON
dec := json.NewDecoder(strings.NewReader(scanner.Text()))
if err := dec.Decode(&t);
err != nil {
panic(err)
}
// Perform the database operation
executionString: = "INSERT INTO observations (datetime) VALUES ($1)"
_, err := db.Exec(executionString, t.Datetime)
if err != nil {
panic(err)
}
}
My JSON file has lines, each containing a datetime value that looks like this:
{ "datetime": 1465793854 }
When the datetime is formatted as a Unix timestamp, the Marshaller complains:
panic: parsing time "1465793854" as ""2006-01-02T15:04:05Z07:00"": cannot parse "1465793854" as """
In the script that generates the JSON (also written in Golang), I tried simply printing the String representation of the Time.time type, producing the following:
{ "datetime": "2016-06-13 00:23:34 -0400 EDT" }
To which the Marshaller complains when I go to parse it:
panic: parsing time ""2016-06-13 00:23:34 -0400 EDT"" as ""2006-01-02T15:04:05Z07:00"": cannot parse " 00:23:34 -0400 EDT"" as "T"
If I also treat this timestamp (which looks pretty standard) as a String and avoid the Marshaling problem, Postgres complains when I try to perform the insertion:
panic: pq: invalid input syntax for type timestamp: "2016-06-13 00:23:34 -0400 EDT"
This is frustrating on a number of levels, but mainly because if I serialize a Time.time type, I would think it should still be understood at the other side of the process.
How can I go about parsing this timestamp to perform the database insertion? Apologizes for the lengthy question and thanks for your help!
JSON unmarshalling of time.Time expects date string to be in RFC 3339 format.
So in your golang program that generates the JSON, instead of simply printing the time.Time value, use Format to print it in RFC 3339 format.
t.Format(time.RFC3339)
if I serialize a Time.time type, I would think it should still be
understood at the other side of the process
If you used the Marshaller interface with the serializing, it would indeed output the date in RFC 3339 format. So the other side of the process will understand it. So you can do that as well.
d := DataBlob{Datetime: t}
enc := json.NewEncoder(fileWriter)
enc.Encode(d)
For reference, if you need custom unmarshalling with time types you need to create your own type with the UnmarshalJSON method. Here's an example:
type Timestamp struct {
time.Time
}
// UnmarshalJSON decodes an int64 timestamp into a time.Time object
func (p *Timestamp) UnmarshalJSON(bytes []byte) error {
// 1. Decode the bytes into an int64
var raw int64
err := json.Unmarshal(bytes, &raw)
if err != nil {
fmt.Printf("error decoding timestamp: %s\n", err)
return err
}
// 2. Parse the unix timestamp
p.Time = time.Unix(raw, 0)
return nil
}
Then use the type in your struct:
type DataBlob struct {
....
Datetime Timestamp `json:"datetime"`
....
}
I am trying to port some code written against go1.3 to current versions and ran into a case where the json parsing behavior is different between versions. We are using a custom unmarshaller for parsing some specific date format. It looks like recent versions pass in the string with additional quotes which 1.3 did not.
Is this a bug or an intentional change? And whats the best way of writing code which is compatible with different versions in this situation. Just go looking for all places where a custom unmarshaller is in use always strip out extra quotes if any? It would be a pity to have to do that - so I am hoping there is a better way.
package main
import "encoding/json"
import "fmt"
import "time"
type Timestamp1 time.Time
func (t *Timestamp1) UnmarshalJSON(b []byte) (err error) {
fmt.Println("String to parse as timestamp:", string(b))
parsedTime, err := time.Parse("2006-01-02T15:04:05", string(b))
if err == nil {
*t = Timestamp1(parsedTime)
return nil
} else {
return err
}
}
type S struct {
LastUpdatedDate Timestamp1 `json:"last_updated_date,string"`
}
func main() {
s := `{"last_updated_date" : "2015-11-03T10:00:00"}`
var s1 S
err := json.Unmarshal([]byte(s), &s1)
fmt.Println(err)
fmt.Println(s1)
}
There was a bug concerning json:",string" tag that was fixed in 1.5. If there isn't a particular reason you need it, you can remove it and simply adjust your format:
// N.B. time is in quotes.
parsedTime, err := time.Parse(`"2006-01-02T15:04:05"`, string(b))
Playground: http://play.golang.org/p/LgWuKcPEuI.
This should work in 1.3 as well as 1.5.
How can I parse this .Net JSON date with Go?
The value comes back unassigned.
It appears to parse up to the date field.
package main
import (
"encoding/json"
"fmt"
"time"
)
type MyStruct struct {
FirstField string
SomeTime time.Time
LastField string
}
type MyStructSlice struct {
MyStructs []MyStruct
}
func main() {
var s MyStructSlice
str := `{"MyStructs":[{"FirstField":"123", "SomeTime":"\/Date(1432187580000-0500)\/", "LastField":"456"}]}`
json.Unmarshal([]byte(str), &s)
fmt.Println(s)
}
Go Playground
I am going to provide a few suggestions. You will have to write the code yourself though ;)
First of all is it possible to change .NET application that produced this JSON to generate something more parsable? If you make it output datetime in RFC3339 format (something like 1990-12-31T15:59:12-08:00) then go will automatically converts it to time.Time instance thanks to http://golang.org/pkg/time/#Time.UnmarshalJSON
If you cannot change the client then you will have to parse this date yourself:
extract time part (1432187580000) from the string. this looks like number of milliseconds (ms) since UNIX epoch. You can convert it to time.Time instance using time.Unix(sec, nsec).
(optional). The time that was created in the last step already accurately represent a point in time. However if you want to add the original timezone to it (e.g. to print it) you will need to:
parse the offset part (-0500) from the string
create time.FixedZone
call http://golang.org/pkg/time/#Time.In on the instance of time.Time created in the first step
Example: http://play.golang.org/p/Pkahyg2vZa
First you model is wrong, it doesn't model the data structure in your JSON. It should be:
type Data struct {
Data []MyStruct `json:"data`
}
type MyStruct struct {
SomeTime string
}
It works with this, try it on the Go Playground.
Problem is that we still have the time as string.
Now if you want SomeTime to be time.Time, you need to parse it yourself, you can do it by implementing json.Unmarshaler:
type Data struct {
Data []MyStruct `json:"data`
}
type MyStruct struct {
SomeTime time.Time
}
func (m *MyStruct) UnmarshalJSON(data []byte) error {
// First unmashal it into a string:
ms := struct{ SomeTime string }{}
if err := json.Unmarshal(data, &ms); err != nil {
return err
}
s := ms.SomeTime
// s is of format: "/Date(1432187580000-0500)/"
// extract millis and time zone offset hours
i1 := strings.Index(s, "(")
i3 := strings.Index(s, ")")
i2 := strings.Index(s, "-")
if i2 < 0 {
i2 = strings.Index(s, "+")
}
if i1 < 0 || i2 < 0 || i3 < 0 {
return errors.New("Invalid format")
}
millis, err := strconv.ParseInt(s[i1+1:i2], 10, 64)
if err != nil {
return err
}
m.SomeTime = time.Unix(0, millis*1000000)
// Apply timezone:
zoneHours, err := strconv.ParseInt(s[i2:i3], 10, 64)
if err != nil {
return err
}
zone := time.FixedZone("something", int(zoneHours)*3600)
m.SomeTime = m.SomeTime.In(zone)
return nil
}
Try it on the Go Playground.
The value comes back unassigned for many reasons. First of all your MyStruct does not look right. Your JSON has data key as the parent key, which consists an array of objects.
But your struct for some reason does not even resembles this. It has to have Date as a field at least. So make your struct look like the json you are receiving.
Another thing is that your time.Time will not be able to parse this string: Date(1432187580000-0500) in a normal date.
P.S. now that you have updated your struct to normal way, you have to find a way to parse your strange sting to a date. If you have power over your .net application, I would rather recommend changing json to a normal timestamp or something that can be easily parsable.
If not, then you have to change SomeTime time.Time to SomeTime string and then parse string to a normal timestamp and then parse this timestamp to a date.
I have the following json data (from an external program, simplified a bit)
I can't change the json format.
[1416495600501595942, {"venue_id": 73, "message_type": "ABC", "sequence": 26686695}]
I'm having trouble unpacking it in Go, I think mostly because it's a list of disparate types.
The obvious thing to do seems to be to do is []interface{}, which works, but in converts it to a float64, which produces a roundoff error that I can't deal with (the number is a timestamp since epoch in nanos).
I can get it to work by unpacking it twice, as []interface{} and []int64, but that's obviously going to hinder performance, and I'm processing large amounts of data in real time.
I tried using struct here because that would get treated as a map, not a list[]
Is there any way to either pass it the format of the data, or make it default to int64 instead of float64? It's always going to be
[int64, map[string]interface{}]
ie I know the format of the upper level list, and that the keys of the map are strings, but the values could be anything (painfully, including decimals, which I think the only thing I can use to interpret them as is floats ...)
package main
import (
"encoding/json"
"fmt"
)
func main() {
j := `[1416495600501595942, {"venue_id": 73, "message_type": "ABC", "sequence": 26686695}]`
b := []byte(j)
fmt.Println(j)
var line []interface{}
var ints []int64
json.Unmarshal(b, &line)
fmt.Println(line)
// fmt.Println(line[0].(int64)) - this doesn't work
fmt.Println(line[0].(float64))
fmt.Println(int64(line[0].(float64)))
json.Unmarshal(b, &ints)
fmt.Println(ints)
}
Output is as follows:
[1416495600501595942, {"venue_id": 73, "message_type": "oKC", "sequence": 26686695}]
[1.416495600501596e+18 map[venue_id:73 message_type:oKC sequence:2.6686695e+07]]
1.416495600501596e+18
1416495600501595904
[1416495600501595942 0]
Solution (thanks to makpoc / dystroy)
package main
import (
"encoding/json"
"fmt"
"bytes"
)
func main() {
j := `[1416495600501595942, {"venue_id": 7.3, "message_type": "oKC", "sequence": 26686695}]`
b := []byte(j)
fmt.Println(j)
var line []interface{}
d := json.NewDecoder(bytes.NewBuffer(b))
d.UseNumber()
if err := d.Decode(&line); err != nil {
panic(err)
}
fmt.Println(line[0])
data := line[1].(map[string]interface{})
fmt.Println(data["venue_id"])
fmt.Println(data["sequence"])
}
According to this answer you can use Decoder -> and UseNumber or a struct instead of directly parsing the value.