Unmarshal incorrectly formatted datetime - json

Background
I am learning Go and I'm trying to do some JSON unmarshaling of a datetime.
I have some JSON produced by a program I wrote in C, I am outputting what I thought was a valid ISO8601 / RFC3339 timezone offset. I'm using strftime with the following format string:
%Y-%m-%dT%H:%M:%S.%f%z
(Note that %f is not supported by strftime natively, I have a wrapper that replaces it with the nanoseconds).
This will then produce the following result:
2016-08-08T21:35:14.052975+0200
Unmarshaling this in Go however will not work:
https://play.golang.org/p/vzOXbzAwdW
package main
import (
"fmt"
"time"
)
func main() {
t, err := time.Parse(time.RFC3339Nano, "2016-08-08T21:35:14.052975+0200")
if err != nil {
panic(err)
}
fmt.Println(t)
}
Output:
panic: parsing time "2016-08-08T21:35:14.052975+0200" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "+0200" as "Z07:00"
(Working example: https://play.golang.org/p/5xcM0aHsSw)
This is because RFC3339 expects the timezone offset to be in the format 02:00 with a :, but strftime outputs it as 0200.
So I need to fix this in my C program to output the correct format.
%z The +hhmm or -hhmm numeric timezone (that is, the hour and
minute offset from UTC). (SU)
Question
However, now I have a bunch of JSON files with this incorrect format:
2016-08-08T21:35:14.052975+0200
instead of the correct (with the : in the timezone offset):
2016-08-08T21:35:14.052975+02:00
but I still want to be able to unmarshal it correctly in my Go program. Preferably two different JSON files with only this difference should parse in the exact same way.
Regarding marshaling back to JSON, the correct format should be used.
This is how I have defined it in my struct:
Time time.Time `json:"time"`
So the question is, what is the "Go" way of doing this?
Also in my code example I am using RFC3339Nano. How would I specify that in the metadata for the struct as well? As I have it now with just json:"time" will that ignore the nano seconds?

You can define your own time field type that supports both formats:
type MyTime struct {
time.Time
}
func (self *MyTime) UnmarshalJSON(b []byte) (err error) {
s := string(b)
// Get rid of the quotes "" around the value.
// A second option would be to include them
// in the date format string instead, like so below:
// time.Parse(`"`+time.RFC3339Nano+`"`, s)
s = s[1:len(s)-1]
t, err := time.Parse(time.RFC3339Nano, s)
if err != nil {
t, err = time.Parse("2006-01-02T15:04:05.999999999Z0700", s)
}
self.Time = t
return
}
type Test struct {
Time MyTime `json:"time"`
}
Try on Go Playground
In the example above we take the predefined format time.RFC3339Nano, which is defined like this:
RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
and remove the :
"2006-01-02T15:04:05.999999999Z0700"
This time format used by time.Parse is described here:
https://golang.org/pkg/time/#pkg-constants
Also see the documentation for time.Parse
https://golang.org/pkg/time/#Parse
P.S. The fact that the year 2006 is used in the time format strings is probably because the first version of Golang was released that year.

You can try https://play.golang.org/p/IsUpuTKENg
package main
import (
"fmt"
"time"
)
func main() {
t, err := time.Parse("2006-01-02T15:04:05.999999999Z0700", "2016-08-08T21:35:14.052975-0200")
if err != nil {
panic(err)
}
fmt.Println(t)
}

Related

Avoid precision loss when marshalling time.Time objects

I have made a go playground reproducing my issue. I have some test that fail because the json Marshal method seems to lose precision during the conversion of a time.Time object.
Running the below code:
package main
import (
"fmt"
"encoding/json"
"time"
)
type myStruct struct{
Now time.Time `json:"time"`
}
func main() {
expect := &myStruct{Now: time.Now()}
fmt.Println(expect.Now.String())
byteData, err := json.Marshal(expect)
if err != nil {
panic(err.Error())
}
actual := &myStruct{}
err = json.Unmarshal(byteData, &actual)
if err != nil {
panic(err.Error())
}
fmt.Println(actual.Now.String())
}
Outputs:
2009-11-10 23:00:00 +0000 UTC m=+0.000000001
2009-11-10 23:00:00 +0000 UTC
The issue is that when I run any test involving the conversion of a time.Time object, I cannot use any standard method to compare the results (testify, reflect...).
Is there a way to either, preserve precision when marshalling a time.Time object, or 'round' the time value to a lower precision ?
use expect := &myStruct{Now: time.Now().Local()} or
expect := &myStruct{Now: time.Now().UTC()}. This will remove your precision part and you can test it successfully.
One can also use Time.Round:
Round returns t stripped of any monotonic clock reading but otherwise
unchanged.
So:
time.Now().Round(0)
will strip away the monotonic offset leaving the rest of the time components intact (timezone etc.)

Golang parse complex json

I am new to golang and json and currently struggle to parse the json out from a system.
I've read a couple of blog posts on dynamic json in go and also tried the tools like json2GoStructs
Parsing my json file with this tools just gave me a huge structs which I found a bit messy. Also I had no idea how to get the info im interested in.
So, here are my problems:
How do I get to the info I am interested in?
What is the best approach to parse complex json?
I am only interested into the following 3 json fields:
Name
Guid
Task -> Property -> Name: Error
I'm thankful for every tip, code snippet or explanation!
This is what I got so far (mostly from a tutorial):
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
func checkErr(err error) {
if err != nil {
panic(err)
}
}
func readFile(filePath string) []byte {
data, err := ioutil.ReadFile(filePath)
checkErr(err)
return data
}
func main() {
path := "/Users/andi/Documents/tmp/wfsJob.json"
data := readFile(path)
var f interface{}
err := json.Unmarshal(data, &f)
checkErr(err)
m := f.(map[string]interface{})
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
}
I can offer you this easy way to using JSON in Golang. With this tool you don't need to parse the whole json file, and you can use it without struct.
Gjson is a great solution for fetching a few fields from JSON string. But it may become slow when many (more than 2) fields must be fetched from distinct parts of the JSON, since it re-parses the JSON on each Get call. Additionally, it requires calling gjson.Valid for validating the incoming JSON, since other methods assume the caller provides valid JSON.
There is an alternative package - fastjson. Like gsjon, it is fast and has nice API. Unlike gjson it validates the input JSON and works faster when many unrelated fields must be obtained from the JSON. Here is a sample code for obtaining fields from the original question:
var p fastjson.Parser
v, err := p.ParseBytes(data)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Name: %s", v.GetStringBytes("Name"))
fmt.Printf("Guid: %s", v.GetStringBytes("Guid"))
fmt.Printf("Error: %s", v.GetStringBytes("Task", "Property", "Name"))

Fully Parsing Timestamps in Golang

I'm trying to make a simple tool which parses JSON-formatted lines in a file and performs an INSERT operation into a database.
I have a struct which looks like this:
type DataBlob struct {
....
Datetime time.Time `json:"datetime, string"`
....
}
And parsing code which looks like this:
scanner := bufio.NewScanner(file)
// Loop through all lines in the file
for scanner.Scan() {
var t DataBlob
// Decode the line, parse the JSON
dec := json.NewDecoder(strings.NewReader(scanner.Text()))
if err := dec.Decode(&t);
err != nil {
panic(err)
}
// Perform the database operation
executionString: = "INSERT INTO observations (datetime) VALUES ($1)"
_, err := db.Exec(executionString, t.Datetime)
if err != nil {
panic(err)
}
}
My JSON file has lines, each containing a datetime value that looks like this:
{ "datetime": 1465793854 }
When the datetime is formatted as a Unix timestamp, the Marshaller complains:
panic: parsing time "1465793854" as ""2006-01-02T15:04:05Z07:00"": cannot parse "1465793854" as """
In the script that generates the JSON (also written in Golang), I tried simply printing the String representation of the Time.time type, producing the following:
{ "datetime": "2016-06-13 00:23:34 -0400 EDT" }
To which the Marshaller complains when I go to parse it:
panic: parsing time ""2016-06-13 00:23:34 -0400 EDT"" as ""2006-01-02T15:04:05Z07:00"": cannot parse " 00:23:34 -0400 EDT"" as "T"
If I also treat this timestamp (which looks pretty standard) as a String and avoid the Marshaling problem, Postgres complains when I try to perform the insertion:
panic: pq: invalid input syntax for type timestamp: "2016-06-13 00:23:34 -0400 EDT"
This is frustrating on a number of levels, but mainly because if I serialize a Time.time type, I would think it should still be understood at the other side of the process.
How can I go about parsing this timestamp to perform the database insertion? Apologizes for the lengthy question and thanks for your help!
JSON unmarshalling of time.Time expects date string to be in RFC 3339 format.
So in your golang program that generates the JSON, instead of simply printing the time.Time value, use Format to print it in RFC 3339 format.
t.Format(time.RFC3339)
if I serialize a Time.time type, I would think it should still be
understood at the other side of the process
If you used the Marshaller interface with the serializing, it would indeed output the date in RFC 3339 format. So the other side of the process will understand it. So you can do that as well.
d := DataBlob{Datetime: t}
enc := json.NewEncoder(fileWriter)
enc.Encode(d)
For reference, if you need custom unmarshalling with time types you need to create your own type with the UnmarshalJSON method. Here's an example:
type Timestamp struct {
time.Time
}
// UnmarshalJSON decodes an int64 timestamp into a time.Time object
func (p *Timestamp) UnmarshalJSON(bytes []byte) error {
// 1. Decode the bytes into an int64
var raw int64
err := json.Unmarshal(bytes, &raw)
if err != nil {
fmt.Printf("error decoding timestamp: %s\n", err)
return err
}
// 2. Parse the unix timestamp
p.Time = time.Unix(raw, 0)
return nil
}
Then use the type in your struct:
type DataBlob struct {
....
Datetime Timestamp `json:"datetime"`
....
}

Differences in parsing json with a custom unmarshaller between golang versions

I am trying to port some code written against go1.3 to current versions and ran into a case where the json parsing behavior is different between versions. We are using a custom unmarshaller for parsing some specific date format. It looks like recent versions pass in the string with additional quotes which 1.3 did not.
Is this a bug or an intentional change? And whats the best way of writing code which is compatible with different versions in this situation. Just go looking for all places where a custom unmarshaller is in use always strip out extra quotes if any? It would be a pity to have to do that - so I am hoping there is a better way.
package main
import "encoding/json"
import "fmt"
import "time"
type Timestamp1 time.Time
func (t *Timestamp1) UnmarshalJSON(b []byte) (err error) {
fmt.Println("String to parse as timestamp:", string(b))
parsedTime, err := time.Parse("2006-01-02T15:04:05", string(b))
if err == nil {
*t = Timestamp1(parsedTime)
return nil
} else {
return err
}
}
type S struct {
LastUpdatedDate Timestamp1 `json:"last_updated_date,string"`
}
func main() {
s := `{"last_updated_date" : "2015-11-03T10:00:00"}`
var s1 S
err := json.Unmarshal([]byte(s), &s1)
fmt.Println(err)
fmt.Println(s1)
}
There was a bug concerning json:",string" tag that was fixed in 1.5. If there isn't a particular reason you need it, you can remove it and simply adjust your format:
// N.B. time is in quotes.
parsedTime, err := time.Parse(`"2006-01-02T15:04:05"`, string(b))
Playground: http://play.golang.org/p/LgWuKcPEuI.
This should work in 1.3 as well as 1.5.

Parse .Net JSON date with Go

How can I parse this .Net JSON date with Go?
The value comes back unassigned.
It appears to parse up to the date field.
package main
import (
"encoding/json"
"fmt"
"time"
)
type MyStruct struct {
FirstField string
SomeTime time.Time
LastField string
}
type MyStructSlice struct {
MyStructs []MyStruct
}
func main() {
var s MyStructSlice
str := `{"MyStructs":[{"FirstField":"123", "SomeTime":"\/Date(1432187580000-0500)\/", "LastField":"456"}]}`
json.Unmarshal([]byte(str), &s)
fmt.Println(s)
}
Go Playground
I am going to provide a few suggestions. You will have to write the code yourself though ;)
First of all is it possible to change .NET application that produced this JSON to generate something more parsable? If you make it output datetime in RFC3339 format (something like 1990-12-31T15:59:12-08:00) then go will automatically converts it to time.Time instance thanks to http://golang.org/pkg/time/#Time.UnmarshalJSON
If you cannot change the client then you will have to parse this date yourself:
extract time part (1432187580000) from the string. this looks like number of milliseconds (ms) since UNIX epoch. You can convert it to time.Time instance using time.Unix(sec, nsec).
(optional). The time that was created in the last step already accurately represent a point in time. However if you want to add the original timezone to it (e.g. to print it) you will need to:
parse the offset part (-0500) from the string
create time.FixedZone
call http://golang.org/pkg/time/#Time.In on the instance of time.Time created in the first step
Example: http://play.golang.org/p/Pkahyg2vZa
First you model is wrong, it doesn't model the data structure in your JSON. It should be:
type Data struct {
Data []MyStruct `json:"data`
}
type MyStruct struct {
SomeTime string
}
It works with this, try it on the Go Playground.
Problem is that we still have the time as string.
Now if you want SomeTime to be time.Time, you need to parse it yourself, you can do it by implementing json.Unmarshaler:
type Data struct {
Data []MyStruct `json:"data`
}
type MyStruct struct {
SomeTime time.Time
}
func (m *MyStruct) UnmarshalJSON(data []byte) error {
// First unmashal it into a string:
ms := struct{ SomeTime string }{}
if err := json.Unmarshal(data, &ms); err != nil {
return err
}
s := ms.SomeTime
// s is of format: "/Date(1432187580000-0500)/"
// extract millis and time zone offset hours
i1 := strings.Index(s, "(")
i3 := strings.Index(s, ")")
i2 := strings.Index(s, "-")
if i2 < 0 {
i2 = strings.Index(s, "+")
}
if i1 < 0 || i2 < 0 || i3 < 0 {
return errors.New("Invalid format")
}
millis, err := strconv.ParseInt(s[i1+1:i2], 10, 64)
if err != nil {
return err
}
m.SomeTime = time.Unix(0, millis*1000000)
// Apply timezone:
zoneHours, err := strconv.ParseInt(s[i2:i3], 10, 64)
if err != nil {
return err
}
zone := time.FixedZone("something", int(zoneHours)*3600)
m.SomeTime = m.SomeTime.In(zone)
return nil
}
Try it on the Go Playground.
The value comes back unassigned for many reasons. First of all your MyStruct does not look right. Your JSON has data key as the parent key, which consists an array of objects.
But your struct for some reason does not even resembles this. It has to have Date as a field at least. So make your struct look like the json you are receiving.
Another thing is that your time.Time will not be able to parse this string: Date(1432187580000-0500) in a normal date.
P.S. now that you have updated your struct to normal way, you have to find a way to parse your strange sting to a date. If you have power over your .net application, I would rather recommend changing json to a normal timestamp or something that can be easily parsable.
If not, then you have to change SomeTime time.Time to SomeTime string and then parse string to a normal timestamp and then parse this timestamp to a date.