I got the following custom type:
type TimeWithoutZone struct {
time.Time
}
The Marshaling works fine:
const timeWithoutZoneFormat = "2006-01-02T15:04:05"
func (t *TimeWithoutZone) MarshalJSON() ([]byte, error) {
stamp := fmt.Sprintf(`"%s"`, t.Time.Format(timeWithoutZoneFormat ))
return []byte(stamp), nil
}
But here the date can not be parsed:
func (t *TimeWithoutZone) UnmarshalJSON(data []byte) (err error) {
log.Println("Parsing: " + string(data))
t.Time, err = time.Parse(`"` + timeWithoutZoneFormat + `"`, string(data))
if err != nil {
return err
}
return nil
}
It logs: Parsing: {"time":"2016-09-06T11:06:16"} but I would expect it to parse just the value of time
What am I doing wrong? here is the related test:
type TimeTestObj struct {
Time TimeWithoutZone `json:"time"`
}
func TestParseDataWithoutTimezone(t *testing.T) {
parsed := TimeWithoutZone{}
data := `{"time":"2016-09-06T11:06:16"}`
err := json.Unmarshal([]byte(data), &parsed)
if err != nil {
t.Error(err)
}
if parsed.Unix() != 1473152776 {
t.Error(parsed.Unix(), "!=", 1473152776)
}
}
All the examples I find, and even the default parser from the Go time package seem to work that way...
Wow I just have the wrong type in this line:
parsed := TimeWithoutZone{}
must be
parsed := TimeTestObj{}
...
Related
I have following code:
func policyDocumentToStr(doc map[string]interface{}) (*string, error) {
policy, err := json.Marshal(doc)
if err != nil {
log.Debugf("Error converting policy document to string. Error %s", err)
return nil, err
}
policyAsString := string(policy)
return &policyAsString, nil
}
I want to write a unit test which would cover the case of json.Marshal(doc) returning an error. Can anybody suggest how can I generate an error? What kind of input to function would result in error at line policy, err := json.Marshal(doc)?
Feed it a value that cannot be represented in JSON. There are many ways to do this, including creating a type with a custom marshaler that always returns an error. But one of the simplest ways is to try to marshal a channel:
x := map[string]interface{}{
"foo": make(chan int),
}
_, err := json.Marshal(x)
fmt.Printf("Marshal error: %s\n", err)
Playground link
Define a type that implements json.Marshaler. Use this type to produce any error you want (including error values from the json package):
type FakeValue struct {
err error
}
func (v FakeValue) MarshalJSON() ([]byte, error) {
if v.err != nil {
return nil, v.err
}
return []byte(`null`), v.err
}
func TestPolicyString(t *testing.T) {
doc := map[string]interface{}{
"fake_error": FakeValue{errors.New("fail!")},
}
_, err := policyDocumentToStr(doc)
if err == nil {
t.Fatal("Got nil, want error")
}
}
Monkey patching is one of the ways to facilitate testing by changing a program at runtime, either by replacing a function or a variable. Here we are swapping the json.Marshal and we can mock it while testing.
var (
jsonMarshal = json.Marshal
)
This is your piece of code
func policyDocumentToStr(doc map[string]interface{}) (*string, error) {
policy, err := jsonMarshal(doc)
if err != nil {
fmt.Printf("Error converting policy document to string. Error %s", err)
return nil, err
}
policyAsString := string(policy)
return &policyAsString, nil
}
The test code would look something like this
import (
"testing"
"errors"
)
func fakemarshal(v interface{}) ([]byte, error) {
return []byte{}, errors.New("Marshalling failed")
}
func restoremarshal(replace func(v interface{}) ([]byte, error)) {
jsonMarshal = replace
}
func TestPolicyDocumentToStr(t * testing.T){
storedMarshal := jsonMarshal
jsonMarshal = fakemarshal
defer restoremarshal(storedMarshal)
input := map[string]interface{} {
"test": "test1",
}
tests := []struct {
name string
arg map[string]interface{}
wantErr string
}{
{
name: "Test if JSON Marshalling fails",
arg: input,
wantErr: "Marshalling failed",
},
}
for _, tt := range tests {
_, gotErr := policyDocumentToStr(tt.arg)
if gotErr != nil && gotErr.Error() != tt.wantErr {
t.Errorf("Expected %s but got %s", tt.wantErr, gotErr.Error())
}
}
}
What is the idiomatic way to unmarshal into time.Duration in Go? How can I make use of time.ParseDuration?
The lack of JSON marshaling and unmarshaling methods on time.Duration was an unfortunate oversight. This should hopefully be resolved in Go2 (see issue #10275).
You can, however, define your own type around time.Duration that supports marshaling to the string representation of the duration and unmarshaling from either the numeric or string representations. Here is an example of such an implementation:
package main
import (
"encoding/json"
"errors"
"fmt"
"time"
)
type Duration struct {
time.Duration
}
func (d Duration) MarshalJSON() ([]byte, error) {
return json.Marshal(d.String())
}
func (d *Duration) UnmarshalJSON(b []byte) error {
var v interface{}
if err := json.Unmarshal(b, &v); err != nil {
return err
}
switch value := v.(type) {
case float64:
d.Duration = time.Duration(value)
return nil
case string:
var err error
d.Duration, err = time.ParseDuration(value)
if err != nil {
return err
}
return nil
default:
return errors.New("invalid duration")
}
}
type Message struct {
Elapsed Duration `json:"elapsed"`
}
func main() {
msgEnc, err := json.Marshal(&Message{
Elapsed: Duration{time.Second * 5},
})
if err != nil {
panic(err)
}
fmt.Printf("%s\n", msgEnc)
var msg Message
if err := json.Unmarshal([]byte(`{"elapsed": "1h"}`), &msg); err != nil {
panic(err)
}
fmt.Printf("%#v\n", msg)
}
https://play.golang.org/p/Zm6hpNR-ZJ2
Just to extend the previous answer. There is another way (very close to Tim's)
type Duration time.Duration
func (d Duration) MarshalJSON() ([]byte, error) {
return json.Marshal(time.Duration(d).String())
}
func (d *Duration) UnmarshalJSON(b []byte) error {
var v interface{}
if err := json.Unmarshal(b, &v); err != nil {
return err
}
switch value := v.(type) {
case float64:
*d = Duration(time.Duration(value))
return nil
case string:
tmp, err := time.ParseDuration(value)
if err != nil {
return err
}
*d = Duration(tmp)
return nil
default:
return errors.New("invalid duration")
}
}
I'm looking for a way to unmarshal a JSON body without having to specify targets for all fields. And then be a able to "remarshal" the body with implicit fields untouched.
Something like this would be good, but doesn't work as a expected: (https://play.golang.org/p/fnVOKrmiFj)
package main
import (
"encoding/json"
"fmt"
)
type Transaction struct {
Field1 string `json:"field1"`
X map[string]interface{} `json:"-"`
}
func main() {
body := []byte(`{"field1": "value1", "field2": "value2"}`)
fmt.Printf("%+v\n", string(body))
var unmarshalledTransaction Transaction
json.Unmarshal(body, &unmarshalledTransaction)
fmt.Printf("%+v\n", unmarshalledTransaction)
remarshalledTransaction, _ := json.Marshal(&unmarshalledTransaction)
fmt.Printf("%+v\n", string(remarshalledTransaction))
}
Gives the output
{"field1": "value1", "field2": "value2"}
{Field1:value1 X:map[]}
{"field1":"value1"}
My expected result would be that unmarshalledTransaction contains the "leftover" fields in the X fields. And they are then restored when Marshalling again.
Can this be done?
You would need to implement the MarshalJSON and UnmarshalJSON interfaces, and write your own logic to remap the fields to the appropriate spots:
func (t *Transaction) MarshalJSON() ([]byte, error) {
data := t.X
data["field1"] = t.Field1
return json.Marshal(data)
}
func (t *Transaction) UnmarshalJSON(data []byte) error {
m := make(map[string]interface{})
json.Unmarshal(data, &m)
t.Field1 = m["field1"].(string)
delete(m, "field1")
t.X = m
return nil
}
https://play.golang.org/p/KBGAsXB0xA
If you want a generic solution (that would work with any struct without knowing the fields in advance), you can implement a function that would un-marshal the body into a struct and also return the "leftover" fields.
For that you'd also need to implement a function that would convert any given struct to a map (to be used to then manipulate maps in a generic way instead of known-in-advance structs).
Like so:
func structToMap(object interface{}) (map[string]interface{}, error) {
tempJson, err := json.Marshal(object)
if err != nil {
return nil, err
}
var theMap map[string]interface{}
err = json.Unmarshal(tempJson, &theMap)
if err != nil {
return nil, err
}
return theMap, nil
}
And then:
func unmarshalWithLeftovers(jsonBody []byte, target interface{}) (map[string]interface{}, error) {
err := json.Unmarshal(jsonBody, target)
if err != nil {
return nil, err
}
structMap, err := structToMap(target)
if err != nil {
return nil, err
}
var leftOvers map[string]interface{}
err = json.Unmarshal(jsonBody, &leftOvers)
if err != nil {
return nil, err
}
for k, _ := range structMap {
delete(leftOvers, k)
}
return leftOvers, nil
}
You can then combine the struct and the leftovers map in a similar fashion to re-marshal everything.
See here a working example with the same type and json string that you used in your question:
https://play.golang.org/p/Fot6YVurHH
Basically, what I want to achieve is to get the content of a directory via os.ReadDir() and then encode the result into json.
Directly doing json.Marshal() cause no exception but gave me an empty result.
So I tried this:
func (f *os.FileInfo) MarshalerJSON() ([]byte, error) {
return f.Name(), nil
}
Then Go tells me that os.FileInfo() is an interface and cannot be extended this way.
What's the best way to do this?
Pack the data into a struct that can be serialized:
http://play.golang.org/p/qDeg2bfik_
type FileInfo struct {
Name string
Size int64
Mode os.FileMode
ModTime time.Time
IsDir bool
}
func main() {
dir, err := os.Open(".")
if err != nil {
log.Fatal(err)
}
entries, err := dir.Readdir(0)
if err != nil {
log.Fatal(err)
}
list := []FileInfo{}
for _, entry := range entries {
f := FileInfo{
Name: entry.Name(),
Size: entry.Size(),
Mode: entry.Mode(),
ModTime: entry.ModTime(),
IsDir: entry.IsDir(),
}
list = append(list, f)
}
output, err := json.Marshal(list)
if err != nil {
log.Fatal(err)
}
log.Println(string(output))
}
Here is a different version that makes the usage seemingly simple. Though you cannot unmarshall it back to the object. This is only applicable if you are sending it to client-end or something.
http://play.golang.org/p/vmm3temCUn
Usage
output, err := json.Marshal(FileInfo{entry})
output, err := json.Marshal(FileInfoList{entries})
Code
type FileInfo struct {
os.FileInfo
}
func (f FileInfo) MarshalJSON() ([]byte, error) {
return json.Marshal(map[string]interface{}{
"Name": f.Name(),
"Size": f.Size(),
"Mode": f.Mode(),
"ModTime": f.ModTime(),
"IsDir": f.IsDir(),
})
}
type FileInfoList struct {
fileInfoList []os.FileInfo
}
//This is inefficient to call multiple times for the same struct
func (f FileInfoList) MarshalJSON() ([]byte, error) {
fileInfoList := make([]FileInfo, 0, len(f.fileInfoList))
for _, val := range f.fileInfoList {
fileInfoList = append(fileInfoList, FileInfo{val})
}
return json.Marshal(fileInfoList)
}
Is it possible to generate an error if a field was not found while parsing a JSON input using Go?
I could not find it in documentation.
Is there any tag that specifies the field as required?
There is no tag in the encoding/json package that sets a field to "required". You will either have to write your own MarshalJSON() method, or do a post check for missing fields.
To check for missing fields, you will have to use pointers in order to distinguish between missing/null and zero values:
type JsonStruct struct {
String *string
Number *float64
}
Full working example:
package main
import (
"fmt"
"encoding/json"
)
type JsonStruct struct {
String *string
Number *float64
}
var rawJson = []byte(`{
"string":"We do not provide a number"
}`)
func main() {
var s *JsonStruct
err := json.Unmarshal(rawJson, &s)
if err != nil {
panic(err)
}
if s.String == nil {
panic("String is missing or null!")
}
if s.Number == nil {
panic("Number is missing or null!")
}
fmt.Printf("String: %s Number: %f\n", *s.String, *s.Number)
}
Playground
You can also override the unmarshalling for a specific type (so a required field buried in a few json layers) without having to make the field a pointer. UnmarshalJSON is defined by the Unmarshaler interface.
type EnumItem struct {
Named
Value string
}
func (item *EnumItem) UnmarshalJSON(data []byte) (err error) {
required := struct {
Value *string `json:"value"`
}{}
all := struct {
Named
Value string `json:"value"`
}{}
err = json.Unmarshal(data, &required)
if err != nil {
return
} else if required.Value == nil {
err = fmt.Errorf("Required field for EnumItem missing")
} else {
err = json.Unmarshal(data, &all)
item.Named = all.Named
item.Value = all.Value
}
return
}
Here is another way by checking your customized tag
you can create a tag for your struct like:
type Profile struct {
Name string `yourprojectname:"required"`
Age int
}
Use reflect to check if the tag is assigned required value
func (p *Profile) Unmarshal(data []byte) error {
err := json.Unmarshal(data, p)
if err != nil {
return err
}
fields := reflect.ValueOf(p).Elem()
for i := 0; i < fields.NumField(); i++ {
yourpojectTags := fields.Type().Field(i).Tag.Get("yourprojectname")
if strings.Contains(yourpojectTags, "required") && fields.Field(i).IsZero() {
return errors.New("required field is missing")
}
}
return nil
}
And test cases are like:
func main() {
profile1 := `{"Name":"foo", "Age":20}`
profile2 := `{"Name":"", "Age":21}`
var profile Profile
err := profile.Unmarshal([]byte(profile1))
if err != nil {
log.Printf("profile1 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile1 unmarshal: %v\n", profile)
err = profile.Unmarshal([]byte(profile2))
if err != nil {
log.Printf("profile2 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile2 unmarshal: %v\n", profile)
}
Result:
profile1 unmarshal: {foo 20}
2009/11/10 23:00:00 profile2 unmarshal error: required field is missing
You can go to Playground to have a look at the completed code
You can just implement the Unmarshaler interface to customize how your JSON gets unmarshalled.
you can also make use of JSON schema validation.
package main
import (
"encoding/json"
"fmt"
"github.com/alecthomas/jsonschema"
"github.com/xeipuuv/gojsonschema"
)
type Bird struct {
Species string `json:"birdType"`
Description string `json:"what it does" jsonschema:"required"`
}
func main() {
var bird Bird
sc := jsonschema.Reflect(&bird)
b, _ := json.Marshal(sc)
fmt.Println(string(b))
loader := gojsonschema.NewStringLoader(string(b))
documentLoader := gojsonschema.NewStringLoader(`{"birdType": "pigeon"}`)
schema, err := gojsonschema.NewSchema(loader)
if err != nil {
panic("nop")
}
result, err := schema.Validate(documentLoader)
if err != nil {
panic("nop")
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, err := range result.Errors() {
// Err implements the ResultError interface
fmt.Printf("- %s\n", err)
}
}
}
Outputs
{"$schema":"http://json-schema.org/draft-04/schema#","$ref":"#/definitions/Bird","definitions":{"Bird":{"required":["birdType","what it does"],"properties":{"birdType":{"type":"string"},"what it does":{"type":"string"}},"additionalProperties":false,"type":"object"}}}
The document is not valid. see errors :
- (root): what it does is required
code example taken from Strict JSON parsing