I want to output an error if the field in json contains a value of null. How can I do it? I have tried "encoding/json". Maybe I need another library.
Code example:
package main
import (
"encoding/json"
"fmt"
"strings"
)
type Item struct {
Value *int
}
func main() {
var jsonBlob = `[
{},
{"Value": null},
{"Value": 0},
{"Value": 1}
]`
var items []Item
err := json.NewDecoder(strings.NewReader(jsonBlob)).Decode(&items)
if err != nil {
fmt.Println("error:", err)
}
for _, a := range items {
if a.Value != nil {
fmt.Println(*a.Value)
} else {
fmt.Println(a.Value)
}
}
}
I got:
<nil>
<nil>
0
1
I want:
<nil>
<error>
0
1
Please help. Many thanks!
If you want to control how a type is unmarshaled, you can implement json.Unmarshaler
Since a map allows you to tell the difference between an unset value, and a null value, unmarshaling first into a generic map[string]interface{} will allow you to inspect the values without tokenizing the JSON.
type Item struct {
Value *int
}
func (i *Item) UnmarshalJSON(b []byte) error {
tmp := make(map[string]interface{})
err := json.Unmarshal(b, &tmp)
if err != nil {
return err
}
val, ok := tmp["Value"]
if ok && val == nil {
return errors.New("Value cannot be nil")
}
if !ok {
return nil
}
f, ok := val.(float64)
if !ok {
return fmt.Errorf("unexpected type %T for Value", val)
}
n := int(f)
i.Value = &n
return nil
}
https://play.golang.org/p/MNDsQpfEJA
Related
If I have a map, I can Marshal it no problem:
package main
import (
"encoding/json"
"os"
)
type object map[int]interface{}
func main() {
obj := object{
1: "one", 2: object{3: "three"},
}
buf, err := json.Marshal(obj)
if err != nil {
panic(err)
}
os.Stdout.Write(buf) // {"1":"one","2":{"3":"three"}}
}
However I want to do the reverse. I tried this:
package main
import (
"encoding/json"
"fmt"
)
func main() {
buf := []byte(`{"1":"one","2":{"3":"three"}}`)
var obj map[int]interface{}
json.Unmarshal(buf, &obj)
// map[int]interface {}{1:"one", 2:map[string]interface {}{"3":"three"}}
fmt.Printf("%#v\n", obj)
}
Only the top level has the correct type. Is it possible to do what I am wanting?
JSON keys are never anything but strings, that's how the spec is defined, and you can see that in the output of you marshal. So when you try the reverse with interface{} as the top level map's value type, the type information for the nested objects is lost. You'd need a custom map type that implements UnmarshalJSON to be able to do what you want.
For example:
type IntKeyMap map[int]interface{}
func (m *IntKeyMap) UnmarshalJSON(data []byte) error {
raw := map[int]json.RawMessage{}
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
for k, v := range raw {
// check if the value is a nested object
if len(v) > 0 && v[0] == '{' && v[len(v)-1] == '}' {
// The following assumes that the nested JSON object's
// key strings represent integers, if that is not the
// case this block will fail.
obj := IntKeyMap{}
if err := json.Unmarshal([]byte(v), &obj); err != nil {
return err
}
(*m)[k] = obj
} else {
var i interface{}
if err := json.Unmarshal([]byte(v), &i); err != nil {
return err
}
(*m)[k] = i
}
}
return nil
}
https://go.dev/play/p/lmyhqD__Uod
I want to parse this JSON :
{
"error": null,
"id": "tutu",
"result": {
"param1": 559,
"param2": "yo",
"param3": {"tab":["a", "b"], "param4":"hello"},
}
}
The problem is that I want a flexible solution if the JSON structure changes: I want to be able to access each field with a key system (like in Javascript) without knowing in advance the JSON structure:
fmt.Println(jsonObj["result"]["param3"]["tab"][1])
Is it possible to do it ?
First, in your case unmarshall directly into map[string]interface{} to save an assertion.
var jsonObj map[string]interface{}
err := json.Unmarshal(b, &jsonObj)
Second, remember to check the assertion
result, ok := jsonObj["result"].(map[string]interface{})
if !ok {
panic("not json obj")
}
param3, ok := result["param3"].(map[string]interface{})
if !ok {
panic("not json obj")
}
tab, ok := param3["tab"].([]interface{})
if !ok {
panic("not json arr")
}
Lastly, you can declare a new type and "hide" the assertions in its methods
type AnyObj map[string]interface{}
func (obj AnyObj) MustObject(name string) AnyObj {
v, ok := obj[name].(map[string]interface{})
if !ok {
panic("not json obj")
}
return AnyObj(v)
}
func (obj AnyObj) MustArray(name string) []interface{} {
v, ok := obj[name].([]interface{})
if !ok {
panic("not json arr")
}
return v
}
Then use like this:
func main() {
var jsonObj AnyObj
err := json.Unmarshal(b, &jsonObj)
if (err != nil) {
panic(err);
}
tab := jsonObj.MustObject("result").MustObject("param3").MustArray("tab")
fmt.Println(tab[1])
}
check it here https://play.golang.org/p/ucdMZ0VEKcr
Yes, it is possible, although the fact that you need to assert the types makes it a bit unwieldy:
package main
import (
"encoding/json"
"fmt"
)
func main() {
b := []byte(`
{
"error": null,
"id": "tutu",
"result": {
"param1": 559,
"param2": "yo",
"param3": {"tab":["a", "b"], "param4": "hello"}
}
}
`)
var jsonObj interface{}
err := json.Unmarshal(b, &jsonObj)
if err != nil {
panic(err)
}
msg := jsonObj.(map[string]interface{})
result := msg["result"].(map[string]interface{})
param3 := result["param3"].(map[string]interface{})
tab := param3["tab"].([]interface{})
fmt.Println(tab[1])
}
This prints
b
like you would expect.
Note that this program will panic: not just if the JSON fails to parse, but also if the JSON does not exactly match the program's expectations: missing keys, different types and so on.
The Generic JSON with interface section of the JSON and Go article on The Go Blog has an example of checking the actual type of the thing at runtime; as was suggested in the comments, it's easy to forget.
this is mine used fastjson:
package main
import (
"fmt"
"github.com/valyala/fastjson"
)
type ParseValue struct {
*fastjson.Value
}
func Parse(b []byte) (*ParseValue, error) {
var p fastjson.Parser
v, err := p.ParseBytes(b)
if err != nil {
return nil, err
}
return &ParseValue{v}, nil
}
func (p *ParseValue) GetString(keys ...string) string {
return string(p.GetStringBytes(keys...))
}
func main() {
b := []byte(`
{
"error": null,
"id": "tutu",
"result": {
"param1": 559,
"param2": "yo",
"param3": {"tab":["a", "b"], "param4": "hello"}
}
}
`)
v, err := Parse(b)
if err != nil {
panic(err)
}
r := v.GetString("result", "param3", "tab", "1")
fmt.Println(r)
rr := v.GetUint("result", "param1")
fmt.Println(rr)
// output:
// b
// 559
}
I am trying to unmarshal some pretty ugly json but can't figure out how. I have:
package main
import "fmt"
import "encoding/json"
type PublicKey struct {
ID int `json:"id"`
Key string `json:"key"`
MyData []struct {
ID string `json:"id"`
Value int `json:"value"`
}
}
func main() {
b := `[
{
"id": 1,
"key": "my_key"
},
[
{
"id": "some_id",
"value": 12
},
{
"id": "anorther_id",
"value": 13
}
]
]`
var pk []PublicKey
err := json.Unmarshal([]byte(b), &pk)
if err != nil {
fmt.Println(err)
}
fmt.Println(pk)
}
For the result I am getting:
[{1 my_key []} {0 []}]
The second slice is empty when it shouldn't be.
EDIT:
The error I get is:
json: cannot unmarshal array into Go struct field PublicKey.key of type main.PublicKey
https://play.golang.org/p/cztXOchiiS5
That is some truly hideous JSON! I have two approaches to handling the mixed array elements and I like the 2nd one better. Here's the first approach using interface and a type switch:
package main
import (
"encoding/json"
"errors"
"fmt"
)
type PublicKey struct {
ID int `json:"id"`
Key string `json:"key"`
}
type MyData struct {
ID string `json:"id"`
Value int `json:"value"`
}
type MixedData struct {
Key []PublicKey
MyData [][]MyData
}
func (md *MixedData) UnmarshalJSON(b []byte) error {
md.Key = []PublicKey{}
md.MyData = [][]MyData{}
var obj []interface{}
err := json.Unmarshal([]byte(b), &obj)
if err != nil {
return err
}
for _, o := range obj {
switch o.(type) {
case map[string]interface{}:
m := o.(map[string]interface{})
id, ok := m["id"].(float64)
if !ok {
return errors.New("public key id must be an int")
}
pk := PublicKey{}
pk.ID = int(id)
pk.Key, ok = m["key"].(string)
if !ok {
return errors.New("public key key must be a string")
}
md.Key = append(md.Key, pk)
case []interface{}:
a := o.([]interface{})
myData := make([]MyData, len(a))
for i, x := range a {
m, ok := x.(map[string]interface{})
if !ok {
return errors.New("data array contains unexpected object")
}
val, ok := m["value"].(float64)
if !ok {
return errors.New("data value must be an int")
}
myData[i].Value = int(val)
myData[i].ID, ok = m["id"].(string)
if !ok {
return errors.New("data id must be a string")
}
md.MyData = append(md.MyData, myData)
}
default:
// got something unexpected, handle somehow
}
}
return nil
}
func main() {
b := `[
{
"id": 1,
"key": "my_key"
},
[
{
"id": "some_id",
"value": 12
},
{
"id": "another_id",
"value": 13
}
]
]`
m := MixedData{}
err := json.Unmarshal([]byte(b), &m)
if err != nil {
fmt.Println(err)
}
fmt.Println(m)
}
https://play.golang.org/p/g8d_AsH-pYY
Hopefully there aren't any unexpected other elements, but they can be handled similarly.
Here is the second that relies more on Go's internal JSON parsing with the help of json.RawMessage. It makes the same assumptions about the contents of the array. It assumes that any objects will Unmarshal into PublicKey instances and any arrays consist of only MyData instances. I also added how to marshal back into the target JSON for symmetry:
package main
import (
"encoding/json"
"fmt"
"os"
)
type PublicKey struct {
ID int `json:"id"`
Key string `json:"key"`
}
type MyData struct {
ID string `json:"id"`
Value int `json:"value"`
}
type MixedData struct {
Keys []PublicKey
MyData [][]MyData
}
func (md *MixedData) UnmarshalJSON(b []byte) error {
md.Keys = []PublicKey{}
md.MyData = [][]MyData{}
obj := []json.RawMessage{}
err := json.Unmarshal([]byte(b), &obj)
if err != nil {
return err
}
for _, o := range obj {
switch o[0] {
case '{':
pk := PublicKey{}
err := json.Unmarshal(o, &pk)
if err != nil {
return err
}
md.Keys = append(md.Keys, pk)
case '[':
myData := []MyData{}
err := json.Unmarshal(o, &myData)
if err != nil {
return err
}
md.MyData = append(md.MyData, myData)
default:
// got something unexpected, handle somehow
}
}
return nil
}
func (md *MixedData) MarshalJSON() ([]byte, error) {
out := make([]interface{}, len(md.Keys)+len(md.MyData))
i := 0
for _, x := range md.Keys {
out[i] = x
i++
}
for _, x := range md.MyData {
out[i] = x
i++
}
return json.Marshal(out)
}
func main() {
b := `[
{
"id": 1,
"key": "my_key"
},
[
{
"id": "some_id",
"value": 12
},
{
"id": "another_id",
"value": 13
}
]
]`
m := MixedData{}
err := json.Unmarshal([]byte(b), &m)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Println(m)
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(m); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
https://play.golang.org/p/ryZzaWKNcN0
Here's an approach that combines json.RawMessage with the trick of using the default unmarshaler in a type that implements json.Unmarshaler by creating a new temporary type that aliases the target type.
The idea is that we unmarshal the incoming array into a raw message and ensure that the array length is what we expect. Then we unmarshal the individual array elements into the custom struct types using their JSON tag annotations. The end result is that we can unmarshal the PublicKey type in the usual way and the UnmarshalJSON code is not terribly difficult to follow once you understand the tricks.
For example (Go Playground):
type PublicKey struct {
ID int `json:"id"`
Key string `json:"key"`
Data []MyData
}
type MyData struct {
ID string `json:"id"`
Value int `json:"value"`
}
func (pk *PublicKey) UnmarshalJSON(bs []byte) error {
// Unmarshal into a RawMessage so we can inspect the array length.
var rawMessage []json.RawMessage
err := json.Unmarshal(bs, &rawMessage)
if err != nil {
return err
}
if len(rawMessage) != 2 {
return fmt.Errorf("expected array of length 2, got %d", len(rawMessage))
}
// Parse the first object as PublicKey using the default unmarshaler
// using a temporary type that is an alias for the target type.
type PublicKey2 PublicKey
var pk2 PublicKey2
err = json.Unmarshal(rawMessage[0], &pk2)
if err != nil {
return err
}
// Parse the second object as []MyData in the usual way.
err = json.Unmarshal(rawMessage[1], &pk2.Data)
if err != nil {
return err
}
// Finally, assign the aliased object to the target object.
*pk = PublicKey(pk2)
return nil
}
func main() {
var pk PublicKey
err := json.Unmarshal([]byte(jsonstr), &pk)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n", pk)
// main.PublicKey{ID:1, Key:"my_key", Data:[]main.MyData{main.MyData{ID:"some_id", Value:12}, main.MyData{ID:"anorther_id", Value:13}}}
}
I need to decode and update only a specific value of a json object.
The issue is that I don't know the full structure of the object. The encoding/json package "ignores"/truncates the fields not provided in the struct so on encoding these fields are lost.
I'm wondering if it's possible to unmarshal only the structure I know, update it and then marshal it without to truncate/remove the unknown structure/information.
I know this is quite old question, but I learned combination of usual struct and json.RawMessage will do the job in the situation. Let me share.
The point is: hold entire data into raw field, and use that for encoding/decoding. Other fields can be derived from there.
package main
import (
"encoding/json"
"log"
)
type Color struct {
Space string
raw map[string]json.RawMessage
}
func (c *Color) UnmarshalJSON(bytes []byte) error {
if err := json.Unmarshal(bytes, &c.raw); err != nil {
return err
}
if space, ok := c.raw["Space"]; ok {
if err := json.Unmarshal(space, &c.Space); err != nil {
return err
}
}
return nil
}
func (c *Color) MarshalJSON() ([]byte, error) {
bytes, err := json.Marshal(c.Space)
if err != nil {
return nil, err
}
c.raw["Space"] = json.RawMessage(bytes)
return json.Marshal(c.raw)
}
func main() {
before := []byte(`{"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10}}`)
log.Println("before: ", string(before))
// decode
color := new(Color)
err := json.Unmarshal(before, color)
if err != nil {
log.Fatal(err)
}
// modify fields of interest
color.Space = "RGB"
// encode
after, err := json.Marshal(color)
if err != nil {
log.Fatal(err)
}
log.Println("after: ", string(after))
}
The output should be like this:
2020/09/03 01:11:33 before: {"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10}}
2020/09/03 01:11:33 after: {"Point":{"Y":255,"Cb":0,"Cr":-10},"Space":"RGB"}
NB: this doesn't preserve key order or indentations.
It seems it's possible.
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
type Color struct {
Space string
Point json.RawMessage // delay parsing until we know the color space
}
type RGB struct {
R uint8
G uint8
B uint8
}
type YCbCr struct {
Y uint8
Cb int8
Cr int8
}
var j = []byte(`
{"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10}}`)
var colors Color
err := json.Unmarshal(j, &colors)
if err != nil {
log.Fatalln("error:", err)
}
colors.Space = "no-space"
b, err := json.Marshal(&colors)
if err != nil {
panic(err)
}
fmt.Printf("b is now %s", b)
return
}
Is it possible to generate an error if a field was not found while parsing a JSON input using Go?
I could not find it in documentation.
Is there any tag that specifies the field as required?
There is no tag in the encoding/json package that sets a field to "required". You will either have to write your own MarshalJSON() method, or do a post check for missing fields.
To check for missing fields, you will have to use pointers in order to distinguish between missing/null and zero values:
type JsonStruct struct {
String *string
Number *float64
}
Full working example:
package main
import (
"fmt"
"encoding/json"
)
type JsonStruct struct {
String *string
Number *float64
}
var rawJson = []byte(`{
"string":"We do not provide a number"
}`)
func main() {
var s *JsonStruct
err := json.Unmarshal(rawJson, &s)
if err != nil {
panic(err)
}
if s.String == nil {
panic("String is missing or null!")
}
if s.Number == nil {
panic("Number is missing or null!")
}
fmt.Printf("String: %s Number: %f\n", *s.String, *s.Number)
}
Playground
You can also override the unmarshalling for a specific type (so a required field buried in a few json layers) without having to make the field a pointer. UnmarshalJSON is defined by the Unmarshaler interface.
type EnumItem struct {
Named
Value string
}
func (item *EnumItem) UnmarshalJSON(data []byte) (err error) {
required := struct {
Value *string `json:"value"`
}{}
all := struct {
Named
Value string `json:"value"`
}{}
err = json.Unmarshal(data, &required)
if err != nil {
return
} else if required.Value == nil {
err = fmt.Errorf("Required field for EnumItem missing")
} else {
err = json.Unmarshal(data, &all)
item.Named = all.Named
item.Value = all.Value
}
return
}
Here is another way by checking your customized tag
you can create a tag for your struct like:
type Profile struct {
Name string `yourprojectname:"required"`
Age int
}
Use reflect to check if the tag is assigned required value
func (p *Profile) Unmarshal(data []byte) error {
err := json.Unmarshal(data, p)
if err != nil {
return err
}
fields := reflect.ValueOf(p).Elem()
for i := 0; i < fields.NumField(); i++ {
yourpojectTags := fields.Type().Field(i).Tag.Get("yourprojectname")
if strings.Contains(yourpojectTags, "required") && fields.Field(i).IsZero() {
return errors.New("required field is missing")
}
}
return nil
}
And test cases are like:
func main() {
profile1 := `{"Name":"foo", "Age":20}`
profile2 := `{"Name":"", "Age":21}`
var profile Profile
err := profile.Unmarshal([]byte(profile1))
if err != nil {
log.Printf("profile1 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile1 unmarshal: %v\n", profile)
err = profile.Unmarshal([]byte(profile2))
if err != nil {
log.Printf("profile2 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile2 unmarshal: %v\n", profile)
}
Result:
profile1 unmarshal: {foo 20}
2009/11/10 23:00:00 profile2 unmarshal error: required field is missing
You can go to Playground to have a look at the completed code
You can just implement the Unmarshaler interface to customize how your JSON gets unmarshalled.
you can also make use of JSON schema validation.
package main
import (
"encoding/json"
"fmt"
"github.com/alecthomas/jsonschema"
"github.com/xeipuuv/gojsonschema"
)
type Bird struct {
Species string `json:"birdType"`
Description string `json:"what it does" jsonschema:"required"`
}
func main() {
var bird Bird
sc := jsonschema.Reflect(&bird)
b, _ := json.Marshal(sc)
fmt.Println(string(b))
loader := gojsonschema.NewStringLoader(string(b))
documentLoader := gojsonschema.NewStringLoader(`{"birdType": "pigeon"}`)
schema, err := gojsonschema.NewSchema(loader)
if err != nil {
panic("nop")
}
result, err := schema.Validate(documentLoader)
if err != nil {
panic("nop")
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, err := range result.Errors() {
// Err implements the ResultError interface
fmt.Printf("- %s\n", err)
}
}
}
Outputs
{"$schema":"http://json-schema.org/draft-04/schema#","$ref":"#/definitions/Bird","definitions":{"Bird":{"required":["birdType","what it does"],"properties":{"birdType":{"type":"string"},"what it does":{"type":"string"}},"additionalProperties":false,"type":"object"}}}
The document is not valid. see errors :
- (root): what it does is required
code example taken from Strict JSON parsing