I have a struct as
type Result struct {
Foo *string
}
I would like to get a json object like
{"Foo": null}
How do I achieve this ?
I have tried a few things:
approach 1:
var res1 *Result
json.Unmarshal(nil, &res1)
approach2
var res1 Result
res1.Foo = nil
I get a res1 struct with Foo as nil
Thanks for help!
Edit:
var res1 *Result -> var res1 Result
Basic types in Go are not nillable. string is a basic type.
In order for Foo to be nillable, one solution is to make it a string pointer. That would look like this:
type Result struct {
Foo *string
}
If you don't like nil, you can also add a boolean field explaining if Foo is present or not:
type Result struct {
Foo string
IsPresent bool // true if Foo is present. false otherwise.
}
But you would need to write a custom JSON deserializer for this. So just making Foo a pointer is what I would do.
Edit after question was changed to mention OP already is using *string:
To go from the JSON string { "Foo" : null } to the Result Go struct listed above in my answer, the json package Unmarshal function can be used:
var r Result
err := json.Unmarshal([]byte(`{ "Foo" : null }`), &r)
if err != nil { /* ... */ }
// now f contains the desired data
To convert from the Result struct to the above JSON string, the json library Marshal function can be used:
var r Result // Foo will be zero'd to nil
jsonStr, err := json.Marshal(&r)
if err != nil { /* ... */ }
// now jsonStr contains `{ "Foo" : null }`
Here are the above two code blocks running in Go Playground.
Related
Trying to convert a struct to []json.RawMessage. My understanding is that json.Marshal() converts it to byte[] as is []json.RawMessage. I cannot however seem to convert between the two. My function expects to receive input as []json.RawMessage.
Have tried several different methods including myIn := json.RawMessage(&myJsonStruct{"string1", "string2"}) and myIn := (*json.RawMessage)(json.Marshal(&myJsonStruct{"string1", "string2"}))
type myJsonStruct struct {
myString string `json:"myString"`
mySecongString string `json:"mySecondString"`
}
myIn := json.Marshal(&myJsonStruct{"string1", "string2"})
myFunction(myIn)
myFunction(receivedIn []json.RawMessage) {
//do work
return
}
There are a few things here:
You need to create a new slice of json.RawMessage in order to pass that expected type into your function myFunction as an argument
Store the result of marshaling your custom struct myJsonStruct in a variable myIn (type []byte)
Create a new variable of myInRaw (type json.RawMessage) and append that to the previously created slice of json.RawMessage.
The above steps will then allow you to pass in the slice of json.RawMessage to your function for further work to be done.
See example below or working example in the playground:
package main
import (
"encoding/json"
)
type myJsonStruct struct {
myString string `json:"myString"`
mySecongString string `json:"mySecondString"`
}
func myFunction(receivedIn []json.RawMessage) {
//do work
return
}
func main() {
var rawJSONSlice []json.RawMessage
myIn, err := json.Marshal(
&myJsonStruct{
myString: "string1",
mySecongString: "string2",
},
)
if err != nil {
// catch err
}
myInRaw := json.RawMessage(myIn)
rawJSONSlice = append(rawJSONSlice, myInRaw)
myFunction(rawJSONSlice)
}
I'm reading a file.json into memory. It's an array of objects, sample:
[
{"id":123123,"language":"ja-JP","location":"Osaka"}
,{"id":33332,"language":"ja-JP","location":"Tokyo"}
,{"id":31231313,"language":"ja-JP","location":"Kobe"}
]
I want to manipulate certain keys in this JSON file, so that they start with uppercase. Meaning
"language" becomes "Language" each time it's found. What I've done so far is to make a struct representing each object, as such:
type sampleStruct struct {
ID int `json:"id"`
Language string `json:"Language"`
Location string `json:"Location"`
}
Here, I define the capitalization. Meaning, id shouldn't be capitalized, but location and language should.
Rest of the code is as such:
func main() {
if len(os.Args) < 2 {
fmt.Println("Missing filename parameter.")
return
}
translationfile, err := ioutil.ReadFile(os.Args[1])
fileIsValid := isValidJSON(string(translationfile))
if !fileIsValid {
fmt.Println("Invalid JSON format for: ", os.Args[1])
return
}
if err != nil {
fmt.Println("Can't read file: ", os.Args[1])
panic(err)
}
}
func isValidJSON(str string) bool {
var js json.RawMessage
return json.Unmarshal([]byte(str), &js) == nil
}
// I'm unsure how to iterate through the JSON objects and only uppercase the objects matched in my struct here.
func upperCaseSpecificKeys()
// ...
Desired output, assuming the struct represents the whole data object, transform each key as desired:
[
{"id":123123,"Language":"ja-JP","Location":"Osaka"}
,{"id":33332,"Language":"ja-JP","Location":"Tokyo"}
,{"id":31231313,"Language":"ja-JP","Location":"Kobe"}
]
The documentation on json.Unmarshal says (with added emphasis):
To unmarshal JSON into a struct, Unmarshal matches incoming object
keys to the keys used by Marshal (either the struct field name or its
tag), preferring an exact match but also accepting a case-insensitive
match
See example here: https://play.golang.org/p/1vv8PaQUOfg
One way is to implement custom marshal method, although not very flexible:
type upStruct struct {
ID int `json:"id"`
Language string
Location string
}
type myStruct struct {
ID int `json:"id"`
Language string `json:"language"`
Location string `json:"location"`
}
func (m myStruct) MarshalJSON() ([]byte, error) {
return json.Marshal(upStruct(m))
}
....
func main() {
var mySArr []myStruct
// 1. Unmarshal the input
err := json.Unmarshal([]byte(myJson), &mySArr)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Input: \n%+v\n", mySArr)
// 2. Then, marshal it using our custom marshal method
val, err := json.Marshal(mySArr)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Output: \n%v\n", string(val))
}
Link to working code: https://play.golang.org/p/T4twqPc34k0
Thanks to mkopriva
I am struggling with deserializing a integer into a string struct field.
The struct field is a string and is expected to be assignable from users of my library. That's why I want it to be a string, since for the purpose of writing it to the database I actually don't care about the value inside.
The users can supply text, but some just assign integers.
Consider this struct:
type Test struct {
Foo string
}
Sometimes I end up with a JSON value that is valid but won't deserialize into the struct due to the Foo field being a integer instead of a string:
{ "foo": "1" } // works
{ "foo": 1 } // doesn't
json.Unmarshal will blow up with the following error:
json: cannot unmarshal number into Go struct field test.Foo of type string
See the reproduction: https://play.golang.org/p/4Qau3umaVm
Now in every other JSON library (in other languages) I have worked in so far, if the target field is a string and you get a integer the deserializer will usually just wrap the int in a string and be done with it. Can this be achieved in Go?
Since I can't really control how the data comes in I need to make json.Unmarshal unsensitive to this - the other solution would be to define Foo as interface{} which needlessly complicates my code with type assertions etc..
Any ideas on how to do this? I basically need the inverse of json:",string"
To handle big structs you can use embedding.
Updated to not discard possibly previously set field values.
func (t *T) UnmarshalJSON(d []byte) error {
type T2 T // create new type with same structure as T but without its method set!
x := struct{
T2 // embed
Foo json.Number `json:"foo"`
}{T2: T2(*t)} // don't forget this, if you do and 't' already has some fields set you would lose them
if err := json.Unmarshal(d, &x); err != nil {
return err
}
*t = T(x.T2)
t.Foo = x.Foo.String()
return nil
}
https://play.golang.org/p/BytXCeHMvt
You can customize how the data structure is Unmarshaler by implementing the json.Unmarshaler interface.
The simplest way to handle unknown types is to unnmarshal the JSON into an intermediate structure, and handle the type assertions and validation during deserialization:
type test struct {
Foo string `json:"foo"`
}
func (t *test) UnmarshalJSON(d []byte) error {
tmp := struct {
Foo interface{} `json:"foo"`
}{}
if err := json.Unmarshal(d, &tmp); err != nil {
return err
}
switch v := tmp.Foo.(type) {
case float64:
t.Foo = strconv.Itoa(int(v))
case string:
t.Foo = v
default:
return fmt.Errorf("invalid value for Foo: %v", v)
}
return nil
}
https://play.golang.org/p/t0eI4wCxdB
I have some JSON of the form:
[{
"type": "car",
"color": "red",
"hp": 85,
"doors": 4
}, {
"type": "plane",
"color": "blue",
"engines": 3
}]
I have types car and plane that satisfy a vehicle interface; I'd like to be able to write:
var v []vehicle
e := json.Unmarshal(myJSON, &v)
... and have JSON fill my slice of vehicles with a car and a plane; instead (and unsurprisingly) I just get "cannot unmarshal object into Go value of type main.vehicle".
For reference, here are suitable definitions of the types involved:
type vehicle interface {
vehicle()
}
type car struct {
Type string
Color string
HP int
Doors int
}
func (car) vehicle() { return }
type plane struct {
Type string
Color string
Engines int
}
func (plane) vehicle() { return }
var _ vehicle = (*car)(nil)
var _ vehicle = (*plane)(nil)
(Note that I'm actually totally uninterested in the t field on car and plane - it could be omitted because this information will, if someone successfully answers this question, be implicit in the dynamic type of the objects in v.)
Is there a way to have the JSON umarhsaller choose which type to use based on some part of the contents (in this case, the type field) of the data being decoded?
(Note that this is not a duplicate of Unmarshal JSON with unknown fields because I want each item in the slice to have a different dynamic type, and from the value of the 'type' property I know exactly what fields to expect—I just don't know how to tell json.Unmarshal how to map 'type' property values onto Go types.)
Taking the answers from the similar question: Unmarshal JSON with unknown fields, we can construct a few ways to unamrshal this JSON object in a []vehicle data structure.
The "Unmarshal with Manual Handling" version can be done by using a generic []map[string]interface{} data structure, then building the correct vehicles from the slice of maps. For brevity, this example does leave out the error checking for missing or incorrectly typed fields which the json package would have done.
https://play.golang.org/p/fAY9JwVp-4
func NewVehicle(m map[string]interface{}) vehicle {
switch m["type"].(string) {
case "car":
return NewCar(m)
case "plane":
return NewPlane(m)
}
return nil
}
func NewCar(m map[string]interface{}) *car {
return &car{
Type: m["type"].(string),
Color: m["color"].(string),
HP: int(m["hp"].(float64)),
Doors: int(m["doors"].(float64)),
}
}
func NewPlane(m map[string]interface{}) *plane {
return &plane{
Type: m["type"].(string),
Color: m["color"].(string),
Engines: int(m["engines"].(float64)),
}
}
func main() {
var vehicles []vehicle
objs := []map[string]interface{}{}
err := json.Unmarshal(js, &objs)
if err != nil {
log.Fatal(err)
}
for _, obj := range objs {
vehicles = append(vehicles, NewVehicle(obj))
}
fmt.Printf("%#v\n", vehicles)
}
We could leverage the json package again to take care of the unmarshaling and type checking of the individual structs by unmarshaling a second time directly into the correct type. This could all be wrapped up into a json.Unmarshaler implementation by defining an UnmarshalJSON method on the []vehicle type to first split up the JSON objects into raw messages.
https://play.golang.org/p/zQyL0JeB3b
type Vehicles []vehicle
func (v *Vehicles) UnmarshalJSON(data []byte) error {
// this just splits up the JSON array into the raw JSON for each object
var raw []json.RawMessage
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
for _, r := range raw {
// unamrshal into a map to check the "type" field
var obj map[string]interface{}
err := json.Unmarshal(r, &obj)
if err != nil {
return err
}
vehicleType := ""
if t, ok := obj["type"].(string); ok {
vehicleType = t
}
// unmarshal again into the correct type
var actual vehicle
switch vehicleType {
case "car":
actual = &car{}
case "plane":
actual = &plane{}
}
err = json.Unmarshal(r, actual)
if err != nil {
return err
}
*v = append(*v, actual)
}
return nil
}
JSON decoding and encoding in Go is actually surprisingly well at recognizing fields inside embedded structs. E.g. decoding or encoding the following structure works when there is no overlapping fields between type A and type B:
type T struct{
Type string `json:"type"`
*A
*B
}
type A struct{
Baz int `json:"baz"`
}
type B struct{
Bar int `json:"bar"`
}
Be aware that if both "baz" and "bar" are set in the JSON for the example above, both the T.A and T.B properties will be set.
If there is overlapping fields between A and B, or just to be able to better discard invalid combinations of fields and type, you need to implement the json.Unmarshaler interface. To not have to first decode fields into a map, you can extend the trick of using embedded structs.
type TypeSwitch struct {
Type string `json:"type"`
}
type T struct {
TypeSwitch
*A
*B
}
func (t *T) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &t.TypeSwitch); err != nil {
return err
}
switch t.Type {
case "a":
t.A = &A{}
return json.Unmarshal(data, t.A)
case "b":
t.B = &B{}
return json.Unmarshal(data, t.B)
default:
return fmt.Errorf("unrecognized type value %q", t.Type)
}
}
type A struct {
Foo string `json:"bar"`
Baz int `json:"baz"`
}
type B struct {
Foo string `json:"foo"`
Bar int `json:"bar"`
}
For marshaling back, json.Marshaler must also be implemented if there is overlapping fields.
Full example: https://play.golang.org/p/UHAdxlVdFQQ
The two passes approach works fine, but there is also the option of the mapstructure package, that was created to do exactly this.
I was facing the same problem.
I'm using the lib github.com/mitchellh/mapstructure together the encoding/json.
I first, unmarshal the json to a map, and use mapstructure to convert the map to my struct, e.g.:
type (
Foo struct {
Foo string `json:"foo"`
}
Bar struct {
Bar string `json:"bar"`
}
)
func Load(jsonStr string, makeInstance func(typ string) any) (any, error) {
// json to map
m := make(map[string]any)
e := json.Unmarshal([]byte(jsonStr), &m)
if e != nil {
return nil, e
}
data := makeInstance(m["type"].(string))
// decoder to copy map values to my struct using json tags
cfg := &mapstructure.DecoderConfig{
Metadata: nil,
Result: &data,
TagName: "json",
Squash: true,
}
decoder, e := mapstructure.NewDecoder(cfg)
if e != nil {
return nil, e
}
// copy map to struct
e = decoder.Decode(m)
return data, e
}
Using:
f, _ := Load(`{"type": "Foo", "foo": "bar"}`, func(typ string) any {
switch typ {
case "Foo":
return &Foo{}
}
return nil
})
If the property is a string you can use .(string) for casting the property because the origin is an interface.
You can use it the next way:
v["type"].(string)
Is there a way, in golang, to see if I can differentiate between a json field being set to null vs a json field not being there when unmarshalled into a struct? Because both set the value in the struct to be nil, but I need to know if the field was there to begin with and to see if someone set it to null.
{
"somefield1":"somevalue1",
"somefield2":null
}
VS
{
"somefield1":"somevalue1",
}
Both jsons will be nil when unmarshalled into a struct.
Any useful resources will be very appreciated!
Use json.RawMessage to "delay" the unmarshaling process to determine the raw byte before deciding to do something:
var data = []byte(`{
"somefield1":"somevalue1",
"somefield2": null
}`)
type Data struct {
SomeField1 string
SomeField2 json.RawMessage
}
func main() {
d := &Data{}
_ = json.Unmarshal(data, &d)
fmt.Println(d.SomeField1)
if len(d.SomeField2) > 0 {
if string(d.SomeField2) == "null" {
fmt.Println("somefield2 is there but null")
} else {
fmt.Println("somefield2 is there and not null")
// Do something with the data
}
} else {
fmt.Println("somefield2 doesn't exist")
}
}
See the playground https://play.golang.org/p/Wganpf4sbO
If you're on Go 1.18+ you can use a simple generic struct to know when a JSON value is undefined or null:
type Optional[T any] struct {
Defined bool
Value *T
}
// UnmarshalJSON is implemented by deferring to the wrapped type (T).
// It will be called only if the value is defined in the JSON payload.
func (o *Optional[T]) UnmarshalJSON(data []byte) error {
o.Defined = true
return json.Unmarshal(data, &o.Value)
}
That's all, you can then just use this type in your structs:
type Payload struct {
Field1 Optional[string] `json:"field1"`
Field2 Optional[bool] `json:"field2"`
Field3 Optional[int32] `json:"field3"`
}
And you'll be able to use the Defined field to know if the field was null or undefined.
Check this playground link for a full example: https://go.dev/play/p/JZfZyVVUABz
Original answer (pre-generics)
Another way to do this, with a custom type:
// OptionalString is a struct that represents a JSON string that can be
// undefined (Defined == false), null (Value == nil && Defined == true) or
// defined with a string value
type OptionalString struct {
Defined bool
Value *string
}
// UnmarshalJSON implements the json.Unmarshaler interface.
// When called, it means that the value is defined in the JSON payload.
func (os *OptionalString) UnmarshalJSON(data []byte) error {
// UnmarshalJSON is called only if the key is present
os.Defined = true
return json.Unmarshal(data, &os.Value)
}
// Payload represents the JSON payload that you want to represent.
type Payload struct {
SomeField1 string `json:"somefield1"`
SomeField2 OptionalString `json:"somefield2"`
}
You can then just use the regular json.Unmarshal function to get your values, for example:
var p Payload
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":null
}`), &p)
fmt.Printf("Should be defined == true and value == nil: \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{"somefield1":"somevalue1"}`), &p)
fmt.Printf("Should be defined == false \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":"somevalue2"
}`), &p)
fmt.Printf("Parsed should be defined == true and value != nil \n%+v\n", p)
if p.SomeField2.Value != nil {
fmt.Printf("SomeField2's value is %s", *p.SomeField2.Value)
}
Should give you this output:
Should be defined == true and value == nil:
{SomeField1:somevalue1 SomeField2:{Defined:true Value:<nil>}}
Should be defined == false
{SomeField1:somevalue1 SomeField2:{Defined:false Value:<nil>}}
Parsed should be defined == true and value != nil
{SomeField1:somevalue1 SomeField2:{Defined:true Value:0xc000010370}}
SomeField2's value is somevalue2
Link to the playground with the full example: https://play.golang.org/p/AUDwPKHBs62
Do note that you will need one struct for each type you want to wrap, so, if you need an optional number you'll need to create an OptionalFloat64 (all JSON numbers can be 64 bit floats) struct with a similar implementation.
If/when generics land in Go, this could be simplified to a single generic struct.
Good question.
I believe you can use https://golang.org/pkg/encoding/json/#RawMessage as:
type MyMessage struct {
somefield1 string
somefield2 json.RawMessage
}
So after unmarshalling you should have []byte("null") in case of null and nil if missing.
Here is a playground code: https://play.golang.org/p/UW8L68K068
If you unmarshall the object into a map[string]interface{} then you can just check if a field is there
type unMarshalledObject map[string]interface{}
json.Unmarshal(input, unMarshalledObject)
_, ok := unMarshalledObject["somefield2"]
Go Playground
If struct field is a pointer, JSON decoder will allocate new variable if the field is present or leave it nil if not. So I suggest to use pointers.
type Data struct {
StrField *string
IntField *int
}
...
if data.StrField != nil {
handle(*data.StrField)
}
By using github.com/golang/protobuf/ptypes/struct and jsonpb github.com/golang/protobuf/jsonpb, you can do like this:
func TestFunTest(t *testing.T) {
p := &pb.KnownTypes{}
e := UnmarshalString(`{"val":null}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":1}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":"string"}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{}`, p)
fmt.Println(e, p)
}
Output:
[ `go test -test.run="^TestFunTest$"` | done: 1.275431416s ]
<nil> val:<null_value:NULL_VALUE >
<nil> val:<number_value:1 >
<nil> val:<string_value:"string" >
<nil>
PASS