Golang Json decode validation won't raise error with interface - json

I am a total noob with Golang and would really appreciate any help on the following
I had this code snippet which was working fine
var settings CloudSettings
type CloudSettings struct {
...
A1 *bool `json:"cloud.feature1,omitempty"`
...
}
err = json.NewDecoder(request.Body).Decode(&settings)
An attempt to send an invalid string would raise this error:
curl ... -d '{"cloud.feature1" : "Junk"}'
"message":"Error:strconv.ParseBool: parsing \"Junk\": invalid syntax Decoding request body."
Now, we have a separate LocalSettings struct and the same function needs to handle cloud/local setting decoding conditionally
So, the code was changed to:
var settings interface{} = CloudSettings{}
// If the request header says local settings
settings = LocalSettings{}
/* After this change Decode() no longer raises any error for invalid strings and accepts anything */
err = json.NewDecoder(request.Body).Decode(&settings)
So the question is why do I see this behavior and how would I fix this ?
If I have 2 separate settings variables, then the entire code from that point onwards would just be duplicated which I want to avoid

In the second snippet, you have an interface initialized to a struct, but passing address of that interface. The interface contains a LocalSettings or CloudSetting value, which cannot be overwritten, so the decoder creates a map[string]interface{}, sets the value of the passed interface to point to that, and unmarshal data. When you run the second snippet, you are not initializing the local settings or cloud settings.
Change:
settings=&CloudSettings{}
or
settings=&LocalSettings{}
and
err = json.NewDecoder(request.Body).Decode(settings)
and it should behave as expected

Based on your question, I'm assuming all fields (even the ones with the same name) have a cloud. or local. prefix in the JSON tags. If that's the case, you can simply embed both options into a single type:
type Wrapper struct {
*CloudSettings
*LocalSettings
}
Then unmarshal into this wrapper type. The JSON tags will ensure the correct field on the correct settings type are populated:
wrapper := &Wrapper{}
if err := json.NewDecoder(request.Body).Decode(&wrapper); err != nil {
// handle
}
// now to work out which settings were passed:
if wrapper.CloudSettings == nil {
fmt.Println("Local settings provided!")
// use wrapper.CloudSettings
} else {
fmt.Println("Cloud settings provided!")
// use wrapper.LocalSettings
}
Playground demo
You mention that we expect to see local settings loaded based on a header value. You can simply unmarshal the payload, and then check whether the header matches the settings type that was loaded. If the header specified local settings, but the payload contained cloud settings, simply return an error response.
Still, I'm assuming here that your JSON tags will be different for both setting types. That doesn't always apply, so if my assumption is incorrect, and some fields share the same JSON tags, then a custom Unmarshal function would be the way to go:
func (w *Wrapper) UnmarshalJSON(data []byte) error {
// say we want to prioritise Local over cloud:
l := LocalSettings{}
if err := json.Unmarshal(data, &l); err == nil {
// we could unmarshal into local without a hitch?
w.CloudSettings = nil // ensure this is blanked out
w.LocalSettings = &l // set local
return nil
}
// we should use cloud settings
c := CloudSettings{}
if err := json.Unmarshal(data, &c); err != nil {
return err
}
w.LocalSettings = nil
w.CloudSettings = &c
return nil
}
This way, any conflicts are taken care of, and we can control which settings take precedence. Again, regardless of the outcome of the JSON unmarshalling, you can simply cross check the header value + which settings type was populated, and take it from there.
Lastly, if there's a sizeable overlap between both settings types, you could just as well unmarshal the payload into both types, and populate both fields in the wrapper type:
func (w *Wrapper) UnmarshalJSON(data []byte) error {
*w = Wrapper{} // make sure we start with a clean slate
l := LocalSettings{}
var localFail err
if err := json.Unmarshal(data, &l); err == nil {
w.LocalSettings = &l // set local
} else {
localFail = err
}
c := CloudSettings{}
if err := json.Unmarshal(data, &c); err == nil {
w.CloudSettings = &c
} else if localFail != nil { // both unmarshal calls failed
return err // maybe wrap/return custom error
}
return nil // one or more unmarshals were successful
}
That should do the trick

Related

How to add map into the same struct which is used to deserialize json configs?

I am working on deserializing json into a struct as shown below and it works fine.
type DataConfigs struct {
ClientMetrics []Client `json:"ClientMetrics"`
}
type Client struct {
ClientId int `json:"clientId"`
.....
.....
}
const (
ConfigFile = "clientMap.json"
)
func ReadConfig(path string) (*DataConfigs, error) {
files, err := utilities.FindFiles(path, ConfigFile)
// check for error here
var dataConfig DataConfigs
body, err := ioutil.ReadFile(files[0])
// check for error here
err = json.Unmarshal(body, &dataConfig)
// check for error here
return &dataConfig, nil
}
Now I am trying to build a map of integer and Client using DataConfigs object that was created as shown above in the code. So I created a method to do the job as shown below and I modified ReadConfig method to do that too.
func ReadConfig(path string, logger log.Logger) (*DataConfigs, error) {
files, err := utilities.FindFiles(path, ConfigFile)
// check for error here
var dataConfig DataConfigs
body, err := ioutil.ReadFile(files[0])
// check for error here
err = json.Unmarshal(body, &dataConfig)
// check for error here
idx := BuildIndex(dataConfig)
// now how to add this idx and dataConfig object in one struct?
return &dataConfig, nil
}
func BuildIndex(dataConfig DataConfigs) map[int]Client {
m := make(map[int]Client)
for _, dataConfig := range dataConfig.ClientMetrics {
m[dataConfig.ClientId] = dataConfig
}
return m
}
My confusion is - Should I modify DataConfigs struct to add idx map too and then return that struct from ReadConfig method or should I create a new struct to handle that?
Basically I want to return DataConfigs struct which has ClientMetrics array along with idx map. How can I do this here? I am slightly confuse because I started with golang recently.
This is basically a design question with multiple options. First, I would avoid adding the map to your original DataConfigs type since it does not match the json representation. This could lead to confusion down the road.
Which option to choose depends on your requirements and preferences. Some ideas from the top of my head:
Have you considered returning the map only? After all, you've got every Client in your map. If you need to iterate all Clients you can iterate all values of your map.
Second option is to return the map in addition to DataConfigs. Go allows to return multiple values from a function as you already do for error handling.
Finally, you could wrap DataConfigs and your map in a new simple struct type as you already guessed.

How to set a default value when unmarshal a json to struct [duplicate]

This question already has answers here:
How to specify default values when parsing JSON in Go
(4 answers)
Closed 3 years ago.
I want to set a default value to a field when unmarshal from a json string.
I know i can set the value i want before unmarshaling, I think it's not a beautiful way.
Is there any way else, like using a "default" tag?
func main() {
in := "{}"
myStruct := StructTest{}
json.Unmarshal([]byte(in), &myStruct)
fmt.Println(myStruct)
}
type StructTest struct {
V int64 `default:1`
}
What you can do is define a custom unmarshal function and decide there if you want to use the default value or not. In case you have other fields in StructTest you will want to create an alias for StructTest in UnmarshalJSON so that other fields will still be treated the same, while V will be overridden.
The snippet below shows one way to do it, also check out this working Go playground example.
type StructTest struct {
V int64
Other string // this field should be unmarshaled the regular way
}
func (st *StructTest) UnmarshalJSON(data []byte) error {
// create alias to prevent endless loop
type Alias StructTest
tmp := struct {
*Alias
V *int64
}{
Alias: (*Alias)(st),
}
// unmarshal into temporary struct
err := json.Unmarshal(data, &tmp)
if err != nil {
return err
}
// check if V was supplied in JSON and set default value if it wasn't
if tmp.V == nil {
st.V = 1 // default
} else {
st.V = *tmp.V
}
return nil
}
EDIT:
Actually for this simple example it can be done even simpler:
func (st *StructTest) UnmarshalJSON(data []byte) error {
st.V = 1 // set default value before unmarshaling
type Alias StructTest // create alias to prevent endless loop
tmp := (*Alias)(st)
return json.Unmarshal(data, tmp)
}
The short answer is no, and for cases like this:
type T struct {
Field1 int
Field2 int
}
...
func foo(data []byte) error {
var x T
if err := json.Unmarshal(data, &x); err != nil {
return err
}
// now set Field1 and/or Field2 to default values if needed
// ... but this is hard to do ...
use(x)
return nil
}
it's simple and easy to do it the other way, i.e.:
func foo(data []byte) error {
x := T{Field1: 99 /* red ballons */, Field2: 42 /* The Answer */ }
if err := json.Unmarshal(data, &x); err != nil {
return err
}
// now Field1 and/or Field2 are already set to default values if needed
use(x)
return nil
}
But what if the default is hard to compute? For instance:
type T2 struct {
Answer int
Question string
}
and function foo should have a default Answer of 42 as before, but the default Question should be the one that the mice were trying to compute, and obviously we don't want to spend a few millenia computing it if we don't have to. So we can't pre-initialize x, and we need to know if a question was provided.1
Yet another alternative, of course, is to decode into a struct with a pointer, then convert that nullable thing to the struct that doesn't have a pointer; we know whether the nullable variant's field was filled in or not because it's non-nil if it was filled-in. This produces code of this sort:
type T_ struct {
Answer int
Question *string
}
and fill in one variable x_ of type T_:
func foo(data []byte) error {
var x T2
x_ := T_{Answer: 42}
if err := json.Unmarshal(data, &x_); err != nil {
return err
}
x.Answer = x_.Answer
if x_.Question = nil {
x.Question = computeTheQuestion(x_.Answer)
} else {
x.Question = *x_.Question
}
use(x)
return nil
}
However, once again I lament (at least slightly) the ability to unmarshal json data with code like this hypothetical interface:
func foo(data []byte) error {
var objectish map[string]interface{}
if err := json.Unmarshal(data, &objectish); err != nil {
return err
}
x := T2{Answer: 42}
if err := json.ReUnmarshal(objectish, &x); err != nil {
return err
}
// We now know that the object-ish decoded JSON has the right
// "shape" for a T, and variable x is filled in with a default
// Answer. Its Question is the empty string if there was an
// empty Question, or if there was *no* Question at all, so
// now let's find out if we need to compute the right Question.
if _, ok := objectish["Question"]; !ok {
x.Question = computeTheQuestion(x.Answer)
}
use(x) // pass x to a hoopy frood
return nil
}
This hypothetical ReUnmarshal—which could actually just be Unmarshal itself, really—would, if given an interface{}, treat its value as resulting from an earlier Unmarshal and just re-type the result. If given a map[string]interface{} it would re-type the object. If given a map[string]map[string]interface{} it would do the obvious thing here as well, and so on. The only change the user sees is that Unmarshal (or ReUnmarshal, if changing the type signature is too rude) now takes:
[]byte, or
string, probably, just for convenience, or
json.RawMessage, because that already does almost the right thing, or
map[string]T for T being any of the types it accepts (including map recursively)
and it does the obvious thing with each of these.
Note that this is pretty similar to using json.RawMessage. However, when using json.RawMessage, we're right back to writing out a variant of the original struct type, with the same field names. See this example in the Go Playground, where we have to declare x_ (rather than objectish) with the type that uses the json.RawMessage argument.
(Alternatively, you can create a type with its own unmarshal function, as in lmazgon's answer. But again you must invent a type, rather than just decoding right into the already-supplied target type.)
1In this case, we could use Question *string, or assume that null strings are not allowed, or something, but this example is pretty reductionist. Suppose instead that we're supposed to compute a bignum with an accurate pi to some number of places, or any other hard but realistic computation. The point here is that there can be situations where pre-loading the defaults is relatively expensive, so we'd like to avoid that.

find and delete nested json object in Go

I have a json document of a Kubernetes Pod, here's an example:
https://github.com/itaysk/kubectl-neat/blob/master/test/fixtures/pod-1-raw.json
I'd like to traverse spec.containers[i].volumeMounts and delete those volumeMount objects where the .name starts with "default-token-". Note that both containers and volumeMounts are arrays.
Using jq it took me 1 min to write this 1 line: try del(.spec.containers[].volumeMounts[] | select(.name | startswith("default-token-"))). I'm trying to rewrite this in Go.
While looking for a good json library I settled on gjson/sjson.
Since sjson doesn't support array accessors (the # syntax), and gjson doesn't support getting the path of result, I looked for workarounds.
I've tried using Result.Index do delete the the result from the byte slice directly, and succeeded, but for the query I wrote (spec.containers.#.volumeMounts.#(name%\"default-token-*\")|0) the Index is always 0 (I tried different variations of it, same result).
So currently I have some code 25 line code that uses gjson to get spec.containers.#.volumeMounts and iterate it's way through the structure and eventually use sjson.Delete to delete.
It works, but it feels way more complicated then I expected it to be.
Is there a better way to do this in Go? I'm willing to switch json library if needed.
EDIT: I would prefer to avoid using a typed schema because I may need to perform this on different types, for some I don't have the full schema.
(also removed some distracting details about my current implemetation)
The easiest thing to do here is parse the JSON into an object, work with that object, then serialise back into JSON.
Kubernetes provides a Go client library that defines the v1.Pod struct you can Unmarshal onto using the stdlib encoding/json:
// import "k8s.io/api/core/v1"
var pod v1.Pod
if err := json.Unmarshal(podBody, &pod); err != nil {
log.Fatalf("parsing pod json: %s", err)
}
From there you can read pod.Spec.Containers and their VolumeMounts:
// Modify.
for c := range pod.Spec.Containers {
container := &pod.Spec.Containers[c]
for i, vol := range container.VolumeMounts {
if strings.HasPrefix(vol.Name, "default-token-") {
// Remove the VolumeMount at index i.
container.VolumeMounts = append(container.VolumeMounts[:i], container.VolumeMounts[i+1:]...)
}
}
}
https://play.golang.org/p/3r5-XKIazhK
If you're worried about losing some arbitrary JSON which might appear in your input, you may instead wish to define var pod map[string]interface{} and then type-cast each of the properties within as spec, ok := pod["spec"].(map[string]interface{}), containers, ok := spec["containers"].([]map[string]interface) and so on.
Hope that helps.
ps. The "removing" is following https://github.com/golang/go/wiki/SliceTricks#delete
To take a totally different approach from before, you could create a
type Root struct {
fields struct {
Spec *Spec `json:"spec,omitempty"`
}
other map[string]interface{}
}
with custom UnmarshalJSON which unmarshals into both fields and other, and custom MarshalJSON which sets other["spec"] = json.RawMessage(spec.MarshalJSON()) before returning json.Marshal(other):
func (v *Root) UnmarshalJSON(b []byte) error {
if err := json.Unmarshal(b, &v.fields); err != nil {
return err
}
if v.other == nil {
v.other = make(map[string]interface{})
}
if err := json.Unmarshal(b, &v.other); err != nil {
return err
}
return nil
}
func (v *Root) MarshalJSON() ([]byte, error) {
var err error
if v.other["spec"], err = rawMarshal(v.fields.Spec); err != nil {
return nil, err
}
return json.Marshal(v.other)
}
func rawMarshal(v interface{}) (json.RawMessage, error) {
b, err := json.Marshal(v)
if err != nil {
return nil, err
}
return json.RawMessage(b), nil
}
You then define these sort of types all of the way down through .spec.containers.volumeMounts and have a Container.MarshalJSON which throws away and VolumeMounts we don't like:
func (v *Container) MarshalJSON() ([]byte, error) {
mounts := v.fields.VolumeMounts
for i, mount := range mounts {
if strings.HasPrefix(mount.fields.Name, "default-token-") {
mounts = append(mounts[:i], mounts[i+1:]...)
}
}
var err error
if v.other["volumeMounts"], err = rawMarshal(mounts); err != nil {
return nil, err
}
return json.Marshal(v.other)
}
Full playground example: https://play.golang.org/p/k1603cchwC7
I wouldn't do this.

How does json.Unmarshal() work and how is it stored in a variable?

Just started with Go and I have a small doubt from a tutorial I am following. I read that Unmarshall is some kind of JSON encoding, my doubt here is: err = json.Unmarshal(body, &p) why are we assigning the encoded body to err and how is p.Stuff.Fruit getting the value when I can't see anything assigned to p.
Note : produce is different package which contains some type and arrays.*
func main() {
url := "http://localhost:12337"
res, err := http.Get(url)
if err != nil {
panic(err)
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
var p produce.Payload
err = json.Unmarshal(body, &p) // I cant get this
if err != nil {
panic(err)
}
// how are these getting the values assigned to them
fmt.Println(p.Stuff.Fruit)
fmt.Println(p.Stuff.Veggies)
}
my doubt here is: err = json.Unmarshal(body, &p) why are we assigning the encoded body to err
You don't. You pass the body to the json.Unmarshal() function, and you assign its return value to the err variable, which tells you if unmarshaling failed (or succeeded if err is nil).
how is p.Stuff.Fruit getting the value when I can't see anything assigned to p
You pass the address of p to json.Unmarshal(), so it has a pointer to your variable, and so if it modifies the pointed value (pointed by the pointer), it modifies the same value you have (p).
See this simple example:
func change(p *int) {
*p = 3
}
func main() {
var i int
fmt.Println("Before", i)
change(&i)
fmt.Println("After", i)
}
Output (try it on the Go Playground):
Before 0
After 3
In the main() function we don't assign anything to the local variable i, but we pass its address to the change() function, which modifies the pointed value, and so if we print the value of i after the change() call, we see its value changed.
You pass the address of p to json.Unmarshal(), so it will be capable of modifying the value stored in p. Under the hood, the json package uses reflection (package reflect) to discover the runtime type of p and modify it according to the JSON document you pass to it to parse.
You have created a pointer to payload using &p address operator. Now It saves the value at an address pointed by p. According to Golang spec Unmarshal takes data in bytes format and interface{} to wrap the variable storing the data at an address.
Unmarshal parses the JSON-encoded data and stores the result in the
value pointed to by v. If v is nil or not a pointer, Unmarshal returns
an InvalidUnmarshalError.
Unmarshal function returns an error
func Unmarshal(data []byte, v interface{}) error
var m Message
err := json.Unmarshal(b, &m)
which is why it is assigned to a variable. You can surpass it using _ operator but it is not a good approach
_ := json.Unmarshal(b, &m)

Encode Set data structure in Golang

I have a Set data structure implemented in Go with the basic operations like Add, Remove, Difference, Union. I am trying to send a http request using the json encoder to encode the request body which contains the object of the form map[string]Set. The Set data structure is defined below:
type Set map[interface{}]struct{}
func NewSet() Set {
set := make(Set)
return set
}
The encoder looks like this:
func (req *Request) BodyContentInJson (val interface{}) error {
buf := bytes.NewBuffer(nil)
enc := json.NewEncoder(buf)
if err := enc.Encode(val); err != nil {
return err
}
req.Obj = val
req.Body = buf
req.BodySize = int64(buf.Len())
return nil
}
This code fails at
if err := enc.Encode(val); err != nil {
return err
}
giving an error:{"errors":["json: unsupported type: Set"]}. Also, the type of val is map[string]interface{}when I debugged it.
How could I possibly encode and marshal/unmarshal JSON content here using the Go's encoder ?
You could write your own UnmarshalJSON method on the *Set type which would then be used by the json.Encoder to encode the json data into the Set. Here's a simple example https://play.golang.org/p/kx1E-jDu5e.
By the way, the reason you're getting the error is because a map key of type interface{} is not supported by the encoding/json package. (https://github.com/golang/go/blob/master/src/encoding/json/encode.go#L697-L699)