Golang: Validate Struct Fields in Slice Items - json

I'm new to Golang.
golang version: 1.17.8
validator: "github.com/go-playground/validator/v10"
I want to validate an incoming JSON payload after loaded into nested struct data structure.
Here's my incoming JSON payload,
{
"name": "Yomiko",
"address": {
"city": "Tokyo",
"street": "Shibaura St"
},
"children":[
{
"lastName": "Takayashi"
}
],
"isEmployed": false
}
Here's my user.go file,
package main
type User struct {
Name string
Address *Address `validate:"required"`
Children []*Child
IsEmployed *bool `validate:"required"`
}
type Address struct {
City string `validate:"required"`
Street string `validate:"required"`
}
type Child struct {
Title string `validate:"required"`
FirstName string
LastName string `validate:"required"`
}
Here's my test function,
func TestUserPayload(t *testing.T) {
actualUserPayload := NewUserPayloadFromFile("userpayload.json")
validate := validator.New()
err := validate.Struct(actualUserPayload)
if err != nil {
t.Error("Validation Error: ", err)
}
}
This test passes. However, I expected it to fail as Child.Title is marked as required. I expected the following error,
Validation Error: Key: 'Child.Title' Error:Field validation for 'Title' failed on the 'required' tag
However, when I loop through the children slice and validate each child struct as follows the test fails as expected,
func TestUserPayload(t *testing.T) {
actualUserPayload := NewUserPayloadFromFile("userpayload.json")
validate := validator.New()
err := validate.Struct(actualUserPayload)
if err != nil {
t.Error("Validation Error: ", err)
}
children := actualUserPayload.Children
for _, child := range children {
err := validate.Struct(child)
if err != nil {
t.Error("Validation Error: ", err)
}
}
}
Is there a straightforward way to do this validation of the items in a slice of structs?

According to the documentation of the validator package, you can use dive in your struct tag to get this behavior. This causes the validator to also validate the nested struct/slice/etc.
So you would need to update your User struct to this:
type User struct {
Name string
Address *Address `validate:"required"`
Children []*Child `validate:"dive"`
IsEmployed *bool `validate:"required"`
}
Here it is working in Go Playground

Related

Unmarshalling of JSON with dynamic keys

I have a scenario where the JSON that has dynamic set of fields that need to get unmarshalled in to a struct.
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
type Status struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status map[string]Status `json:"status"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
var person Person
if err := dec.Decode(&person); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(person)
fmt.Println(person.Status["bvu62fu6dq"])
}
}
The output:
{john 23 map[]}
{false}
When it gets unmarshalled, the nested status struct is not being correctly resolved to the value in the JSON (shows false even with true value in JSON), is there any issue in the code?
Your types don't really match with the JSON you have:
type Status struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status map[string]Status `json:"status"`
}
Maps to JSON that looks something like this:
{
"name": "foo",
"age": 12,
"status": {
"some-string": {
"Status": true
}
}
}
The easiest way to unmarshal data with a mix of known/unknown fields in a go type is to have something like this:
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"` // skip this key
}
Then, first unmarshal the known data:
var p Person
if err := json.Unmarshal([]byte(jsonStream), &p); err != nil {
panic(err)
}
// then unmarshal the rest of the data
if err := json.Unmarshal([]byte(jsonStream), &p.Random); err != nil {
panic(err)
}
Now the Random map will contain every and all data, including the name and age fields. Seeing as you've got those tagged on the struct, these keys are known, so you can easily delete them from the map:
delete(p.Random, "name")
delete(p.Random, "age")
Now p.Random will contain all the unknown keys and their respective values. These values apparently will be an object with a field status, which is expected to be a boolean. You can set about using type assertions and convert them all over to a more sensible type, or you can take a shortcut and marshal/unmarshal the values. Update your Person type like so:
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"`
Statuses map[string]Status `json:"-"`
}
Now take the clean Random value, marshal it and unmarshal it back into the Statuses field:
b, err := json.Marshal(p.Random)
if err != nil {
panic(err)
}
if err := json.Unmarshal(b, &p.Statuses); err != nil {
panic(err)
}
// remove Random map
p.Random = nil
The result is Person.Statuses["bvu62fu6dq"].Status is set to true
Demo
Cleaning this all up, and marshalling the data back
Now because our Random and Statuses fields are tagged to be ignored for JSON marshalling (json:"-"), marshalling this Person type won't play nice when you want to output the original JSON from these types. It's best to wrap this logic up in a custom JSON (un)-Marshaller interface. You can either use some intermediary types in your MarshalJSON and UnmarshalJSON methods on the Person type, or just create a map and set the keys you need:
func (p Person) MarshalJSON() ([]byte, error) {
data := make(map[string]interface{}, len(p.Statuses) + 2) // 2 being the extra fields
// copy status fields
for k, v := range p.Statuses {
data[k] = v
}
// add known keys
data["name"] = p.Name
data["age"] = p.Age
return json.Marshal(data) // return the marshalled map
}
Similarly, you can do the same thing for UnmarshalJSON, but you'll need to create a version of the Person type that doesn't have the custom handling:
type intermediaryPerson struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"`
}
// no need for the tags and helper fields anymore
type Person struct {
Name string
Age int
Statuses map[string]Status // Status type doesn't change
}
func (p *Person) UnmarshalJSON(data []byte) error {
i := intermediaryPerson{}
if err := json.Unmarshal(data, &i); err != nil {
return err
}
if err := json.Unmarshal(data, &i.Random); err != nil {
return err
}
delete(i.Random, "name")
delete(i.Random, "age")
stat, err := json.Marshal(i.Random)
if err != nil {
return err
}
// copy known fields
p.Name = i.Name
p.Age = i.Age
return json.Unmarshal(stat, &p.Statuses) // set status fields
}
In cases like this, it's common to create a type that handles the known fields and embed that, though:
type BasePerson struct {
Name string `json:"name"`
Age int `json:"age"`
}
and embed that in both the intermediary and the "main"/exported type:
type interPerson struct {
BasePerson
Random map[string]interface{} `json:"-"`
}
type Person struct {
BasePerson
Statuses map[string]Status
}
That way, you can just unmarshal the known fields directly into the BasePerson type, assign it, and then handle the map:
func (p *Person) UnmarshalJSON(data []byte) error {
base := BasePerson{}
if err := json.Unmarshal(data, &base); err != nil {
return err
}
p.BasePerson = base // takes care of all known fields
unknown := map[string]interface{}{}
if err := json.Unmarshal(data, unknown); err != nil {
return err
}
// handle status stuff same as before
delete(unknown, "name") // remove known fields
// marshal unknown key map, then unmarshal into p.Statuses
}
Demo 2
This is how I'd go about it. It allows for calls to json.Marshal and json.Unmarshal to look just like any other type, it centralises the handling of unknown fields in a single place (the implementation of the marshaller/unmarshaller interface), and leaves you with a single Person type where every field contains the required data, in a usable format. It's a tad inefficient in that it relies on unmarshalling/marshalling/unmarshalling the unknown keys. You could do away with that, like I said, using type assertions and iterating over the unknown map instead, faffing around with something like this:
for k, v := range unknown {
m, ok := v.(map[string]interface{})
if !ok {
continue // not {"status": bool}
}
s, ok := m["status"]
if !ok {
continue // status key did not exist, ignore
}
if sb, ok := s.(bool); ok {
// ok, we have a status bool value
p.Statuses[k] = Status{
Status: sb,
}
}
}
But truth be told, the performance difference won't be that great (it's micro optimisation IMO), and the code is a tad too verbose to my liking. Be lazy, optimise when needed, not whenever
Type doesn't meet with your json value.
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
For above json your code should look like below snnipet to work (some modifications in your existing code).
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"strings"
)
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
type bvu62fu6dq struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status bvu62fu6dq `json:"bvu62fu6dq"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
var person Person
if err := dec.Decode(&person); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(person)
fmt.Println(person.Status)
}
}
Based on your json data you have to map with type fields.
Run code snippet

How to unmarshal a field that can be an array or a string in Go?

I'm trying to unmarshal this file:
{
"#babel/code-frame#7.0.0": {
"licenses": "MIT",
"repository": "https://github.com/babel/babel/tree/master/packages/babel-code-frame",
"publisher": "Sebastian McKenzie",
"email": "sebmck#gmail.com",
"path": "/Users/lislab/workspace/falcon-enrolment/frontend-customer/node_modules/#babel/code-frame",
"licenseFile": "/Users/lislab/workspace/falcon-enrolment/frontend-customer/node_modules/#babel/code-frame/LICENSE"
},
"json-schema#0.2.3": {
"licenses": [
"AFLv2.1",
"BSD"
],
"repository": "https://github.com/kriszyp/json-schema",
"publisher": "Kris Zyp",
"path": "/Users/lislab/workspace/falcon-enrolment/frontend-customer/node_modules/json-schema",
"licenseFile": "/Users/lislab/workspace/falcon-enrolment/frontend-customer/node_modules/json-schema/README.md"
}
}
into this struct:
type Dependency struct {
Name string
URL string
Version string
License string
}
using these instructions:
dependencies := map[string]*json.RawMessage{}
err = json.Unmarshal(file, &dependencies)
// boilerplate
for key, value := range dependencies {
depVal := map[string]string{}
err = json.Unmarshal(*value, &depVal)
// boilerplate
result = append(result, depVal)
}
The problem with this is that in "json-schema#0.2.3" we have an array of licenses instead of a string, and due to that I obviously get
json: cannot unmarshal array into Go value of type string
Is there a way to automatically deal with the field license which can be an array or a string?
Thanks
Unfortunately, there is no real automatic solution for this provided by the json package.
But you can unmarshal the dependencies to a map[string]*json.RawMessage instead of map[string]string. json.RawMessage is just a []byte, so you can decide on the type of the message based on the first byte.
Example:
for _, value := range dependencies {
depVal := map[string]*json.RawMessage{}
_ = json.Unmarshal(*value, &depVal)
// check if the first character of the RawMessage is a bracket
if rune([]byte(*depVal["licenses"])[0]) == '[' {
var licenses []string
json.Unmarshal(*depVal["licenses"], &licenses)
fmt.Println(licenses)
// do something with the array
}
result = append(result, Dependency{
URL: string(*depVal["repository"]),
License: string(*depVal["licenses"]),
})
}
Another solution would be to use 2 structs. One contains the dependencies as string, the other as array. You can then try to call json.Unmarshal on both of them.
Example:
type Dependency struct {
Licenses string
// other fields
}
type DependencyWithArr struct {
Licenses []string
// other fields
}
// in your function
for _, value := range dependencies {
type1 := Dependency{}
type2 := DependencyWithArr{}
err = json.Unmarshal(*value, &type1)
if err != nil {
err = json.Unmarshal(*value, &type2)
// use the array type
} else {
// use the single string type
}
}

How to list unknown fields after json parsing

Imagine we have following Go structs:
type Config struct {
Name string `json:"name,omitempty"`
Params []Param `json:"params,omitempty"`
}
type Param struct {
Name string `json:"name,omitempty"`
Value string `json:"value,omitempty"`
}
and following json:
{
"name": "parabolic",
"subdir": "pb",
"params": [{
"name": "input",
"value": "in.csv"
}, {
"name": "output",
"value": "out.csv",
"tune": "fine"
}]
}
and we do unmarshalling:
cfg := Config{}
if err := json.Unmarshal([]byte(cfgString), &cfg); err != nil {
log.Fatalf("Error unmarshalling json: %v", err)
}
fmt.Println(cfg)
https://play.golang.org/p/HZgo0jxbQrp
Output would be {parabolic [{input in.csv} {output out.csv}]} which makes sense - unknown fields were ignored.
Question: how to find out which fields were ignored?
I.e. getIgnoredFields(cfg, cfgString) would return ["subdir", "params[1].tune"]
(There is a DisallowUnknownFields option but it's different: this option would result Unmarshal in error while question is how to still parse json without errors and find out which fields were ignored)
Not sure if that is the best way but what I did is:
If current-level type is map:
Check that all map keys are known.
It could be either if keys are struct field names or map keys.
If not known - add to list of unknown fields
Repeat recursively for value that corresponds to each key
If current level type is array:
Run recursively for each element
Code:
// ValidateUnknownFields checks that provided json
// matches provided struct. If that is not the case
// list of unknown fields is returned.
func ValidateUnknownFields(jsn []byte, strct interface{}) ([]string, error) {
var obj interface{}
err := json.Unmarshal(jsn, &obj)
if err != nil {
return nil, fmt.Errorf("error while unmarshaling json: %v", err)
}
return checkUnknownFields("", obj, reflect.ValueOf(strct)), nil
}
func checkUnknownFields(keyPref string, jsn interface{}, strct reflect.Value) []string {
var uf []string
switch concreteVal := jsn.(type) {
case map[string]interface{}:
// Iterate over map and check every value
for field, val := range concreteVal {
fullKey := fmt.Sprintf("%s.%s", keyPref, field)
subStrct := getSubStruct(field, strct)
if !subStrct.IsValid() {
uf = append(uf, fullKey[1:])
} else {
subUf := checkUnknownFields(fullKey, val, subStrct)
uf = append(uf, subUf...)
}
}
case []interface{}:
for i, val := range concreteVal {
fullKey := fmt.Sprintf("%s[%v]", keyPref, i)
subStrct := strct.Index(i)
uf = append(uf, checkUnknownFields(fullKey, val, subStrct)...)
}
}
return uf
}
Full version: https://github.com/yb172/json-unknown/blob/master/validator.go

How to create multiple validation methods for one endpoint?

I want to make a validation api in order to validate a set of json requests regarding specific set of rules. To do that I want to use just one endpoint and call functions that correspond to the specific json struct. I know that there is no method overloading in go so I am kind of stumped.
...
type requestBodyA struct {
SomeField string `json:"someField"`
SomeOtherField string `json:"someOtherField"`
}
type requestBodyB struct {
SomeDifferentField string `json:"someDifferentField"`
SomeOtherDifferentField string `json:"someOtherDifferentField"`
}
type ValidationService interface {
ValidateRequest(ctx context.Context, s string) (err error)
}
type basicValidationService struct{}
...
So in order to validate lots of different json requests, is it better to create structs for each and every json request? Or should I create these dynamically? How can I know what kind of request is sent if I only have one endpoint?
If you have a single endpoint/rpc that has to accept different JSON types, you'll need to tell it how to distinguish between them, somehow. One option is to have something like:
type request struct {
bodyA *requestBodyA
bodyB *requestBodyB
}
Then, populate these fields in a container JSON object appropriately. The json module will only populate bodyA if a bodyA key is present, otherwise leaving it a nil, and so on.
Here's a more complete example:
type RequestBodyFoo struct {
Name string
Balance float64
}
type RequestBodyBar struct {
Id int
Ref int
}
type Request struct {
Foo *RequestBodyFoo
Bar *RequestBodyBar
}
func (r *Request) Show() {
if r.Foo != nil {
fmt.Println("Request has Foo:", *r.Foo)
}
if r.Bar != nil {
fmt.Println("Request has Bar:", *r.Bar)
}
}
func main() {
bb := []byte(`
{
"Foo": {"Name": "joe", "balance": 4591.25}
}
`)
var req Request
if err := json.Unmarshal(bb, &req); err != nil {
panic(err)
}
req.Show()
var req2 Request
bb = []byte(`
{
"Bar": {"Id": 128992, "Ref": 801472}
}
`)
if err := json.Unmarshal(bb, &req2); err != nil {
panic(err)
}
req2.Show()
}
Another option is to do it more dynamically with maps, but it's likely that the method above will be sufficient.

Clean way to conditionally unmarshal JSON to struct

I'm sending requests to a JSON API, and it either returns an error...
{
"error": {
"code": 404,
"message": "Document not found.",
"status": "NOT_FOUND"
}
}
or the data.
{
"name": "projectname",
"fields": {
"userId": {
"stringValue": "erw9384rjidfge"
}
},
"createTime": "2018-06-28T00:52:25.638791Z",
"updateTime": "2018-06-28T00:52:25.638791Z"
}
Here are the corresponding structs
type HttpError struct {
Code int `json:"code"`
Message string `json:"message"`
Status string `json:"status"`
}
type Document struct {
Name string `json:"name"`
Fields struct {
UserID struct {
StringValue string `json:"stringValue"`
} `json:"userId"`
} `json:"fields"`
CreateTime time.Time `json:"createTime"`
UpdateTime time.Time `json:"updateTime"`
}
Once I get the response, how do I cleanly/concisely unmarshal to the correct struct? I've seen a lot of ugly solutions (maybe Go's fault instead of the writers).
func getDocument() {
resp, _ := httpClient.Get("example.com")
defer resp.Body.Close()
bodyBytes, _ := ioutil.ReadAll(resp.Body)
var data map[string]interface{}
// How to unmarshal to either HttpError or Document??
err = json.Unmarshal([]byte(bodyBytes), &data)
}
By the way, I can't use the Go Firestore client library because reasons.
You can use an struct type inside your unmarshal method; with pointers to establish what's been unmarshalled.
Note: This code assumes there is no overlap of top level json keys... error / name / fields / etc.
type outer struct {
*HttpError `json:"error"`
*Document
}
var out outer
if err := json.Unmarshal(bodyBytes, &out); err != nil {
// error handling
}
if out.HttpErr != nil {
// handle error json case
}
// Here you can use out.Document, probably worth check if it is nil first.
Runnable example