How to create multiple validation methods for one endpoint? - json

I want to make a validation api in order to validate a set of json requests regarding specific set of rules. To do that I want to use just one endpoint and call functions that correspond to the specific json struct. I know that there is no method overloading in go so I am kind of stumped.
...
type requestBodyA struct {
SomeField string `json:"someField"`
SomeOtherField string `json:"someOtherField"`
}
type requestBodyB struct {
SomeDifferentField string `json:"someDifferentField"`
SomeOtherDifferentField string `json:"someOtherDifferentField"`
}
type ValidationService interface {
ValidateRequest(ctx context.Context, s string) (err error)
}
type basicValidationService struct{}
...
So in order to validate lots of different json requests, is it better to create structs for each and every json request? Or should I create these dynamically? How can I know what kind of request is sent if I only have one endpoint?

If you have a single endpoint/rpc that has to accept different JSON types, you'll need to tell it how to distinguish between them, somehow. One option is to have something like:
type request struct {
bodyA *requestBodyA
bodyB *requestBodyB
}
Then, populate these fields in a container JSON object appropriately. The json module will only populate bodyA if a bodyA key is present, otherwise leaving it a nil, and so on.
Here's a more complete example:
type RequestBodyFoo struct {
Name string
Balance float64
}
type RequestBodyBar struct {
Id int
Ref int
}
type Request struct {
Foo *RequestBodyFoo
Bar *RequestBodyBar
}
func (r *Request) Show() {
if r.Foo != nil {
fmt.Println("Request has Foo:", *r.Foo)
}
if r.Bar != nil {
fmt.Println("Request has Bar:", *r.Bar)
}
}
func main() {
bb := []byte(`
{
"Foo": {"Name": "joe", "balance": 4591.25}
}
`)
var req Request
if err := json.Unmarshal(bb, &req); err != nil {
panic(err)
}
req.Show()
var req2 Request
bb = []byte(`
{
"Bar": {"Id": 128992, "Ref": 801472}
}
`)
if err := json.Unmarshal(bb, &req2); err != nil {
panic(err)
}
req2.Show()
}
Another option is to do it more dynamically with maps, but it's likely that the method above will be sufficient.

Related

How to handle missing fields in a JSON response dynamically in Go

I'm working on a Go wrapper for an API and I noticed that two of the JSON fields stay empty when they don't have any data.
Basically the API returns a set of information on a given url, and if it was visited at least once, everything is okay and I get a full json that I then Unmarshal into a struct:
{
"stats":{
"status":1,
"date":"09.07.2019",
"title":"Test",
"devices":{
"dev":[
{
"tag":"Desktop"
}
],
"sys":[
{
"tag":"GNU/Linux "
},
{
"tag":"Windows 10"
}
],
"bro":[
{
"tag":"Firefox 67.0"
},
{
"tag":"Chrome 62.0"
}
]
},
"refs":[
{
"link":"www.google.com"
}
]
}
}
This is the struct I'm using:
type Stats struct {
Stats struct {
Status int `json:"status"`
Date string `json:"date"`
Title string `json:"title"`
Devices struct {
Dev []struct {
Tag string `json:"tag"`
} `json:"dev"`
Sys []struct {
Tag string `json:"tag"`
} `json:"sys"`
Bro []struct {
Tag string `json:"tag"`
} `json:"bro"`
} `json:"devices"`
Refs []struct {
Link string `json:"link"`
} `json:"refs"`
} `json:"stats"`
}
When a new url is given, then things become a little bit weird:
{
"stats": {
"status": 1,
"date": "09.07.2019",
"title": "Test2",
"devices": [
],
"refs": [
]
}
}
As you can see, the fields "dev", "sys" and "bro" just disappear because they're not used and when I try to Unmarshal the JSON into the same struct I get json: cannot unmarshal array into Go struct field Stats.device of type [...]
I tried to use two different structs to handle both the responses but I'm sure that there's a way to handle them gracefully with just one.
Any help would be appreciated, thanks!
I finally managed to make it work with an ugly workaround.
I changed my struct to
type Stats struct {
Status int `json:"status"`
Date string `json:"date"`
Title string `json:"title"`
Devices interface{} `json:"devices"`
Refs interface{} `json:"refs"`
}
Then I can finally Unmarshal the JSON in both cases, but I get a map[string]interface{} when an object is passed and an empty interface{} when an empty array is passed. In order to fix this inconsistency, I simply check for the data type and force the use of a JSON intermediate conversion in order to unpack the map[string]interface{} value inside a custom Devices struct:
// Devices contains devices information
type Devices struct {
Dev []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"dev"`
Sys []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"sys"`
Bro []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"bro"`
}
The algorithms I use are the following:
//ForceDevicesToRightType uses a json conversion as intermediary for filling the Stats.Devices
// struct with map[string]interface{} values
func ForceDevicesToRightType(dev interface{}) (Devices, error) {
temp, err := json.Marshal(dev)
if err != nil {
return Devices{}, err
}
// Use a temporary variable of the right type
var devices Devices
err = json.Unmarshal(temp, &devices)
if err != nil {
return Devices{}, err
}
return devices, nil
}
// ForceRefsToRightType uses a json conversion as intermediary for filling the Stats.Refs
// struct with map[string]interface{} values
func ForceRefsToRightType(refs interface{}) (Refs, error) {
temp, err := json.Marshal(refs)
if err != nil {
return Refs{}, err
}
// Use a temporary variable of the right type
var references Refs
err = json.Unmarshal(temp, &references)
if err != nil {
return Refs{}, err
}
return references, nil
}
Since the compiler knows that both Devices and Refs fields are interface{} I cannot simply access any methods after the conversion, so I simply make a cast of the right type and everything works fine.
For example, if I wanted to access the Dev sub-struct, this is the proper way:
y, _ := GetStats()
fmt.Println(y.Devices.(Devices).Dev)
It's ugly, but it works.
Thank you very much for your help, I hope that this method will save you an headache!

How do I capitalize all keys in a JSON array?

I'm reading a file.json into memory. It's an array of objects, sample:
[
{"id":123123,"language":"ja-JP","location":"Osaka"}
,{"id":33332,"language":"ja-JP","location":"Tokyo"}
,{"id":31231313,"language":"ja-JP","location":"Kobe"}
]
I want to manipulate certain keys in this JSON file, so that they start with uppercase. Meaning
"language" becomes "Language" each time it's found. What I've done so far is to make a struct representing each object, as such:
type sampleStruct struct {
ID int `json:"id"`
Language string `json:"Language"`
Location string `json:"Location"`
}
Here, I define the capitalization. Meaning, id shouldn't be capitalized, but location and language should.
Rest of the code is as such:
func main() {
if len(os.Args) < 2 {
fmt.Println("Missing filename parameter.")
return
}
translationfile, err := ioutil.ReadFile(os.Args[1])
fileIsValid := isValidJSON(string(translationfile))
if !fileIsValid {
fmt.Println("Invalid JSON format for: ", os.Args[1])
return
}
if err != nil {
fmt.Println("Can't read file: ", os.Args[1])
panic(err)
}
}
func isValidJSON(str string) bool {
var js json.RawMessage
return json.Unmarshal([]byte(str), &js) == nil
}
// I'm unsure how to iterate through the JSON objects and only uppercase the objects matched in my struct here.
func upperCaseSpecificKeys()
// ...
Desired output, assuming the struct represents the whole data object, transform each key as desired:
[
{"id":123123,"Language":"ja-JP","Location":"Osaka"}
,{"id":33332,"Language":"ja-JP","Location":"Tokyo"}
,{"id":31231313,"Language":"ja-JP","Location":"Kobe"}
]
The documentation on json.Unmarshal says (with added emphasis):
To unmarshal JSON into a struct, Unmarshal matches incoming object
keys to the keys used by Marshal (either the struct field name or its
tag), preferring an exact match but also accepting a case-insensitive
match
See example here: https://play.golang.org/p/1vv8PaQUOfg
One way is to implement custom marshal method, although not very flexible:
type upStruct struct {
ID int `json:"id"`
Language string
Location string
}
type myStruct struct {
ID int `json:"id"`
Language string `json:"language"`
Location string `json:"location"`
}
func (m myStruct) MarshalJSON() ([]byte, error) {
return json.Marshal(upStruct(m))
}
....
func main() {
var mySArr []myStruct
// 1. Unmarshal the input
err := json.Unmarshal([]byte(myJson), &mySArr)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Input: \n%+v\n", mySArr)
// 2. Then, marshal it using our custom marshal method
val, err := json.Marshal(mySArr)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Output: \n%v\n", string(val))
}
Link to working code: https://play.golang.org/p/T4twqPc34k0
Thanks to mkopriva

Is there a way to have json.Unmarshal() select struct type based on "type" property?

I have some JSON of the form:
[{
"type": "car",
"color": "red",
"hp": 85,
"doors": 4
}, {
"type": "plane",
"color": "blue",
"engines": 3
}]
I have types car and plane that satisfy a vehicle interface; I'd like to be able to write:
var v []vehicle
e := json.Unmarshal(myJSON, &v)
... and have JSON fill my slice of vehicles with a car and a plane; instead (and unsurprisingly) I just get "cannot unmarshal object into Go value of type main.vehicle".
For reference, here are suitable definitions of the types involved:
type vehicle interface {
vehicle()
}
type car struct {
Type string
Color string
HP int
Doors int
}
func (car) vehicle() { return }
type plane struct {
Type string
Color string
Engines int
}
func (plane) vehicle() { return }
var _ vehicle = (*car)(nil)
var _ vehicle = (*plane)(nil)
(Note that I'm actually totally uninterested in the t field on car and plane - it could be omitted because this information will, if someone successfully answers this question, be implicit in the dynamic type of the objects in v.)
Is there a way to have the JSON umarhsaller choose which type to use based on some part of the contents (in this case, the type field) of the data being decoded?
(Note that this is not a duplicate of Unmarshal JSON with unknown fields because I want each item in the slice to have a different dynamic type, and from the value of the 'type' property I know exactly what fields to expect—I just don't know how to tell json.Unmarshal how to map 'type' property values onto Go types.)
Taking the answers from the similar question: Unmarshal JSON with unknown fields, we can construct a few ways to unamrshal this JSON object in a []vehicle data structure.
The "Unmarshal with Manual Handling" version can be done by using a generic []map[string]interface{} data structure, then building the correct vehicles from the slice of maps. For brevity, this example does leave out the error checking for missing or incorrectly typed fields which the json package would have done.
https://play.golang.org/p/fAY9JwVp-4
func NewVehicle(m map[string]interface{}) vehicle {
switch m["type"].(string) {
case "car":
return NewCar(m)
case "plane":
return NewPlane(m)
}
return nil
}
func NewCar(m map[string]interface{}) *car {
return &car{
Type: m["type"].(string),
Color: m["color"].(string),
HP: int(m["hp"].(float64)),
Doors: int(m["doors"].(float64)),
}
}
func NewPlane(m map[string]interface{}) *plane {
return &plane{
Type: m["type"].(string),
Color: m["color"].(string),
Engines: int(m["engines"].(float64)),
}
}
func main() {
var vehicles []vehicle
objs := []map[string]interface{}{}
err := json.Unmarshal(js, &objs)
if err != nil {
log.Fatal(err)
}
for _, obj := range objs {
vehicles = append(vehicles, NewVehicle(obj))
}
fmt.Printf("%#v\n", vehicles)
}
We could leverage the json package again to take care of the unmarshaling and type checking of the individual structs by unmarshaling a second time directly into the correct type. This could all be wrapped up into a json.Unmarshaler implementation by defining an UnmarshalJSON method on the []vehicle type to first split up the JSON objects into raw messages.
https://play.golang.org/p/zQyL0JeB3b
type Vehicles []vehicle
func (v *Vehicles) UnmarshalJSON(data []byte) error {
// this just splits up the JSON array into the raw JSON for each object
var raw []json.RawMessage
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
for _, r := range raw {
// unamrshal into a map to check the "type" field
var obj map[string]interface{}
err := json.Unmarshal(r, &obj)
if err != nil {
return err
}
vehicleType := ""
if t, ok := obj["type"].(string); ok {
vehicleType = t
}
// unmarshal again into the correct type
var actual vehicle
switch vehicleType {
case "car":
actual = &car{}
case "plane":
actual = &plane{}
}
err = json.Unmarshal(r, actual)
if err != nil {
return err
}
*v = append(*v, actual)
}
return nil
}
JSON decoding and encoding in Go is actually surprisingly well at recognizing fields inside embedded structs. E.g. decoding or encoding the following structure works when there is no overlapping fields between type A and type B:
type T struct{
Type string `json:"type"`
*A
*B
}
type A struct{
Baz int `json:"baz"`
}
type B struct{
Bar int `json:"bar"`
}
Be aware that if both "baz" and "bar" are set in the JSON for the example above, both the T.A and T.B properties will be set.
If there is overlapping fields between A and B, or just to be able to better discard invalid combinations of fields and type, you need to implement the json.Unmarshaler interface. To not have to first decode fields into a map, you can extend the trick of using embedded structs.
type TypeSwitch struct {
Type string `json:"type"`
}
type T struct {
TypeSwitch
*A
*B
}
func (t *T) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &t.TypeSwitch); err != nil {
return err
}
switch t.Type {
case "a":
t.A = &A{}
return json.Unmarshal(data, t.A)
case "b":
t.B = &B{}
return json.Unmarshal(data, t.B)
default:
return fmt.Errorf("unrecognized type value %q", t.Type)
}
}
type A struct {
Foo string `json:"bar"`
Baz int `json:"baz"`
}
type B struct {
Foo string `json:"foo"`
Bar int `json:"bar"`
}
For marshaling back, json.Marshaler must also be implemented if there is overlapping fields.
Full example: https://play.golang.org/p/UHAdxlVdFQQ
The two passes approach works fine, but there is also the option of the mapstructure package, that was created to do exactly this.
I was facing the same problem.
I'm using the lib github.com/mitchellh/mapstructure together the encoding/json.
I first, unmarshal the json to a map, and use mapstructure to convert the map to my struct, e.g.:
type (
Foo struct {
Foo string `json:"foo"`
}
Bar struct {
Bar string `json:"bar"`
}
)
func Load(jsonStr string, makeInstance func(typ string) any) (any, error) {
// json to map
m := make(map[string]any)
e := json.Unmarshal([]byte(jsonStr), &m)
if e != nil {
return nil, e
}
data := makeInstance(m["type"].(string))
// decoder to copy map values to my struct using json tags
cfg := &mapstructure.DecoderConfig{
Metadata: nil,
Result: &data,
TagName: "json",
Squash: true,
}
decoder, e := mapstructure.NewDecoder(cfg)
if e != nil {
return nil, e
}
// copy map to struct
e = decoder.Decode(m)
return data, e
}
Using:
f, _ := Load(`{"type": "Foo", "foo": "bar"}`, func(typ string) any {
switch typ {
case "Foo":
return &Foo{}
}
return nil
})
If the property is a string you can use .(string) for casting the property because the origin is an interface.
You can use it the next way:
v["type"].(string)

Adjusting the structure when unmarshalling json in go

Unmarshalling JSON (that I do not control) like this:
{
"states": {
"state": [
{ ...}
]
}
}
into a struct like:
type Device struct {
States struct{ State []State }
}
var dev Device
I get an ugly syntax to access a state:
dev.States.State[0]
I would like to be able to transform the object so I can do
dev.States[0]
Can this be done with tags (omitted in the above example because not needed), or with another method, or do I have to first unmarshal to a struct like the above then manually remap to a struct as I want it?
All you have to do is implement the Unmarshaler interface just adding the method UnmarshalJSON(data []byte) error and including the logic that you need after Unmarshal; and if you want to do the inverse operation(marshal) just implement the Marshaler interface.
type State struct {
Name string `json:"name"`
}
type Device struct {
States []State
}
func (dev *Device) UnmarshalJSON(data []byte) error {
// anonymous struct with the real backbone of the json
tmp := struct {
States struct {
State []State `json:"state"`
} `json:"states"`
}{}
if err := json.Unmarshal(data, &tmp); err != nil {
return err
}
dev.States = tmp.States.State
return nil
}
Full example: https://play.golang.org/p/gNpS13ED_i

How to tell the client they need to send an integer instead of a string, from a Go server?

Let's say I have the following Go struct on the server
type account struct {
Name string
Balance int
}
I want to call json.Decode on the incoming request to parse it into an account.
var ac account
err := json.NewDecoder(r.Body).Decode(&ac)
If the client sends the following request:
{
"name": "test#example.com",
"balance": "3"
}
Decode() will return the following error:
json: cannot unmarshal string into Go value of type int
Now it's possible to parse that back into "you sent a string for Balance, but you really should have sent an integer", but it's tricky, because you don't know the field name. It also gets a lot trickier if you have a lot of fields in the request - you don't know which one failed to parse.
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
You can use a custom type with custom unmarshaling algorythm for your "Balance" field.
Now there are two possibilities:
Handle both types:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Int int
type account struct {
Name string
Balance Int
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var s string
err = json.Unmarshal(b, &s)
if err == nil {
var n int
n, err = strconv.Atoi(s)
if err != nil {
return
}
*i = Int(n)
return
}
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
Handle only a numeric type, and fail anything else with a sensible error:
package main
import (
"encoding/json"
"fmt"
)
type Int int
type account struct {
Name string
Balance Int
}
type FormatError struct {
Want string
Got string
Offset int64
}
func (fe *FormatError) Error() string {
return fmt.Sprintf("Invalid data format at %d: want: %s, got: %s",
fe.Offset, fe.Want, fe.Got)
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
return
}
if ute, ok := err.(*json.UnmarshalTypeError); ok {
err = &FormatError{
Want: "number",
Got: ute.Value,
Offset: ute.Offset,
}
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %#v\n", err)
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
There is a third possibility: write custom unmarshaler for the whole account type, but it requires more involved code because you'd need to actually iterate over the input JSON data using the methods of the
encoding/json.Decoder type.
After reading your
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
more carefully, I admit having a custom parser for the whole type is the only sensible possibility unless you are OK with a 3rd-party package implementing a parser supporting validation via JSON schema (I think I'd look at this first as juju is a quite established product).
A solution for this could be to use a type assertion by using a map to unmarshal the JSON data into:
type account struct {
Name string
Balance int
}
var str = `{
"name": "test#example.com",
"balance": "3"
}`
func main() {
var testing = map[string]interface{}{}
err := json.Unmarshal([]byte(str), &testing)
if err != nil {
fmt.Println(err)
}
val, ok := testing["balance"]
if !ok {
fmt.Println("missing field balance")
return
}
nv, ok := val.(float64)
if !ok {
fmt.Println("balance should be a number")
return
}
fmt.Printf("%+v\n", nv)
}
See http://play.golang.org/p/iV7Qa1RrQZ
The type assertion here is done using float64 because it is the default number type supported by Go's JSON decoder.
It should be noted that this use of interface{} is probably not worth the trouble.
The UnmarshalTypeError (https://golang.org/pkg/encoding/json/#UnmarshalFieldError) contains an Offset field that could allow retrieving the contents of the JSON data that triggered the error.
You could for example return a message of the sort:
cannot unmarshal string into Go value of type int near `"balance": "3"`
It would seem that here provides an implementation to work around this issue in Go only.
type account struct {
Name string
Balance int `json:",string"`
}
In my estimation the more correct and sustainable approach is for you to create a client library in something like JavaScript and publish it into the NPM registry for others to use (private repository would work the same way). By providing this library you can tailor the API for the consumers in a meaningful way and prevent errors creeping into your main program.