Unmarshalling JSON (that I do not control) like this:
{
"states": {
"state": [
{ ...}
]
}
}
into a struct like:
type Device struct {
States struct{ State []State }
}
var dev Device
I get an ugly syntax to access a state:
dev.States.State[0]
I would like to be able to transform the object so I can do
dev.States[0]
Can this be done with tags (omitted in the above example because not needed), or with another method, or do I have to first unmarshal to a struct like the above then manually remap to a struct as I want it?
All you have to do is implement the Unmarshaler interface just adding the method UnmarshalJSON(data []byte) error and including the logic that you need after Unmarshal; and if you want to do the inverse operation(marshal) just implement the Marshaler interface.
type State struct {
Name string `json:"name"`
}
type Device struct {
States []State
}
func (dev *Device) UnmarshalJSON(data []byte) error {
// anonymous struct with the real backbone of the json
tmp := struct {
States struct {
State []State `json:"state"`
} `json:"states"`
}{}
if err := json.Unmarshal(data, &tmp); err != nil {
return err
}
dev.States = tmp.States.State
return nil
}
Full example: https://play.golang.org/p/gNpS13ED_i
Related
I want to unmarshal a JSON object where one field contains a JSON string into one coherent object. How do I do that in Go?
Example:
Input:
{
"foo":1,
"bar":"{\\"a\\":\\"Hello\\"}"
}
Go type:
type Child struct {
A string `json:"a"`
}
type Main struct {
Foo int `json:"foo"`
Bar Child `json:"bar"`
}
I guess I'd need to implement a custom UnmarshalJSON implementation on one of the types, but its twisting my head to figure out on which one and how.
I guess you want to treat this as if the JSON String were just part of the surrounding JSON object? If so, then yes, as you suggest, a custom UnmarshalJSON method on Child should accomplish this.
func (c *Child) UnmarshalJSON(p []byte) error {
var jsonString string
if err := json.Unmarshal(p, &jsonString); err != nil {
return err // Means the string was invalid
}
type C Child // A new type that doesn't have UnmarshalJSON method
return json.Unmarshal([]byte(jsonString), (*C)(c))
}
See it in the playground
if i were to create a custom UnmarshalJson for that data, I would create an auxiliary struct auxMain that has the same fields as the main struct but with Bar field as string. Then it unmarshals the JSON data into this auxiliary struct, extracting the Foo field and the Bar field as a string. After that, it unmarshals the Bar field as string into the Child struct, and assigns the extracted Foo field and the Child struct to the Main struct.
It's a round about way but seems to work in the playground.
func (m *Main) UnmarshalJSON(b []byte) error {
type auxMain struct {
Foo int `json:"foo"`
Bar string `json:"bar"`
}
var a auxMain
if err := json.Unmarshal(b, &a); err != nil {
return err
}
var child Child
if err := json.Unmarshal([]byte(a.Bar), &child); err != nil {
return err
}
m.Foo = a.Foo
m.Bar = child
return nil
}
try it out in the PlayGround and see: https://go.dev/play/p/wWIceUxu1tj
Don't know if this is what you are looking for.
I want to make a validation api in order to validate a set of json requests regarding specific set of rules. To do that I want to use just one endpoint and call functions that correspond to the specific json struct. I know that there is no method overloading in go so I am kind of stumped.
...
type requestBodyA struct {
SomeField string `json:"someField"`
SomeOtherField string `json:"someOtherField"`
}
type requestBodyB struct {
SomeDifferentField string `json:"someDifferentField"`
SomeOtherDifferentField string `json:"someOtherDifferentField"`
}
type ValidationService interface {
ValidateRequest(ctx context.Context, s string) (err error)
}
type basicValidationService struct{}
...
So in order to validate lots of different json requests, is it better to create structs for each and every json request? Or should I create these dynamically? How can I know what kind of request is sent if I only have one endpoint?
If you have a single endpoint/rpc that has to accept different JSON types, you'll need to tell it how to distinguish between them, somehow. One option is to have something like:
type request struct {
bodyA *requestBodyA
bodyB *requestBodyB
}
Then, populate these fields in a container JSON object appropriately. The json module will only populate bodyA if a bodyA key is present, otherwise leaving it a nil, and so on.
Here's a more complete example:
type RequestBodyFoo struct {
Name string
Balance float64
}
type RequestBodyBar struct {
Id int
Ref int
}
type Request struct {
Foo *RequestBodyFoo
Bar *RequestBodyBar
}
func (r *Request) Show() {
if r.Foo != nil {
fmt.Println("Request has Foo:", *r.Foo)
}
if r.Bar != nil {
fmt.Println("Request has Bar:", *r.Bar)
}
}
func main() {
bb := []byte(`
{
"Foo": {"Name": "joe", "balance": 4591.25}
}
`)
var req Request
if err := json.Unmarshal(bb, &req); err != nil {
panic(err)
}
req.Show()
var req2 Request
bb = []byte(`
{
"Bar": {"Id": 128992, "Ref": 801472}
}
`)
if err := json.Unmarshal(bb, &req2); err != nil {
panic(err)
}
req2.Show()
}
Another option is to do it more dynamically with maps, but it's likely that the method above will be sufficient.
I'm reading a file.json into memory. It's an array of objects, sample:
[
{"id":123123,"language":"ja-JP","location":"Osaka"}
,{"id":33332,"language":"ja-JP","location":"Tokyo"}
,{"id":31231313,"language":"ja-JP","location":"Kobe"}
]
I want to manipulate certain keys in this JSON file, so that they start with uppercase. Meaning
"language" becomes "Language" each time it's found. What I've done so far is to make a struct representing each object, as such:
type sampleStruct struct {
ID int `json:"id"`
Language string `json:"Language"`
Location string `json:"Location"`
}
Here, I define the capitalization. Meaning, id shouldn't be capitalized, but location and language should.
Rest of the code is as such:
func main() {
if len(os.Args) < 2 {
fmt.Println("Missing filename parameter.")
return
}
translationfile, err := ioutil.ReadFile(os.Args[1])
fileIsValid := isValidJSON(string(translationfile))
if !fileIsValid {
fmt.Println("Invalid JSON format for: ", os.Args[1])
return
}
if err != nil {
fmt.Println("Can't read file: ", os.Args[1])
panic(err)
}
}
func isValidJSON(str string) bool {
var js json.RawMessage
return json.Unmarshal([]byte(str), &js) == nil
}
// I'm unsure how to iterate through the JSON objects and only uppercase the objects matched in my struct here.
func upperCaseSpecificKeys()
// ...
Desired output, assuming the struct represents the whole data object, transform each key as desired:
[
{"id":123123,"Language":"ja-JP","Location":"Osaka"}
,{"id":33332,"Language":"ja-JP","Location":"Tokyo"}
,{"id":31231313,"Language":"ja-JP","Location":"Kobe"}
]
The documentation on json.Unmarshal says (with added emphasis):
To unmarshal JSON into a struct, Unmarshal matches incoming object
keys to the keys used by Marshal (either the struct field name or its
tag), preferring an exact match but also accepting a case-insensitive
match
See example here: https://play.golang.org/p/1vv8PaQUOfg
One way is to implement custom marshal method, although not very flexible:
type upStruct struct {
ID int `json:"id"`
Language string
Location string
}
type myStruct struct {
ID int `json:"id"`
Language string `json:"language"`
Location string `json:"location"`
}
func (m myStruct) MarshalJSON() ([]byte, error) {
return json.Marshal(upStruct(m))
}
....
func main() {
var mySArr []myStruct
// 1. Unmarshal the input
err := json.Unmarshal([]byte(myJson), &mySArr)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Input: \n%+v\n", mySArr)
// 2. Then, marshal it using our custom marshal method
val, err := json.Marshal(mySArr)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Output: \n%v\n", string(val))
}
Link to working code: https://play.golang.org/p/T4twqPc34k0
Thanks to mkopriva
I have some JSON of the form:
[{
"type": "car",
"color": "red",
"hp": 85,
"doors": 4
}, {
"type": "plane",
"color": "blue",
"engines": 3
}]
I have types car and plane that satisfy a vehicle interface; I'd like to be able to write:
var v []vehicle
e := json.Unmarshal(myJSON, &v)
... and have JSON fill my slice of vehicles with a car and a plane; instead (and unsurprisingly) I just get "cannot unmarshal object into Go value of type main.vehicle".
For reference, here are suitable definitions of the types involved:
type vehicle interface {
vehicle()
}
type car struct {
Type string
Color string
HP int
Doors int
}
func (car) vehicle() { return }
type plane struct {
Type string
Color string
Engines int
}
func (plane) vehicle() { return }
var _ vehicle = (*car)(nil)
var _ vehicle = (*plane)(nil)
(Note that I'm actually totally uninterested in the t field on car and plane - it could be omitted because this information will, if someone successfully answers this question, be implicit in the dynamic type of the objects in v.)
Is there a way to have the JSON umarhsaller choose which type to use based on some part of the contents (in this case, the type field) of the data being decoded?
(Note that this is not a duplicate of Unmarshal JSON with unknown fields because I want each item in the slice to have a different dynamic type, and from the value of the 'type' property I know exactly what fields to expect—I just don't know how to tell json.Unmarshal how to map 'type' property values onto Go types.)
Taking the answers from the similar question: Unmarshal JSON with unknown fields, we can construct a few ways to unamrshal this JSON object in a []vehicle data structure.
The "Unmarshal with Manual Handling" version can be done by using a generic []map[string]interface{} data structure, then building the correct vehicles from the slice of maps. For brevity, this example does leave out the error checking for missing or incorrectly typed fields which the json package would have done.
https://play.golang.org/p/fAY9JwVp-4
func NewVehicle(m map[string]interface{}) vehicle {
switch m["type"].(string) {
case "car":
return NewCar(m)
case "plane":
return NewPlane(m)
}
return nil
}
func NewCar(m map[string]interface{}) *car {
return &car{
Type: m["type"].(string),
Color: m["color"].(string),
HP: int(m["hp"].(float64)),
Doors: int(m["doors"].(float64)),
}
}
func NewPlane(m map[string]interface{}) *plane {
return &plane{
Type: m["type"].(string),
Color: m["color"].(string),
Engines: int(m["engines"].(float64)),
}
}
func main() {
var vehicles []vehicle
objs := []map[string]interface{}{}
err := json.Unmarshal(js, &objs)
if err != nil {
log.Fatal(err)
}
for _, obj := range objs {
vehicles = append(vehicles, NewVehicle(obj))
}
fmt.Printf("%#v\n", vehicles)
}
We could leverage the json package again to take care of the unmarshaling and type checking of the individual structs by unmarshaling a second time directly into the correct type. This could all be wrapped up into a json.Unmarshaler implementation by defining an UnmarshalJSON method on the []vehicle type to first split up the JSON objects into raw messages.
https://play.golang.org/p/zQyL0JeB3b
type Vehicles []vehicle
func (v *Vehicles) UnmarshalJSON(data []byte) error {
// this just splits up the JSON array into the raw JSON for each object
var raw []json.RawMessage
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
for _, r := range raw {
// unamrshal into a map to check the "type" field
var obj map[string]interface{}
err := json.Unmarshal(r, &obj)
if err != nil {
return err
}
vehicleType := ""
if t, ok := obj["type"].(string); ok {
vehicleType = t
}
// unmarshal again into the correct type
var actual vehicle
switch vehicleType {
case "car":
actual = &car{}
case "plane":
actual = &plane{}
}
err = json.Unmarshal(r, actual)
if err != nil {
return err
}
*v = append(*v, actual)
}
return nil
}
JSON decoding and encoding in Go is actually surprisingly well at recognizing fields inside embedded structs. E.g. decoding or encoding the following structure works when there is no overlapping fields between type A and type B:
type T struct{
Type string `json:"type"`
*A
*B
}
type A struct{
Baz int `json:"baz"`
}
type B struct{
Bar int `json:"bar"`
}
Be aware that if both "baz" and "bar" are set in the JSON for the example above, both the T.A and T.B properties will be set.
If there is overlapping fields between A and B, or just to be able to better discard invalid combinations of fields and type, you need to implement the json.Unmarshaler interface. To not have to first decode fields into a map, you can extend the trick of using embedded structs.
type TypeSwitch struct {
Type string `json:"type"`
}
type T struct {
TypeSwitch
*A
*B
}
func (t *T) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &t.TypeSwitch); err != nil {
return err
}
switch t.Type {
case "a":
t.A = &A{}
return json.Unmarshal(data, t.A)
case "b":
t.B = &B{}
return json.Unmarshal(data, t.B)
default:
return fmt.Errorf("unrecognized type value %q", t.Type)
}
}
type A struct {
Foo string `json:"bar"`
Baz int `json:"baz"`
}
type B struct {
Foo string `json:"foo"`
Bar int `json:"bar"`
}
For marshaling back, json.Marshaler must also be implemented if there is overlapping fields.
Full example: https://play.golang.org/p/UHAdxlVdFQQ
The two passes approach works fine, but there is also the option of the mapstructure package, that was created to do exactly this.
I was facing the same problem.
I'm using the lib github.com/mitchellh/mapstructure together the encoding/json.
I first, unmarshal the json to a map, and use mapstructure to convert the map to my struct, e.g.:
type (
Foo struct {
Foo string `json:"foo"`
}
Bar struct {
Bar string `json:"bar"`
}
)
func Load(jsonStr string, makeInstance func(typ string) any) (any, error) {
// json to map
m := make(map[string]any)
e := json.Unmarshal([]byte(jsonStr), &m)
if e != nil {
return nil, e
}
data := makeInstance(m["type"].(string))
// decoder to copy map values to my struct using json tags
cfg := &mapstructure.DecoderConfig{
Metadata: nil,
Result: &data,
TagName: "json",
Squash: true,
}
decoder, e := mapstructure.NewDecoder(cfg)
if e != nil {
return nil, e
}
// copy map to struct
e = decoder.Decode(m)
return data, e
}
Using:
f, _ := Load(`{"type": "Foo", "foo": "bar"}`, func(typ string) any {
switch typ {
case "Foo":
return &Foo{}
}
return nil
})
If the property is a string you can use .(string) for casting the property because the origin is an interface.
You can use it the next way:
v["type"].(string)
I am writing a Go library that will decode JSON into a struct. The JSON has a fairly simple common schema, but I want consumers of this library to be able to decode additional fields into their own structs that embed the common struct, avoiding the need to use maps. Ideally, I'd like to decode the JSON only once.
Currently it looks something like this. (Error handling removed for brevity.)
The JSON:
{ "CommonField": "foo",
"Url": "http://example.com",
"Name": "Wolf" }
The library code:
// The base JSON request.
type BaseRequest struct {
CommonField string
}
type AllocateFn func() interface{}
type HandlerFn func(interface{})
type Service struct {
allocator AllocateFn
handler HandlerFn
}
func (Service *s) someHandler(data []byte) {
v := s.allocator()
json.Unmarshal(data, &v)
s.handler(v)
}
The app code:
// The extended JSON request
type MyRequest struct {
BaseRequest
Url string
Name string
}
func allocator() interface{} {
return &MyRequest{}
}
func handler(v interface{}) {
fmt.Printf("%+v\n", v);
}
func main() {
s := &Service{allocator, handler}
// Run s, eventually s.someHandler() is called
}
The thing I don't like about this setup is the allocator function. All implementations are simply going to return a new BaseRequest "sub-type". In a more dynamic language I would pass the type of MyRequest in instead, and instantiate inside the library. Do I have a similar option in Go?
There are several ways to handle this. One idea which is both simple and convenient is defining a richer Request type that you provide to the handler, instead of handing off the raw type. This way you can implement the default behavior in a friendly way, and support the edge cases. This would also avoid the need to embed the default type on custom types, and allow you to expand functionality without breaking clients.
For inspiration:
type Request struct {
CommonField string
rawJSON []byte
}
func (r *Request) Unmarshal(value interface{}) error {
return json.Unmarshal(r.rawJSON, value)
}
func handler(req *Request) {
// Use common data.
fmt.Println(req.CommonField)
// If necessary, poke into the underlying message.
var myValue MyType
err := req.Unmarshal(&myValue)
// ...
}
func main() {
service := NewService(handler)
// ...
}
I think json.RawMessage is used to delay decoding subsets of JSON. In your case you can maybe do something like this:
package main
import (
↦ "encoding/json"
↦ "fmt"
)
type BaseRequest struct {
↦ CommonField string
↦ AppData json.RawMessage
}
type AppData struct {
↦ AppField string
}
var someJson string = `
{
↦ "CommonField": "foo",
↦ "AppData": {
↦ ↦ "AppField": "bar"
↦ }
}
`
func main() {
↦ var baseRequest BaseRequest
↦ if err := json.Unmarshal([]byte(someJson), &baseRequest); err != nil {
↦ ↦ panic(err)
↦ }
↦ fmt.Println("Parsed BaseRequest", baseRequest)
↦ var appData AppData
↦ if err := json.Unmarshal(baseRequest.AppData, &appData); err != nil {
↦ ↦ panic(err)
↦ }
↦ fmt.Println("Parsed AppData", appData)
}
Another way I came up with is to use reflection.
Tweaking my original example, the library code becomes:
// The base JSON request.
type BaseRequest struct {
CommonField string
}
type HandlerFn func(interface{})
type Service struct {
typ reflect.Type
handler HandlerFn
}
func (Service *s) someHandler(data []byte) {
v := reflect.New(s.typ).Interface()
json.Unmarshal(data, &v)
s.handler(v)
}
and the app code becomes:
// The extended JSON request
type MyRequest struct {
BaseRequest
Url string
Name string
}
func handler(v interface{}) {
fmt.Printf("%+v\n", v);
}
func main() {
s := &Service{reflect.TypeOf(MyRequest{}), handler}
// Run s, eventually s.someHandler() is called
}
I haven't decided if I like this any better. Maybe the way to go is just to simply unmarshal the data twice.