Related
My goal is to generate from a json a map[string]interface structure , where all nested []interface{} are casted into []map[string]interface{} . This is because we are using the module https://github.com/ivahaev/go-xlsx-templater to fill xlsx from json . And all the data is expected to be into a map[string]interface{} struct where all the nested []interface{} are []map[string]interface.
Having a json as an input like the following :
{
"totalAmount": 4,
"subtotal": 4,
"Vendors": [{
"MethodOfTenders": [{
"order": 1,
"fees": 2
}, {
"order": 1,
"fees": 2
}],
"subtotalFees": 4
},
{
"MethodOfTenders": [{
"order": 1,
"fees": 2
}, {
"order": 1,
"fees": 1
}],
"subtotalFees": 3
}
]
}
When unmarshalling into an map[string]interface{} .I get the following struct :
map[string]interface{}:
"totalAmount" : 4 interface{}(float64)
"subtotal" : 4 interface{}(float64)
"Vendors" : interface{}([]interface{})
[0]: interface(map[string]interface{})
"MethodOfTenders" : interface{}([]interface{})
[0] : interface(map[string]interface{})
"order" : 1 interface{}(float64)
"fees" : 2 interface{}(float64)
[1] : interface(map[string]interface{})
"order" : 1 interface{}(float64),
"fees" : 2 interface{}(float64)
"subtotalFees" : 4 interface{}(float64)
[1]: interface(map[string]interface{})
"MethodOfTenders" : interface{}([]interface{})
[0] : interface(map[string]interface{})
"order" : 1 interface{}(float64)
"fees" : 2 interface{}(float64)
[1] : interface(map[string]interface{})
"order" : 1 interface{}(float64),
"fees" : 2 interface{}(float64)
"subtotalFees" : 3 interface{}(float64)
After doing some parsing , ranging each []interface{} and creating a a []map[string]interface{} to store each one of the nested map[string]interface .
I got the desired result where all []interfaces{} are []map[string]interface{}
map[string]interface{}:
"totalAmount" : 4 interface{}(float64)
"subtotal" : 4 interface{}(float64)
"Vendors" : interface{}([]map[string]interface{})
[0]: interface(map[string]interface{})
"MethodOfTenders" : interface{}([]map[string]interface{})
[0] : interface(map[string]interface{})
"order" : 1 interface{}(float64)
"fees" : 2 interface{}(float64)
[1] : interface(map[string]interface{})
"order" : 1 interface{}(float64),
"fees" : 2 interface{}(float64)
"subtotalFees" : 4 interface{}(float64)
[1]: interface(map[string]interface{})
"MethodOfTenders" : interface{}([]map[string]interface{})
[0] : interface(map[string]interface{})
"order" : 1 interface{}(float64)
"fees" : 2 interface{}(float64)
[1] : interface(map[string]interface{})
"order" : 1 interface{}(float64),
"fees" : 2 interface{}(float64)
"subtotalFees" : 3 interface{}(float64)
Is there any way to walk al the map[string]interface recursively and change the []interfaces{} for []map[string]interface ?
EDIT:
Here the repository with the full template and json , currently working with the reflection approach .
template and json repo
Update
So I've since noticed you added a link to a repo containing what you have so far. The template is weird (some of the values in there aren't part of your sample data). I've removed all fields except for the ones that were actually in the sample data JSON you have. As it turns out, the package you are using is quite rough around the edges (it does take care of type assertions in nested values, but not at the top level). I'd personally either fork the package and fix that issue, but in the mean while, if you know what fields you want to iterate over, I got it all to work in about 5 minutes with this:
var jsonMap map[string]interface{}
data := sampleData()
err := json.Unmarshal(data, &jsonMap)
if err != nil {
panic("Final log")
}
ctx := jsonMap
v := jsonMap["Vendors"]
vs := v.([]interface{})
vendors := make([]map[string]interface{}, 0, len(vs))
for _, v := range vs {
m := v.(map[string]interface{})
vendors = append(vendors, m)
}
jsonMap["Vendors"] = vendors
doc := xlst.New()
err = doc.ReadTemplate("export_support_template.xlsx")
if err != nil {
fmt.Println("ERROR OPENING THE TEMPLATE: ", err)
panic("error opening template")
}
err = doc.Render(ctx)
if err != nil {
fmt.Println("ERROR RENDERING THE TEMPLATE: ", err)
panic("error rendering template")
}
err = doc.Save("report.xlsx")
As you can see, I just unmarshalled the data. I then focused on the Vendors key, cast it to a slice (v.([]inteface{}), created a new variable of type []map[string]interface{}, and copied over the data casting the elements of the slice to maps in this simple loop:
for _, v := range vs {
m := v.(map[string]interface{})
vendors = append(vendors, m)
}
Then, I just reassigned the Vendors key in the original map (ctx["Vendors"] = vendors), and passed that in to the template. Less than 10 lines of code needed, and everything worked like a charm. No need for reflection, or any other magic. Just straightforward type casts. Without this, the package you're using does complain about the use of range on an interface{} type (instead of first checking to see if the key can be successfully cast to a []interface{} first). I removed all other functions from your main file (except for the sample data one), ran go build and executed ./json_marshaller. It churned out a correctly populated xlsx file no problem. Easy.
I'd still recommend you actually create a PR to the templater package so you don't have to handle this stuff yourself. It should be a fairly straightforward change, but for the time being: this approach works perfectly well.
Include your template
Looking at the repository of the go-xlsx-templater, I don't really see why you'd need anything other than this:
data := map[string]interface{}
_ = json.Unmarshal(input, &data)
The reason why this is perfectly fine, is because go-xlsx-templater performs the type conversion of all interface{} values within this map already. It will traverse the map, and check if any of the values are of the type []interface{}, it'll then iterate over this slice and check for map[string]interface{} values inside the slice. It's all in the source code. Going by their own documentation, it should work just fine if your template looks somewhat like this:
| Total: | {{ totalAmount }} |
| Subtotal: | {{ subtotal }} |
| | Vendor subtotal | fees | order |
| {{ range Vendors }} |
| | | {{ MethodsOfTender.Fees }} | {{ MethodsOfTender.order }} |
| | {{ subtotalFees }} |
| {{ end }} |
You should end up with the spreadsheet being populated correctly:
| Total: | 4 | | |
| Subtotal: | 4 | | |
| | Vendor subtotal | fees | order |
| | | 2 | 1 |
| | | 2 | 1 |
| | 4 | | |
| | | 2 | 1 |
| | | 1 | 1 |
| | 3 | | |
Essentially, the data-set you're showing is no different to the example the repo gives in their very own main README file. The context data itself contains nested maps, and a screenshot of a spreadsheet template showing how to use it. Putting it bluntly, I feel like your question is pretty much a case of RTFM...
Without your template, however, there's no way to be sure, so I'm leaving my initial answer in full below...
If I understand what you're asking correctly, you're looking for a way to use type assertions/casts to extract keys from a map of type map[string]interface{} and use the values of type []interface{} as []map[string]interface{}. In this particular example, you need the top level key Vendors to be accessible as []map[string]interface{} (as opposed to interface{} or []interface{}). If that's what you're looking for, you can do this quite easily:
data := map[string]interface{}{}
if err := json.Unmarshal(jsonBytes, &data); err != nil {
// handle error
}
nested := map[string][]map[string]interface{}{} // subset of data that needs to be accessible as slices of maps
for k, v := range data {
if s, ok := v.([]interface); ok {
// this key is of type []interface, see if we can use it as slice of maps
if mapS, err := sliceAnyToMap(s); err == nil {
nested[k] = mapS
} // handle error if needed
}
}
func sliceAnyToMap(s []any) ([]map[string]interface{}, error) {
ret := make([]map[string]interface{}, 0, len(s))
for _, v := range s {
if m, ok := v.(map[string]interface{}); ok {
ret = append(ret, m)
} else {
return nil, errors.New("slice contains non-map data")
}
}
return ret, nil
}
With this, you'll end up with a variable called nested which will contain keys like Vendors, with the data accessible as a []map[string]interface{}. This can be passed on to your templater to populate the spreadsheet with the vendors data.
Demo here
Now, inside some of the Vendors key, there is some data which is in turn a a map, which still can't be accessed directly. Breaking up this sliceAnyToMap stuff into a function will make it easier to convert all data you need fairly easily.
Now, having said all this, I do personally think this is very much an X-Y problem. The data you've shown here looks to be of a very well defined structure. Rather than faffing around with map[string]interface{} and the like, it would be a whole lot easier to read, write, and maintain if you were to use actual types. Based on what you've included in your question, a couple of types like this would be all it takes:
type Data struct {
Total float64 `json:"totalAmount"`
Subtotal float64 `json:"subtotal"`
Vendors []Vendor `json:"Vendors"`
}
type Vendor struct {
MethodOfTenders []MOT `json:"MethodOfTenders"`
SubtotalFees float64 `json:"subtotalFees"`
}
type MOT struct {
Fees float64 `json:"fees"`
Order float64 `json:"order"`
}
With these types, you can quickly, and easily parse the given input into a format that is much, much easier to use further down the line:
data := Data{}
if err := json.Unmarshal(input, &data); err != nil {
// handle error
}
for _, vendor := range data.Vendors {
fmt.Printf("Vendor subtotal fees: %f\n", vendor.SubtotalFees)
for i, mot := range vendor.MethodOfTenders {
fmt.Printf("MOT %d:\nOrder: %f\nFees: %f\n\n", i+1, mot.Order, mot.Fees)
}
fmt.Println("-----")
}
Demo here
Last couple of things I'm wondering is whether or not all numberic values should be float64. Sure, it makes sense for amounts and/or fees (assuming they're monetary values), but the Order field probably contains an order number, which I don't expect to be a value like 0.123.
Finally, if you need to pass this data on to a templater of sorts which expects a map, then it's a fairly trivial thing to do using the aforementioned types all the same. There's a couple of ways to do this. The easy way being:
func (d Data) ToMap() (map[string]interface{}, error) {
raw, err := json.Marshal(d)
if err != nil {
return nil, err
}
asMap := map[string]interface{}{}
if err := json.Unmarshal(raw, &asMap); err != nil {
return nil, err
}
return asMap, nil
}
JSON marshalling is quick to implement, but it's a tad inefficient, so you if performance matters a lot, then you could spend a few minutes writing some methods like:
func (d Data) ToMap() map[string]interface{} {
vendors := make([]map[string]interface{}, 0, len(d.Vendors))
for _, v := range d.Vendors {
vendors = append(vendors, v.ToMap())
}
return map[string]interface{}{
"totalAmount": d.Total,
"subtotal": d.Subtotal,
"Vendors": vendors,
}
}
func (v Vendor) ToMap() map[string]interface{} {
mot := make([]map[string]interface{}, 0, len(v.MethodsOfTender))
for _, m := range v.MethodsOfTender {
mot = append(mot, m.ToMap())
}
return map[string]interface{
"MethodsOfTender": mot,
"subtotalFees": v.SubtotalFees,
}
}
func (m MOT) ToMap() map[string]interface{} {
return map[string]interface{}{
"fees": m.Fees,
"order": m.Order,
}
}
Then, turning your Data object into a map is as simple as:
dataMap := data.ToMap()
package main
import (
"encoding/json"
"fmt"
)
func main() {
s := `{
"totalAmount": 4,
"subtotal": 4,
"Vendors": [{
"MethodOfTenders": [
{
"order": 1,
"fees": 2
}, {
"order": 1,
"fees": 2
}
],
"subtotalFees": 4
},
{
"MethodOfTenders": [
{
"order": 1,
"fees": 2
}, {
"order": 1,
"fees": 1
}
],
"subtotalFees": 3
}
]
}
`
var data map[string]interface{}
if err := json.Unmarshal([]byte(s), &data); err != nil {
fmt.Println(err)
}
fmt.Printf("%+v\n", data)
}
Playground
When Unmarshal from json it is generating expected output which is required by go-xlsx-templater
Note: Created a fix for the issue stated in the question here raised PR. After the fix no conversion is required.
After applying the fix the remaining code with sampleData() function which is not included in the example
package main
import (
"encoding/json"
"fmt"
"log"
"os"
"reflect"
"strings"
xlst "github.com/ivahaev/go-xlsx-templater"
)
func main() {
wd, err := os.Getwd()
if err != nil {
log.Fatalf(err.Error())
}
path := strings.Join([]string{wd, ""}, "")
doc := xlst.New()
err = doc.ReadTemplate(path + "/export_support_template.xlsx")
if err != nil {
fmt.Println("ERROR OPENING THE TEMPLATE: ", err)
panic("error opening template")
}
var ctx map[string]interface{}
data := sampleData()
err := json.Unmarshal(data, &ctx)
if err != nil {
fmt.Println("Final log")
}
err = doc.Render(ctx)
if err != nil {
fmt.Println("ERROR RENDERING THE TEMPLATE: ", err)
panic("error rendering template")
}
err = doc.Save(path + "/report.xlsx")
if err != nil {
fmt.Println("ERROR SAVING THE TEMPLATE: ", err)
panic("error saving template")
}
}
The trick is performed using reflection. Go down recursively and check the kind of values:
// "Casts" map values to the desired type recursively
func castMap(m map[string]any) map[string]any {
for k := range m {
switch reflect.ValueOf(m[k]).Kind() {
case reflect.Map:
mm, ok := m[k].(map[string]any)
if !ok {
panic(fmt.Errorf("Expected map[string]any, got %T", m[k]))
}
m[k] = castMap(mm)
case reflect.Slice, reflect.Array:
ma, ok := m[k].([]any)
if !ok {
panic(fmt.Errorf("Expected []any, got %T", m[k]))
}
m[k] = castArray(ma)
default:
// fmt.Printf("%s: %T, kind %v\n", k, m[k], reflect.ValueOf(m[k]).Kind())
continue
}
}
return m
}
// "Casts" slice elements to the desired types recursively
func castArray(a []any) []map[string]any {
res := []map[string]any{}
for i := range a {
switch reflect.ValueOf(a[i]).Kind() {
case reflect.Map:
am, ok := a[i].(map[string]any)
if !ok {
panic(fmt.Errorf("Expected map[string]any, got %T", a[i]))
}
am = castMap(am)
res = append(res, am)
default:
panic(fmt.Errorf("Expected map[string]any, got %T", a[i]))
}
}
return res
}
Full example with main is here: https://go.dev/play/p/MEQRe-f3dY1
It's output:
before: map[string]interface {}{"Vendors":[]interface {}{map[string]interface {}{"MethodOfTenders":[]interface {}{map[string]interface {}{"fees":2, "order":1}, map[string]interface {}{"fees":2, "order":1}}, "subtotalFees":4}, map[string]interface {}{"MethodOfTenders":[]interface {}{map[string]interface {}{"fees":2, "order":1}, map[string]interface {}{"fees":1, "order":1}}, "subtotalFees":3}}, "subtotal":4, "totalAmount":4}
after: map[string]interface {}{"Vendors":[]map[string]interface {}{map[string]interface {}{"MethodOfTenders":[]map[string]interface {}{map[string]interface {}{"fees":2, "order":1}, map[string]interface {}{"fees":2, "order":1}}, "subtotalFees":4}, map[string]interface {}{"MethodOfTenders":[]map[string]interface {}{map[string]interface {}{"fees":2, "order":1}, map[string]interface {}{"fees":1, "order":1}}, "subtotalFees":3}}, "subtotal":4, "totalAmount":4}
See? "Vendors" was []interface {}, became []map[string]interface {}
"Vendors[].MethodOfTenders" were []interface {}, became []map[string]interface {}
The functions panic if they see something unexpected. Feel free to modify them to returning error if needed.
UPDATE
Here is the same recursive algoritm using type assertions.
// "Casts" map values to the desired type recursively
func castMap(m map[string]any) map[string]any {
for k := range m {
mm, ok := m[k].(map[string]any)
if ok {
m[k] = castMap(mm)
continue
}
ma, ok := m[k].([]any)
if ok {
m[k] = castArray(ma)
continue
}
}
return m
}
// "Casts" slice elements to the desired types recursively
func castArray(a []any) []map[string]any {
res := []map[string]any{}
for i := range a {
am, ok := a[i].(map[string]any)
if ok {
am = castMap(am)
res = append(res, am)
} else {
panic(fmt.Errorf("Expected map[string]any, got %T", a[i]))
}
}
return res
}
Full code https://go.dev/play/p/IbOhQqpisie
BENCHMARK
One of the commenters made a claim that reflection is expensive. This is not the case here:
goos: windows
goarch: amd64
pkg: example.org/try/test
cpu: Intel(R) Core(TM) i7-8550U CPU # 1.80GHz
BenchmarkCastReflect-8 382574 2926 ns/op 2664 B/op 29 allocs/op
BenchmarkCastType-8 418257 2934 ns/op 2664 B/op 29 allocs/op
The benchmark is here: https://go.dev/play/p/Ro8PeVQy8kA
Playground doesn't execute benchmarks. It should be run on a real CPU
I have a scenario where the JSON that has dynamic set of fields that need to get unmarshalled in to a struct.
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
type Status struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status map[string]Status `json:"status"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
var person Person
if err := dec.Decode(&person); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(person)
fmt.Println(person.Status["bvu62fu6dq"])
}
}
The output:
{john 23 map[]}
{false}
When it gets unmarshalled, the nested status struct is not being correctly resolved to the value in the JSON (shows false even with true value in JSON), is there any issue in the code?
Your types don't really match with the JSON you have:
type Status struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status map[string]Status `json:"status"`
}
Maps to JSON that looks something like this:
{
"name": "foo",
"age": 12,
"status": {
"some-string": {
"Status": true
}
}
}
The easiest way to unmarshal data with a mix of known/unknown fields in a go type is to have something like this:
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"` // skip this key
}
Then, first unmarshal the known data:
var p Person
if err := json.Unmarshal([]byte(jsonStream), &p); err != nil {
panic(err)
}
// then unmarshal the rest of the data
if err := json.Unmarshal([]byte(jsonStream), &p.Random); err != nil {
panic(err)
}
Now the Random map will contain every and all data, including the name and age fields. Seeing as you've got those tagged on the struct, these keys are known, so you can easily delete them from the map:
delete(p.Random, "name")
delete(p.Random, "age")
Now p.Random will contain all the unknown keys and their respective values. These values apparently will be an object with a field status, which is expected to be a boolean. You can set about using type assertions and convert them all over to a more sensible type, or you can take a shortcut and marshal/unmarshal the values. Update your Person type like so:
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"`
Statuses map[string]Status `json:"-"`
}
Now take the clean Random value, marshal it and unmarshal it back into the Statuses field:
b, err := json.Marshal(p.Random)
if err != nil {
panic(err)
}
if err := json.Unmarshal(b, &p.Statuses); err != nil {
panic(err)
}
// remove Random map
p.Random = nil
The result is Person.Statuses["bvu62fu6dq"].Status is set to true
Demo
Cleaning this all up, and marshalling the data back
Now because our Random and Statuses fields are tagged to be ignored for JSON marshalling (json:"-"), marshalling this Person type won't play nice when you want to output the original JSON from these types. It's best to wrap this logic up in a custom JSON (un)-Marshaller interface. You can either use some intermediary types in your MarshalJSON and UnmarshalJSON methods on the Person type, or just create a map and set the keys you need:
func (p Person) MarshalJSON() ([]byte, error) {
data := make(map[string]interface{}, len(p.Statuses) + 2) // 2 being the extra fields
// copy status fields
for k, v := range p.Statuses {
data[k] = v
}
// add known keys
data["name"] = p.Name
data["age"] = p.Age
return json.Marshal(data) // return the marshalled map
}
Similarly, you can do the same thing for UnmarshalJSON, but you'll need to create a version of the Person type that doesn't have the custom handling:
type intermediaryPerson struct {
Name string `json:"name"`
Age int `json:"age"`
Random map[string]interface{} `json:"-"`
}
// no need for the tags and helper fields anymore
type Person struct {
Name string
Age int
Statuses map[string]Status // Status type doesn't change
}
func (p *Person) UnmarshalJSON(data []byte) error {
i := intermediaryPerson{}
if err := json.Unmarshal(data, &i); err != nil {
return err
}
if err := json.Unmarshal(data, &i.Random); err != nil {
return err
}
delete(i.Random, "name")
delete(i.Random, "age")
stat, err := json.Marshal(i.Random)
if err != nil {
return err
}
// copy known fields
p.Name = i.Name
p.Age = i.Age
return json.Unmarshal(stat, &p.Statuses) // set status fields
}
In cases like this, it's common to create a type that handles the known fields and embed that, though:
type BasePerson struct {
Name string `json:"name"`
Age int `json:"age"`
}
and embed that in both the intermediary and the "main"/exported type:
type interPerson struct {
BasePerson
Random map[string]interface{} `json:"-"`
}
type Person struct {
BasePerson
Statuses map[string]Status
}
That way, you can just unmarshal the known fields directly into the BasePerson type, assign it, and then handle the map:
func (p *Person) UnmarshalJSON(data []byte) error {
base := BasePerson{}
if err := json.Unmarshal(data, &base); err != nil {
return err
}
p.BasePerson = base // takes care of all known fields
unknown := map[string]interface{}{}
if err := json.Unmarshal(data, unknown); err != nil {
return err
}
// handle status stuff same as before
delete(unknown, "name") // remove known fields
// marshal unknown key map, then unmarshal into p.Statuses
}
Demo 2
This is how I'd go about it. It allows for calls to json.Marshal and json.Unmarshal to look just like any other type, it centralises the handling of unknown fields in a single place (the implementation of the marshaller/unmarshaller interface), and leaves you with a single Person type where every field contains the required data, in a usable format. It's a tad inefficient in that it relies on unmarshalling/marshalling/unmarshalling the unknown keys. You could do away with that, like I said, using type assertions and iterating over the unknown map instead, faffing around with something like this:
for k, v := range unknown {
m, ok := v.(map[string]interface{})
if !ok {
continue // not {"status": bool}
}
s, ok := m["status"]
if !ok {
continue // status key did not exist, ignore
}
if sb, ok := s.(bool); ok {
// ok, we have a status bool value
p.Statuses[k] = Status{
Status: sb,
}
}
}
But truth be told, the performance difference won't be that great (it's micro optimisation IMO), and the code is a tad too verbose to my liking. Be lazy, optimise when needed, not whenever
Type doesn't meet with your json value.
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
For above json your code should look like below snnipet to work (some modifications in your existing code).
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"strings"
)
const jsonStream = `{
"name": "john",
"age": 23,
"bvu62fu6dq": {
"status": true
}
}`
type bvu62fu6dq struct {
Status bool
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
Status bvu62fu6dq `json:"bvu62fu6dq"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
var person Person
if err := dec.Decode(&person); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(person)
fmt.Println(person.Status)
}
}
Based on your json data you have to map with type fields.
Run code snippet
Some context, I'm designing a backend that will receive JSON post data, but the nature of the data is that it has fields that are unstructured. My general research tells me this is a static language vs unstructured data problem.
Normally if you can create a struct for it if the data is well known and just unmarshal into the struct. I have create custom unmarshaling functions for nested objects.
The issue now is that one of the fields could contain an object with an arbitrary number of keys. To provide some code context:
properties: {
"k1": "v1",
"k2": "v2",
"k3": "v3",
...
}
type Device struct {
id: string,
name: string,
status: int,
properties: <what would i put here?>
}
So its hard to code an explicit unmarshaling function for it. Should put a type of map[string]string{}? How would it work if the values were not all strings then? And what if that object itself had nested values/objects as well?
You can make the Properties field as map[string]interface{} so that it can accommodate different types of values.I created a small code for your scenario as follows:
package main
import (
"encoding/json"
"fmt"
)
type Device struct {
Id string
Name string
Status int
Properties map[string]interface{}
}
func main() {
devObj := Device{}
data := []byte(`{"Id":"101","Name":"Harold","Status":1,"properties":{"key1":"val1"}}`)
if err := json.Unmarshal(data, &devObj); err != nil {
panic(err)
}
fmt.Println(devObj)
devObj2 := Device{}
data2 := []byte(`{"Id":"102","Name":"Thanor","Status":1,"properties":{"k1":25,"k2":"someData"}}`)
if err := json.Unmarshal(data2, &devObj2); err != nil {
panic(err)
}
fmt.Println(devObj2)
devObj3 := Device{}
data3 := []byte(`{"Id":"101","Name":"GreyBeard","Status":1,"properties":{"k1":25,"k2":["data1","data2"]}}`)
if err := json.Unmarshal(data3, &devObj3); err != nil {
panic(err)
}
fmt.Println(devObj3)
}
Output:
{101 Harold 1 map[key1:val1]}
{102 Thanor 1 map[k1:25 k2:someData]}
{101 GreyBeard 1 map[k1:25 k2:[data1 data2]]}
I would use one of the popular Go JSON parsers that don't require parsing to a pre-defined struct. An added benefit, and the primary reason they were created, is that they are much faster than encoding/json because they don't use reflection, interface{} or certain other approaches.
Here are two:
https://github.com/buger/jsonparser - 4.1k GitHub stars
https://github.com/valyala/fastjson - 1.3k GitHub stars
Using github.com/buger/jsonparser, a property could be retrieved using the GetString function:
func GetString(data []byte, keys ...string) (val string, err error)
Here's a full example:
package main
import (
"fmt"
"strconv"
"github.com/buger/jsonparser"
)
func main() {
jsondata := []byte(`
{
"properties": {
"k1": "v1",
"k2": "v2",
"k3": "v3"
}
}`)
for i := 1; i > 0; i++ {
key := "k" + strconv.Itoa(i)
val, err := jsonparser.GetString(jsondata, "properties", key)
if err == jsonparser.KeyPathNotFoundError {
break
} else if err != nil {
panic(err)
}
fmt.Printf("found: key [%s] val [%s]\n", key, val)
}
}
See it run on Go Playground: https://play.golang.org/p/ykAM4gac8zT
Is there an easy way to check if each field of myStruct was mapped by using json.Unmarshal(jsonData, &myStruct).
The only way I could image is to define each field of a struct as pointer, otherwise you will always get back an initialized struct.
So every jsonString that is an object (even an empty one {}) will return an initialized struct and you cannot tell if the json represented your struct.
The only solution I could think of is quite uncomfortable:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
Name *string `json:name`
Age *int `json:age`
Male *bool `json:male`
}
func main() {
var p *Person
err := json.Unmarshal([]byte("{}"), &p)
// handle parse error
if err != nil {
return
}
// handle json did not match error
if p.Name == nil || p.Age == nil || p.Male == nil {
return
}
// now use the fields with dereferencing and hope you did not forget a nil check
fmt.Println("Hello " + *p.Name)
}
Maybe one could use a library like govalidator and use SetFieldsRequiredByDefault. But then you still have to execute the validation and still you are left with the whole pointer dereferencing for value retrieval and the risk of nil pointer.
What I would like is a function that returns my unmarshaled json as a struct or an error if the fields did not match. The only thing the golang json library offers is an option to fail on unknown fields but not to fail on missing fields.
Any idea?
Another way would be to implement your own json.Unmarshaler which uses reflection (similar to the default json unmarshaler):
There are a few points to consider:
if speed is of great importance to you then you should write a benchmark to see how big the impact of the extra reflection is. I suspect its negligible but it can't hurt to write a small go benchmark to get some numbers.
the stdlib will unmarshal all numbers in your json input into floats. So if you use reflection to set integer fields then you need to provide the corresponding conversion yourself (see TODO in example below)
the json.Decoder.DisallowUnknownFields function will not work as expected with your type. You need to implement this yourself (see example below)
if you decide to take this approach you will make your code more complex and thus harder to understand and maintain. Are you actually sure you must know if fields are omitted? Maybe you can refactor your fields to make good usage of the zero values?
Here a fully executable test of this approach:
package sandbox
import (
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
)
type Person struct {
Name string
City string
}
func (p *Person) UnmarshalJSON(data []byte) error {
var m map[string]interface{}
err := json.Unmarshal(data, &m)
if err != nil {
return err
}
v := reflect.ValueOf(p).Elem()
t := v.Type()
var missing []string
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
val, ok := m[field.Name]
delete(m, field.Name)
if !ok {
missing = append(missing, field.Name)
continue
}
switch field.Type.Kind() {
// TODO: if the field is an integer you need to transform the val from float
default:
v.Field(i).Set(reflect.ValueOf(val))
}
}
if len(missing) > 0 {
return errors.New("missing fields: " + strings.Join(missing, ", "))
}
if len(m) > 0 {
extra := make([]string, 0, len(m))
for field := range m {
extra = append(extra, field)
}
// TODO: consider sorting the output to get deterministic errors:
// sort.Strings(extra)
return errors.New("unknown fields: " + strings.Join(extra, ", "))
}
return nil
}
func TestJSONDecoder(t *testing.T) {
cases := map[string]struct {
in string
err string
expected Person
}{
"Empty object": {
in: `{}`,
err: "missing fields: Name, City",
expected: Person{},
},
"Name missing": {
in: `{"City": "Berlin"}`,
err: "missing fields: Name",
expected: Person{City: "Berlin"},
},
"Age missing": {
in: `{"Name": "Friedrich"}`,
err: "missing fields: City",
expected: Person{Name: "Friedrich"},
},
"Unknown field": {
in: `{"Name": "Friedrich", "City": "Berlin", "Test": true}`,
err: "unknown fields: Test",
expected: Person{Name: "Friedrich", City: "Berlin"},
},
"OK": {
in: `{"Name": "Friedrich", "City": "Berlin"}`,
expected: Person{Name: "Friedrich", City: "Berlin"},
},
}
for name, c := range cases {
t.Run(name, func(t *testing.T) {
var actual Person
r := strings.NewReader(c.in)
err := json.NewDecoder(r).Decode(&actual)
switch {
case err != nil && c.err == "":
t.Errorf("Expected no error but go %v", err)
case err == nil && c.err != "":
t.Errorf("Did not return expected error %v", c.err)
case err != nil && err.Error() != c.err:
t.Errorf("Expected error %q but got %v", c.err, err)
}
if !reflect.DeepEqual(c.expected, actual) {
t.Errorf("\nWant: %+v\nGot: %+v", c.expected, actual)
}
})
}
}
You could compare p with a empty struct, instead of comparing each field with nil.
// handle json did not match error
if p == Person{} {
return
}
Since Person{} will initialize with the 0 value of each field, this will result in each property that is pointers to be nil, strings will be "", ints will be 0, and so on.
I've created an API in Go that, upon being called, performs a query, creates an instance of a struct, and then encodes that struct as JSON before sending back to the caller. I'd now like to allow the caller to be able to select the specific fields they would like returned by passing in a "fields" GET parameter.
This means depending on the fields value(s), my struct would change. Is there any way to remove fields from a struct? Or at least hide them in the JSON response dynamically? (Note: Sometimes I have empty values so the JSON omitEmpty tag will not work here) If neither of these are possible, is there a suggestion on a better way to handle this?
A smaller version of the structs I'm using are below:
type SearchResult struct {
Date string `json:"date"`
IdCompany int `json:"idCompany"`
Company string `json:"company"`
IdIndustry interface{} `json:"idIndustry"`
Industry string `json:"industry"`
IdContinent interface{} `json:"idContinent"`
Continent string `json:"continent"`
IdCountry interface{} `json:"idCountry"`
Country string `json:"country"`
IdState interface{} `json:"idState"`
State string `json:"state"`
IdCity interface{} `json:"idCity"`
City string `json:"city"`
} //SearchResult
type SearchResults struct {
NumberResults int `json:"numberResults"`
Results []SearchResult `json:"results"`
} //type SearchResults
I then encode and output the response like so:
err := json.NewEncoder(c.ResponseWriter).Encode(&msg)
The question is asking for fields to be dynamically selected based on the caller-provided list of fields. This isn't possible to be done with the statically-defined json struct tag.
If what you want is to always skip a field to json-encode, then of course use json:"-" to ignore the field. (Note also that this is not required if your field is unexported; those fields are always ignored by the json encoder.) This isn't what the question asks.
To quote the comment on the json:"-" answer:
This [the json:"-" answer] is the answer most people ending up here from searching would want, but it's not the answer to the question.
I'd use a map[string]interface{} instead of a struct in this case. You can easily remove fields by calling the delete built-in on the map for the fields to remove.
That is, if you can't query only for the requested fields in the first place.
use `json:"-"`
// Field is ignored by this package.
Field int `json:"-"`
// Field appears in JSON as key "myName".
Field int `json:"myName"`
// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`
// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`
doc : http://golang.org/pkg/encoding/json/#Marshal
Another way to do this is to have a struct of pointers with the ,omitempty tag. If the pointers are nil, the fields won't be Marshalled.
This method will not require additional reflection or inefficient use of maps.
Same example as jorelli using this method: http://play.golang.org/p/JJNa0m2_nw
You can use the reflect package to select the fields that you want by reflecting on the field tags and selecting the json tag values. Define a method on your SearchResults type that selects the fields you want and returns them as a map[string]interface{}, and then marshal that instead of the SearchResults struct itself. Here's an example of how you might define that method:
func fieldSet(fields ...string) map[string]bool {
set := make(map[string]bool, len(fields))
for _, s := range fields {
set[s] = true
}
return set
}
func (s *SearchResult) SelectFields(fields ...string) map[string]interface{} {
fs := fieldSet(fields...)
rt, rv := reflect.TypeOf(*s), reflect.ValueOf(*s)
out := make(map[string]interface{}, rt.NumField())
for i := 0; i < rt.NumField(); i++ {
field := rt.Field(i)
jsonKey := field.Tag.Get("json")
if fs[jsonKey] {
out[jsonKey] = rv.Field(i).Interface()
}
}
return out
}
and here's a runnable solution that shows how you would call this method and marshal your selection: http://play.golang.org/p/1K9xjQRnO8
I just published sheriff, which transforms structs to a map based on tags annotated on the struct fields. You can then marshal (JSON or others) the generated map. It probably doesn't allow you to only serialize the set of fields the caller requested, but I imagine using a set of groups would allow you to cover most cases. Using groups instead of the fields directly would most likely also increase cache-ability.
Example:
package main
import (
"encoding/json"
"fmt"
"log"
"github.com/hashicorp/go-version"
"github.com/liip/sheriff"
)
type User struct {
Username string `json:"username" groups:"api"`
Email string `json:"email" groups:"personal"`
Name string `json:"name" groups:"api"`
Roles []string `json:"roles" groups:"api" since:"2"`
}
func main() {
user := User{
Username: "alice",
Email: "alice#example.org",
Name: "Alice",
Roles: []string{"user", "admin"},
}
v2, err := version.NewVersion("2.0.0")
if err != nil {
log.Panic(err)
}
o := &sheriff.Options{
Groups: []string{"api"},
ApiVersion: v2,
}
data, err := sheriff.Marshal(o, user)
if err != nil {
log.Panic(err)
}
output, err := json.MarshalIndent(data, "", " ")
if err != nil {
log.Panic(err)
}
fmt.Printf("%s", output)
}
Take three ingredients:
The reflect package to loop over all the fields of a struct.
An if statement to pick up the fields you want to Marshal, and
The encoding/json package to Marshal the fields of your liking.
Preparation:
Blend them in a good proportion. Use reflect.TypeOf(your_struct).Field(i).Name() to get a name of the ith field of your_struct.
Use reflect.ValueOf(your_struct).Field(i) to get a type Value representation of an ith field of your_struct.
Use fieldValue.Interface() to retrieve the actual value (upcasted to type interface{}) of the fieldValue of type Value (note the bracket use - the Interface() method produces interface{}
If you luckily manage not to burn any transistors or circuit-breakers in the process you should get something like this:
func MarshalOnlyFields(structa interface{},
includeFields map[string]bool) (jsona []byte, status error) {
value := reflect.ValueOf(structa)
typa := reflect.TypeOf(structa)
size := value.NumField()
jsona = append(jsona, '{')
for i := 0; i < size; i++ {
structValue := value.Field(i)
var fieldName string = typa.Field(i).Name
if marshalledField, marshalStatus := json.Marshal((structValue).Interface()); marshalStatus != nil {
return []byte{}, marshalStatus
} else {
if includeFields[fieldName] {
jsona = append(jsona, '"')
jsona = append(jsona, []byte(fieldName)...)
jsona = append(jsona, '"')
jsona = append(jsona, ':')
jsona = append(jsona, (marshalledField)...)
if i+1 != len(includeFields) {
jsona = append(jsona, ',')
}
}
}
}
jsona = append(jsona, '}')
return
}
Serving:
serve with an arbitrary struct and a map[string]bool of fields you want to include, for example
type magic struct {
Magic1 int
Magic2 string
Magic3 [2]int
}
func main() {
var magic = magic{0, "tusia", [2]int{0, 1}}
if json, status := MarshalOnlyFields(magic, map[string]bool{"Magic1": true}); status != nil {
println("error")
} else {
fmt.Println(string(json))
}
}
Bon Appetit!
I created this function to convert struct to JSON string by ignoring some fields. Hope it will help.
func GetJSONString(obj interface{}, ignoreFields ...string) (string, error) {
toJson, err := json.Marshal(obj)
if err != nil {
return "", err
}
if len(ignoreFields) == 0 {
return string(toJson), nil
}
toMap := map[string]interface{}{}
json.Unmarshal([]byte(string(toJson)), &toMap)
for _, field := range ignoreFields {
delete(toMap, field)
}
toJson, err = json.Marshal(toMap)
if err != nil {
return "", err
}
return string(toJson), nil
}
Example: https://play.golang.org/p/nmq7MFF47Gp
You can use tagging attribute "omitifempty" or make optional fields pointers and leave those you want skipped uninitialized.
Here is how I defined my structure.
type User struct {
Username string `json:"username" bson:"username"`
Email string `json:"email" bson:"email"`
Password *string `json:"password,omitempty" bson:"password"`
FullName string `json:"fullname" bson:"fullname"`
}
And inside my function set user.Password = nil for not to be Marshalled.
I didn't have the same problem but similar. Below code solves your problem too, of course if you don't mind performance issue. Before implement that kind of solution to your system I recommend you to redesign your structure if you can. Sending variable structure response is over-engineering. I believe a response structure represents a contract between a request and resource and it should't be depend requests.(you can make un-wanted fields null, I do). In some cases we have to implement this design, if you believe you are in that cases here is the play link and code I use.
type User2 struct {
ID int `groups:"id" json:"id,omitempty"`
Username string `groups:"username" json:"username,omitempty"`
Nickname string `groups:"nickname" json:"nickname,omitempty"`
}
type User struct {
ID int `groups:"private,public" json:"id,omitempty"`
Username string `groups:"private" json:"username,omitempty"`
Nickname string `groups:"public" json:"nickname,omitempty"`
}
var (
tagName = "groups"
)
//OmitFields sets fields nil by checking their tag group value and access control tags(acTags)
func OmitFields(obj interface{}, acTags []string) {
//nilV := reflect.Value{}
sv := reflect.ValueOf(obj).Elem()
st := sv.Type()
if sv.Kind() == reflect.Struct {
for i := 0; i < st.NumField(); i++ {
fieldVal := sv.Field(i)
if fieldVal.CanSet() {
tagStr := st.Field(i).Tag.Get(tagName)
if len(tagStr) == 0 {
continue
}
tagList := strings.Split(strings.Replace(tagStr, " ", "", -1), ",")
//fmt.Println(tagList)
// ContainsCommonItem checks whether there is at least one common item in arrays
if !ContainsCommonItem(tagList, acTags) {
fieldVal.Set(reflect.Zero(fieldVal.Type()))
}
}
}
}
}
//ContainsCommonItem checks if arrays have at least one equal item
func ContainsCommonItem(arr1 []string, arr2 []string) bool {
for i := 0; i < len(arr1); i++ {
for j := 0; j < len(arr2); j++ {
if arr1[i] == arr2[j] {
return true
}
}
}
return false
}
func main() {
u := User{ID: 1, Username: "very secret", Nickname: "hinzir"}
//assume authenticated user doesn't has permission to access private fields
OmitFields(&u, []string{"public"})
bytes, _ := json.Marshal(&u)
fmt.Println(string(bytes))
u2 := User2{ID: 1, Username: "very secret", Nickname: "hinzir"}
//you want to filter fields by field names
OmitFields(&u2, []string{"id", "nickname"})
bytes, _ = json.Marshal(&u2)
fmt.Println(string(bytes))
}
I also faced this problem, at first I just wanted to specialize the responses in my http handler. My first approach was creating a package that copies the information of a struct to another struct and then marshal that second struct. I did that package using reflection, so, never liked that approach and also I wasn't dynamically.
So I decided to modify the encoding/json package to do this. The functions Marshal, MarshalIndent and (Encoder) Encode additionally receives a
type F map[string]F
I wanted to simulate a JSON of the fields that are needed to marshal, so it only marshals the fields that are in the map.
https://github.com/jtorz/jsont
package main
import (
"fmt"
"log"
"net/http"
"github.com/jtorz/jsont/v2"
)
type SearchResult struct {
Date string `json:"date"`
IdCompany int `json:"idCompany"`
Company string `json:"company"`
IdIndustry interface{} `json:"idIndustry"`
Industry string `json:"industry"`
IdContinent interface{} `json:"idContinent"`
Continent string `json:"continent"`
IdCountry interface{} `json:"idCountry"`
Country string `json:"country"`
IdState interface{} `json:"idState"`
State string `json:"state"`
IdCity interface{} `json:"idCity"`
City string `json:"city"`
} //SearchResult
type SearchResults struct {
NumberResults int `json:"numberResults"`
Results []SearchResult `json:"results"`
} //type SearchResults
func main() {
msg := SearchResults{
NumberResults: 2,
Results: []SearchResult{
{
Date: "12-12-12",
IdCompany: 1,
Company: "alfa",
IdIndustry: 1,
Industry: "IT",
IdContinent: 1,
Continent: "america",
IdCountry: 1,
Country: "México",
IdState: 1,
State: "CDMX",
IdCity: 1,
City: "Atz",
},
{
Date: "12-12-12",
IdCompany: 2,
Company: "beta",
IdIndustry: 1,
Industry: "IT",
IdContinent: 1,
Continent: "america",
IdCountry: 2,
Country: "USA",
IdState: 2,
State: "TX",
IdCity: 2,
City: "XYZ",
},
},
}
fmt.Println(msg)
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
//{"numberResults":2,"results":[{"date":"12-12-12","idCompany":1,"idIndustry":1,"country":"México"},{"date":"12-12-12","idCompany":2,"idIndustry":1,"country":"USA"}]}
err := jsont.NewEncoder(w).Encode(msg, jsont.F{
"numberResults": nil,
"results": jsont.F{
"date": nil,
"idCompany": nil,
"idIndustry": nil,
"country": nil,
},
})
if err != nil {
log.Fatal(err)
}
})
http.ListenAndServe(":3009", nil)
}
The question is now a bit old, but I came across the same issue a little while ago, and as I found no easy way to do this, I built a library fulfilling this purpose.
It allows to easily generate a map[string]interface{} from a static struct.
https://github.com/tuvistavie/structomap
To extend chhaileng answer, here is the version that remove all occurrences of a field with recursion
// GetJSONWithOutFields - Description: return a string representation of an interface with specified fields removed
func GetJSONWithOutFields(obj interface{}, ignoreFields ...string) (string, error) {
toJson, err := json.Marshal(obj)
if err != nil {
return "", err
}
if len(ignoreFields) == 0 {
return string(toJson), nil
}
toMap := map[string]interface{}{}
err = json.Unmarshal(toJson, &toMap)
if err != nil {
return "", err
}
for _, field := range ignoreFields {
DeleteField(toMap, field)
}
toJson, err = json.Marshal(toMap)
if err != nil {
return "", err
}
return string(toJson), nil
}
// DeleteField - Description: recursively delete field
func DeleteField(toMap map[string]interface{}, field string) {
delete(toMap, field)
for _, v := range toMap {
if m, isMap := v.(map[string]interface{}); isMap {
DeleteField(m, field)
}
}
}