I have a Set data structure implemented in Go with the basic operations like Add, Remove, Difference, Union. I am trying to send a http request using the json encoder to encode the request body which contains the object of the form map[string]Set. The Set data structure is defined below:
type Set map[interface{}]struct{}
func NewSet() Set {
set := make(Set)
return set
}
The encoder looks like this:
func (req *Request) BodyContentInJson (val interface{}) error {
buf := bytes.NewBuffer(nil)
enc := json.NewEncoder(buf)
if err := enc.Encode(val); err != nil {
return err
}
req.Obj = val
req.Body = buf
req.BodySize = int64(buf.Len())
return nil
}
This code fails at
if err := enc.Encode(val); err != nil {
return err
}
giving an error:{"errors":["json: unsupported type: Set"]}. Also, the type of val is map[string]interface{}when I debugged it.
How could I possibly encode and marshal/unmarshal JSON content here using the Go's encoder ?
You could write your own UnmarshalJSON method on the *Set type which would then be used by the json.Encoder to encode the json data into the Set. Here's a simple example https://play.golang.org/p/kx1E-jDu5e.
By the way, the reason you're getting the error is because a map key of type interface{} is not supported by the encoding/json package. (https://github.com/golang/go/blob/master/src/encoding/json/encode.go#L697-L699)
Related
I have a config file in YAML format, which I am trying to output as JSON via an http API call. I am unmarshalling using gopkg.in/yaml.v2. Yaml can have non-string keys, which means that the yaml is unmarshalled as map[interface{}]interface{}, which is not supported by Go's JSON marshaller. Therefore I convert to map[string]interface{} before unmarshalling. But I still get: json: unsupported type: map[interface {}]interface" {}. I don't understand. The variable cfy is not map[interface{}]interface{}.
import (
"io/ioutil"
"net/http"
"encoding/json"
"gopkg.in/yaml.v2"
)
func GetConfig(w http.ResponseWriter, r *http.Request) {
cfy := make(map[interface{}]interface{})
f, err := ioutil.ReadFile("config/config.yml")
if err != nil {
// error handling
}
if err := yaml.Unmarshal(f, &cfy); err != nil {
// error handling
}
//convert to a type that json.Marshall can digest
cfj := make(map[string]interface{})
for key, value := range cfy {
switch key := key.(type) {
case string:
cfj[key] = value
}
}
j, err := json.Marshal(cfj)
if err != nil {
// errr handling. We get: "json: unsupported type: map[interface {}]interface" {}
}
w.Header().Set("content-type", "application/json")
w.Write(j)
}
Your solution only converts values at the "top" level. If a value is also a map (nested map), your solution does not convert those.
Also you only "copy" the values with string keys, the rest will be left out of the result map.
Here's a function that recursively converts nested maps:
func convert(m map[interface{}]interface{}) map[string]interface{} {
res := map[string]interface{}{}
for k, v := range m {
switch v2 := v.(type) {
case map[interface{}]interface{}:
res[fmt.Sprint(k)] = convert(v2)
default:
res[fmt.Sprint(k)] = v
}
}
return res
}
Testing it:
m := map[interface{}]interface{}{
1: "one",
"two": 2,
"three": map[interface{}]interface{}{
"3.1": 3.1,
},
}
m2 := convert(m)
data, err := json.Marshal(m2)
if err != nil {
panic(err)
}
fmt.Println(string(data))
Output (try it on the Go Playground):
{"1":"one","three":{"3.1":3.1},"two":2}
Some things to note:
To covert interface{} keys, I used fmt.Sprint() which will handle all types. The switch could have a dedicated string case for keys that are already string values to avoid calling fmt.Sprint(). This is solely for performance reasons, the result will be the same.
The above convert() function does not go into slices. So for example if the map contains a value which is a slice ([]interface{}) which may also contain maps, those will not be converted. For a full solution, see the lib below.
There is a lib github.com/icza/dyno which has an optimized, built-in support for this (disclosure: I'm the author). Using dyno, this is how it would look like:
var m map[interface{}]interface{} = ...
m2 := dyno.ConvertMapI2MapS(m)
dyno.ConvertMapI2MapS() also goes into and converts maps in []interface{} slices.
Also see possible duplicate: Convert yaml to json without struct
There is some inconvenience while processing JSON response from a web server.
For example, I don't know the data structure (and don't want to model it) of the JSON in advance, and just want to get the value from it!
So, for Python, I can just write
value = response["body"][4]["data"]["uid"] //response is a dictionary
But for Golang, I need to do the assertion for every element!
value := response["body"].([]interface{})[4].(map[string]interface{})["data"].(map[string]interface{})["uid"]
//response is a map[string]interface{}
This is what I write in golang to get the value I need. Do you have any suggestion on it? It there any useful tips for this kind of case?
If you model your JSON object with a struct and you unmarshal into a value of that, then you don't need those ugly indices and type assertions, you can simply refer to struct fields.
Note that you don't have to be afraid of the response being complex, you only need to model the parts you intend to use. E.g. if the response is an object with a hundred fields but you only need 2, then create a struct containing only those 2 fields.
If you don't want to model your JSON object (or can't because it's dynamic), then you may write a general utility function which gets a value based on the path (series of map keys and slice indices), which you can see in this answer: Taking a JSON string, unmarshaling it into a map[string]interface{}, editing, and marshaling it into a []byte seems more complicated then it should be
And last you may use 3rd party libs which already contain this helper functionality, such as https://github.com/icza/dyno (disclosure: I'm the author).
Using github.com/icza/dyno, it would look like this:
value, err := dyno.Get(response, "body", 4, "data", "uid")
As per icza you can create struct for JSON object. But in case you are getting JSON structure you don't know from start. Then you can create a dynamic parsing using reflections for interface which will be a recursive function to parse JSON data.
func main(){
var data interface{}
err := json.Unmarshal([]bytes(file.json), &data)
if err != nil {
panic(err)
}
var itemData map[string]interface{}
itemsMap := data.(map[string]interface{})
jsonParsedObject := interate(itemsMap)
log.Println(jsonParsedObject)
}
func iterate(data interface{}) interface{} {
if reflect.ValueOf(data).Kind() == reflect.Slice {
d := reflect.ValueOf(data)
tmpData := make([]interface{}, d.Len())
returnSlice := make([]interface{}, d.Len())
for i := 0; i < d.Len(); i++ {
tmpData[i] = d.Index(i).Interface()
}
for i, v := range tmpData {
returnSlice[i] = iterate(v)
}
return returnSlice
} else if reflect.ValueOf(data).Kind() == reflect.Map {
d := reflect.ValueOf(data)
tmpData := make(map[string]interface{})
for _, k := range d.MapKeys() {
typeOfValue := reflect.TypeOf(d.MapIndex(k).Interface()).Kind()
if typeOfValue == reflect.Map || typeOfValue == reflect.Slice {
tmpData[k.String()] = iterate(d.MapIndex(k).Interface())
} else {
tmpData[k.String()] = d.MapIndex(k).Interface()
}
}
return tmpData
}
return data
}
I have the following type in Golang:
type Base64Data []byte
In order to support unmarshalling a base64 encoded string to this type, I did the following:
func (b *Base64Data) UnmarshalJSON(data []byte) error {
if len(data) == 0 {
return nil
}
content, err := base64.StdEncoding.DecodeString(string(data[1 : len(data)-1]))
if err != nil {
return err
}
*b = []byte(xml)
return nil
}
Now I also want to be able to marshal and unmarshal it to mongo database, using mgo Golang library.
The problem is that I already have documents there stored as base64 encoded string, so I have to maintain that.
I tried to do the following:
func (b Base64Data) GetBSON() (interface{}, error) {
return base64.StdEncoding.EncodeToString([]byte(b)), nil
}
func (b *Base64DecodedXml) SetBSON(raw bson.Raw) error {
var s string
var err error
if err = raw.Unmarshal(&s); err != nil {
return err
}
*b, err = base64.StdEncoding.DecodeString(s)
return err
}
So that after unmarshaling, the data is already decoded, so I need to encode it back, and return it as a string so it will be written to db as a string (and vice versa)
For that I implemented bson getter and setter, but it seems only the getter is working properly
JSON unmarshaling from base64 encoded string works, as well marshaling it to database. but unmarshling setter seems to not be called at all.
Can anyone suggest what I'm missing, so that I'll be able to properly hold the data decoded in memory, but encoded string type?
This is a test I tried to run:
b := struct {
Value shared.Base64Data `json:"value" bson:"value"`
}{}
s := `{"value": "PHJvb3Q+aGVsbG88L3Jvb3Q+"}`
require.NoError(t, json.Unmarshal([]byte(s), &b))
t.Logf("%v", string(b.Value))
b4, err := bson.Marshal(b)
require.NoError(t, err)
t.Logf("%v", string(b4))
require.NoError(t, bson.Unmarshal(b4, &b))
t.Logf("%v", string(b.Value))
You can't marshal any value with bson.Marshal(), only maps and struct values.
If you want to test it, pass a map, e.g. bson.M to bson.Marshal():
var x = Base64Data{0x01, 0x02, 0x03}
dd, err := bson.Marshal(bson.M{"data": x})
fmt.Println(string(dd), err)
Your code works as-is, and as you intend it to. Try to insert a wrapper value to verify it:
c := sess.DB("testdb").C("testcoll")
var x = Base64Data{0x01, 0x02, 0x03}
if err := c.Insert(bson.M{
"data": x,
}); err != nil {
panic(err)
}
This will save the data as a string, being the Base64 encoded form.
Of course if you want to load it back into a value of type Base64Data, you will also need to define the SetBSON(raw Raw) error method too (bson.Setter interface).
In the following example from Web Development with Go by Shiju Varghese, which is for implementing a HTTP server using a new MongoDB session for each HTTP request:
Why is json package's Decode method used in PostCategory function?
Why is json package's Marshal method used in GetCategories function?
At first I thought that Decode in PostCategory and Marshal in GetCategories are opposite to each other, but later I found that there is a Unmarshal method and maybe a Encode one in the json package. So I asked a question earlier.
Here is the program
package main
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
var session *mgo.Session
type (
Category struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Name string
Description string
}
DataStore struct {
session *mgo.Session
}
)
//Close mgo.Session
func (d *DataStore) Close() {
d.session.Close()
}
//Returns a collection from the database.
func (d *DataStore) C(name string) *mgo.Collection {
return d.session.DB("taskdb").C(name)
}
//Create a new DataStore object for each HTTP request
func NewDataStore() *DataStore {
ds := &DataStore{
session: session.Copy(),
}
return ds
}
//Insert a record
func PostCategory(w http.ResponseWriter, r *http.Request) {
var category Category
// Decode the incoming Category json
err := json.NewDecoder(r.Body).Decode(&category)
if err != nil {
panic(err)
}
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
//Insert record
err = c.Insert(&category)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusCreated)
}
//Read all records
func GetCategories(w http.ResponseWriter, r *http.Request) {
var categories []Category
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
iter := c.Find(nil).Iter()
result := Category{}
for iter.Next(&result) {
categories = append(categories, result)
}
w.Header().Set("Content-Type", "application/json")
j, err := json.Marshal(categories)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusOK)
w.Write(j)
}
func main() {
var err error
session, err = mgo.Dial("localhost")
if err != nil {
panic(err)
}
r := mux.NewRouter()
r.HandleFunc("/api/categories", GetCategories).Methods("GET")
r.HandleFunc("/api/categories", PostCategory).Methods("POST")
server := &http.Server{
Addr: ":8080",
Handler: r,
}
log.Println("Listening...")
server.ListenAndServe()
}
I think the main reason for using a json.NewDecoder here is to read directly from response's body (r.Body) here, since NewDecoder takes an io.Reader as an input.
You could have used json.Unmarshal but then you'd have to first read response body into a []byte and pass that value to json.Unmarshal. NewDecoder is more convenient here.
TL;DR — Marshal/Unmarshal take and return byte slices, while Encode/Decode do the same thing, but read the bytes from a stream such as a network connection (readers and writers).
The encoding/json package uses the Encoder and Decoder types to act on streams of data, that is, io.Reader's and io.Writer's. This means that you can take data directly from a network socket (or an HTTP body in this case which implements io.Reader) and transform it to JSON as the bytes come in. Doing it this way, we can go ahead and start processing that JSON as soon as any data is available but before we've received the whole document (on a slow network connection with a big document this could save us a lot of time, and for some streaming protocols with "infinitely sized" document streams this is absolutely necessary!)
Marshal and Unmarshal however operate on byte slices, which means that you have to have the entirety of the JSON document in memory before you can use them. In your example, the author uses Marshal because they already have a []byte slice so there's no point in constructing a buffer using the byte slice, then making an encoder that uses that buffer, then calling encode: Instead they can just let Marshal do that for them.
In reality, Marshal/Unmarshal are just convenience methods on top of Encoders and Decoders. If we look at the source for Unmarshal, we see that under the hood it's just constructing an encoder (or the internal representation of an encoder, but trust me, they're the same thing, if you want proof you can look at the Encode method source and see that it's also creating an encodeState) and then returning the output bytes:
func Marshal(v interface{}) ([]byte, error) {
e := &encodeState{}
err := e.marshal(v)
if err != nil {
return nil, err
}
return e.Bytes(), nil
}
In python you can take a json object and grab a specific item from it without declaring a struct, saving to a struct then obtaining the value like in Go. Is there a package or easier way to store a specific value from json in Go?
python
res = res.json()
return res['results'][0]
Go
type Quotes struct {
AskPrice string `json:"ask_price"`
}
quote := new(Quotes)
errJson := json.Unmarshal(content, "e)
if errJson != nil {
return "nil", fmt.Errorf("cannot read json body: %v", errJson)
}
You can decode into a map[string]interface{} and then get the element by key.
func main() {
b := []byte(`{"ask_price":"1.0"}`)
data := make(map[string]interface{})
err := json.Unmarshal(b, &data)
if err != nil {
panic(err)
}
if price, ok := data["ask_price"].(string); ok {
fmt.Println(price)
} else {
panic("wrong type")
}
}
Structs are often preferred as they are more explicit about the type. You only have to declare the fields in the JSON you care about, and you don't need to type assert the values as you would with a map (encoding/json handles that implicitly).
Try either fastjson or jsonparser. jsonparser is optimized for the case when a single JSON field must be selected, while fastjson is optimized for the case when multiple unrelated JSON fields must be selected.
Below is an example code for fastjson:
var p fastjson.Parser
v, err := p.Parse(content)
if err != nil {
log.Fatal(err)
}
// obtain v["ask_price"] as float64
price := v.GetFloat64("ask_price")
// obtain v["results"][0] as generic JSON value
result0 := v.Get("results", "0")