There are multiple nodes in my system which communicate through RPC. I am trying to send a map[string] interface{} to another node through RPC. Sender uses json.Marshal and receiver uses json.Unmarshal to get the map.
Let us say at the sender side, map contains [1] => 2 where 2 is of type uint32.
The problem is Unmarshal tries to find the type of underlying data and converts 2 to float64 type according to its default behavior as specified here https://blog.golang.org/json-and-go. Later, casting float64 to uint32 causes panic.
I refered to How to unmarshal json into interface{} in golang? . But for this, we need to know the type of data. In my case data can be of any type so I want to keep it as interface{}. How do I unmarshal from interface{} to interface{}?
Unfortunately using the encoding/json package you can't, because type information is not transmitted, and JSON numbers by default are unmarshaled into values of float64 type if type information is not present. You would need to define struct types where you explicitly state the field is of type uint32.
Alternatively you may opt to use encoding/gob which does transmit and preserve type information. See this example:
m := map[string]interface{}{"1": uint32(1)}
b := &bytes.Buffer{}
gob.NewEncoder(b).Encode(m)
var m2 map[string]interface{}
gob.NewDecoder(b).Decode(&m2)
fmt.Printf("%T\n%#v\n", m2["1"], m2)
Output (try it on the Go Playground):
uint32
map[string]interface {}{"1":0x1}
The downside of gob is that it's Go-specific unlike the language and platform independent JSON.
Related
I have a struct which looks like this, where PrivateKey & PublicKey are own types mapping to []byte:
type Secret struct {
Cert *x509.Certificate
ValidFor string
Private PrivateKey
Public PublicKey
}
This struct is embedded as a field in another structure, which gets marshalled into a JSON. Marshalling the outer structure seems to work fine and the JSON looks okay, however unmarshalling seems to cause the following error:
json: cannot unmarshal number json: cannot unmarshal number 54368953042[...number shortened...] into Go struct field Certificate.Secrets.Cert.PublicKey of type float64
Obviously, it seems that the unmarshaller is tripping on the big.int which contains the public key. Now, I found a solution online which tells me to first unmarshal to a map[string]interface{}, however given that my structs are a bit more nested, this seems like an exhaustive solution.
Now I'm wondering, is there any easier way to marshal & unmarshal a x509.Certificate with big.Ints in it? Or is the best way indeed to manually store, replace & restore problematic fields?
I am trying to learn Backend development by building a vary basic REST API using gorilla mux library in Go (following this tutorial)
Here's the code that I have built so far:
package main
import (
"encoding/json"
"net/http"
"github.com/gorilla/mux"
)
// Post represents single post by user
type Post struct {
Title string `json:"title"`
Body string `json:"body"`
Author User `json:"author"`
}
// User is struct that represnets a user
type User struct {
FullName string `json:"fullName"`
Username string `json:"username"`
Email string `json:"email"`
}
var posts []Post = []Post{}
func main() {
router := mux.NewRouter()
router.HandleFunc("/posts", addItem).Methods("POST")
http.ListenAndServe(":5000", router)
}
func addItem(w http.ResponseWriter, req *http.Request) {
var newPost Post
json.NewDecoder(req.Body).Decode(&newPost)
posts = append(posts, newPost)
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(posts)
}
However, I'm really confused about what exactly is happening in json.NewDecoder and json.NewEncoder part.
As far as I understand, ultimately data transfer over internet in a REST API will happen in form of bytes/binary format (encoded in UTF-8 i guess?). So json.NewEncoder is converting Go data strcuture to JSON string and json.NewDecoder is doing the opposite (correct me if i'm wrong).
So who is responsible here for converting this JSON string to UTF-8
encoding for data transfer? Is that also part of what json.NewDecoder and json.NewEncoder
do?
Also, if these 2 functions are only serializing/de-serializing
to/from JSON, why the name encoder and decoder (isn't encoding
always related to binary data conversion?). Honestly i'm pretty confused with the terms encoding, serialization, marshaling and the difference between them
Can someone just explain how exactly is data transfer happening here at each conversion level (json, binary, in-memory data structure)?
First, We have to understand that the Encoding process doesn't actually mean that it translates types and returns a JSON representation of a type. The process that gives you the JSON representation is called the Marshaling process and could be done by calling the json.Marshal function.
On the other hand, the Encoding process means that we want to get the JSON encoding of any type and to write(encode) it on a stream that implements io.Writer interface. As we can see the func NewEncoder(w io.Writer) *Encoder receives an io.Writer interface as a parameter and returns a *json.Encoder object. When the method encoder.Encode() is being called, it does the Marshaling process and then writes the result to the io.Writer that we have passed when creating a new Encoder object. You could see the implementation of json.Encoder.Encode() here.
So, if you asked who does do the encoding process to the http stream, the answer is the http.ResponseWriter. ResponseWriter implements the io.Writer interface and when the Encode() method is being called, the encoder will Marshal the object to a JSON encoding representation and then call the func Write([]byte) (int, error) which is a contract method of the io.Writer interface and it will do the writing process to the http stream.
In summary, I could say that Marshal and Unmarshal mean that we want to get the JSON representation of any type and vice-versa. While Encode means that we want to do the Marshaling process and then write(encode) the result to any stream object. And Decode means that we want to get(decode) a json object from any stream and then do the Unmarshaling process.
The json.Encoder produced by the call to json.NewEncoder directly produces its output in UTF-8. No conversion is necessary. (In fact, Go does not have a representation for textual data that is distinct from UTF-8 encoded sequences of bytes — even a string is just an immutable array of bytes under the hood.)
Go uses the term encode for serialisation and decode for deserialisation, whether the serialised form is binary or textual. Do not think too much about the terminology — consider encode and seralise as synonyms.
code and error message
I was experiencing this problem when using golang's elasticsearch client "elastic".
*json.RawMessage is a type of []byte.If I not write []byte() to cast type, it said cannot use 'resp.Source' (type *json.RawMessage) as type []byte.
The environment is go 1.10.3 and elastic 6.x
A *json.RawMesasge is not a []byte. It's a pointer.
Dereference the pointer to convert to a slice of bytes:
byteSlice := []byte(*pointerToRawMessage)
When you unmarshal JSON to []interface{} is there any way to automatically detect the type besides some standard types like bool, int and string?
What I noticed is the following, Let's say I marshal [uuid.UUID, bool] then the JSON I get looks like:
[[234,50,7,116,194,41,64,225,177,151,60,195,60,45,123,106],true]
When I unmarshal it again, I get the types as shown through reflect:
[]interface{}, bool
I don't understand why it picked []interface{}. If it cannot detect it, shouldn't it be at least interface{}?
In any case, my question is, is it possible to unmarshal any type when the target is of type []interface{}? It seems to work for standard types like string, bool, int but for custom types I don't think that's possible, is it? You can define custom JSON marshal/unmarshal methods but that only works if you decode it into a target type so that it can look up which custom marshal/unmarshal methods to use.
You can unmarshal any type into a value of type interface{}. If you use a value of type []interface{}, you can only unmarshal JSON arrays into it, but yes, the elements of the array may be of any type.
Since you're using interface{} or []interface{}, yes, type information is not available, and it's up to the encoding/json package to choose the best it sees fit. For example, for JSON objects it will choose map[string]interface{}. The full list of default types is documented in json.Unmarshal():
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Obviously if your JSON marshaling/unmarshaling logic needs some pre- / postprocessing, the json package will not miraculously find it out. It can know about those only if you unmarshal into values of specific types (which implement json.Unmarshaler). The json package will still be able to unmarshal them to the default types, but custom logic will obviously not run on them.
I am attempting to define a custom JSON marshaler to display some time information in specific formats. Ideally, I'd like to have a struct that stores created/modified values and then embed them into structs that need to keep track of that information. In addition, I want to define a custom date format in the JSON marshaler to use from a client-side application.
I currently have two structs
type Timestamp struct {
Created time.Time
Modified time.Time
}
type Company struct {
Id string
Name string
Timestamp
}
I want to embed the Timestamp struct into objects that will need to record when items are updated/created. Nothing insane there.
My issues comes when I define
func (t Timestamp) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
CreatedFormatted string
}{
CreatedFormatted: t.Created.Format("Monday Jan _2 15:04:05 2006"),
})
}
When I go to marshal Company, I only see the json for the Timestamp but nothing for the Company. I would have thought that the contents of the Company struct and the Timestamp struct would have been displayed. Am I doing something wrong here?
When a type embedded in a struct provides a method, that method becomes a part of the embedding struct. Since Company embeds Timestamp, Timestamp's MarshalJSON is available for Company as well. When json is looking to marshal a Company it sees that it has a MarshalJSON method and calls it — and the method it finds only marshals the timestamp field. The default behavior of structs (to marshal each field into its own key in a JSON object) is overridden.
What you can do:
Don't use struct embedding, just make Timestamp a regular field (even if you declare it as Timestamp Timestamp). Then Company won't inherit its methods and JSON will work as expected (but other parts of your code that expect embedding might change). Or:
Give Company its own MarshalJSON method that marshalls all of the fields including the timestamp. You could do this by
a. Copying the fields into a different type that is identical to Company but without the embedding, and marshalling that.
b. Copying the fields into a map and marshalling that.
c. Marshalling each of the fields independently and string-pasting fmt.Sprintf({"key1":%s,"key2":%s,...}, m1, m2, ...) yourself.
By embedding Timestamp in Company you not only shared member variables, but also the methods. This means you provided Company.MarshalJSON method, which is then used by json package for marshalling the whole structure. In order to see all the fields you would need to implement an explicit marshaler for Company structure as well.
If you only want to format the timestamp in a specific way, the other solution would be to provide your own time.Time and providing JSON marshaller there.
For example:
type AccessTime time.Time
func (t AccessTime) MarshalJSON() ([]byte, error) {
return json.Marshal(time.Time(t).Format("Monday Jan _2 15:04:05 2006"))
}
type Timestamp struct {
Created AccessTime
Modified AccessTime
}
https://play.golang.org/p/PhZXPauSyz