I have a struct which looks like this, where PrivateKey & PublicKey are own types mapping to []byte:
type Secret struct {
Cert *x509.Certificate
ValidFor string
Private PrivateKey
Public PublicKey
}
This struct is embedded as a field in another structure, which gets marshalled into a JSON. Marshalling the outer structure seems to work fine and the JSON looks okay, however unmarshalling seems to cause the following error:
json: cannot unmarshal number json: cannot unmarshal number 54368953042[...number shortened...] into Go struct field Certificate.Secrets.Cert.PublicKey of type float64
Obviously, it seems that the unmarshaller is tripping on the big.int which contains the public key. Now, I found a solution online which tells me to first unmarshal to a map[string]interface{}, however given that my structs are a bit more nested, this seems like an exhaustive solution.
Now I'm wondering, is there any easier way to marshal & unmarshal a x509.Certificate with big.Ints in it? Or is the best way indeed to manually store, replace & restore problematic fields?
Related
I am trying to learn Backend development by building a vary basic REST API using gorilla mux library in Go (following this tutorial)
Here's the code that I have built so far:
package main
import (
"encoding/json"
"net/http"
"github.com/gorilla/mux"
)
// Post represents single post by user
type Post struct {
Title string `json:"title"`
Body string `json:"body"`
Author User `json:"author"`
}
// User is struct that represnets a user
type User struct {
FullName string `json:"fullName"`
Username string `json:"username"`
Email string `json:"email"`
}
var posts []Post = []Post{}
func main() {
router := mux.NewRouter()
router.HandleFunc("/posts", addItem).Methods("POST")
http.ListenAndServe(":5000", router)
}
func addItem(w http.ResponseWriter, req *http.Request) {
var newPost Post
json.NewDecoder(req.Body).Decode(&newPost)
posts = append(posts, newPost)
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(posts)
}
However, I'm really confused about what exactly is happening in json.NewDecoder and json.NewEncoder part.
As far as I understand, ultimately data transfer over internet in a REST API will happen in form of bytes/binary format (encoded in UTF-8 i guess?). So json.NewEncoder is converting Go data strcuture to JSON string and json.NewDecoder is doing the opposite (correct me if i'm wrong).
So who is responsible here for converting this JSON string to UTF-8
encoding for data transfer? Is that also part of what json.NewDecoder and json.NewEncoder
do?
Also, if these 2 functions are only serializing/de-serializing
to/from JSON, why the name encoder and decoder (isn't encoding
always related to binary data conversion?). Honestly i'm pretty confused with the terms encoding, serialization, marshaling and the difference between them
Can someone just explain how exactly is data transfer happening here at each conversion level (json, binary, in-memory data structure)?
First, We have to understand that the Encoding process doesn't actually mean that it translates types and returns a JSON representation of a type. The process that gives you the JSON representation is called the Marshaling process and could be done by calling the json.Marshal function.
On the other hand, the Encoding process means that we want to get the JSON encoding of any type and to write(encode) it on a stream that implements io.Writer interface. As we can see the func NewEncoder(w io.Writer) *Encoder receives an io.Writer interface as a parameter and returns a *json.Encoder object. When the method encoder.Encode() is being called, it does the Marshaling process and then writes the result to the io.Writer that we have passed when creating a new Encoder object. You could see the implementation of json.Encoder.Encode() here.
So, if you asked who does do the encoding process to the http stream, the answer is the http.ResponseWriter. ResponseWriter implements the io.Writer interface and when the Encode() method is being called, the encoder will Marshal the object to a JSON encoding representation and then call the func Write([]byte) (int, error) which is a contract method of the io.Writer interface and it will do the writing process to the http stream.
In summary, I could say that Marshal and Unmarshal mean that we want to get the JSON representation of any type and vice-versa. While Encode means that we want to do the Marshaling process and then write(encode) the result to any stream object. And Decode means that we want to get(decode) a json object from any stream and then do the Unmarshaling process.
The json.Encoder produced by the call to json.NewEncoder directly produces its output in UTF-8. No conversion is necessary. (In fact, Go does not have a representation for textual data that is distinct from UTF-8 encoded sequences of bytes — even a string is just an immutable array of bytes under the hood.)
Go uses the term encode for serialisation and decode for deserialisation, whether the serialised form is binary or textual. Do not think too much about the terminology — consider encode and seralise as synonyms.
I'm new to Go. I was trying to fetch and marshal json data to a struct. My sample data looks like this:
var reducedFieldData = []byte(`[
{"model":"Traverse","vin":"1gnkrhkd6ej111234"}
,{"model":"TL","vin":"19uua66265a041234"}
]`)
If I define the struct for receiving the data like this:
type Vehicle struct {
Model string
Vin string
}
The call to Unmarshal works as expected. However, if I use lower case for the fields ("model" and "vin") which actually matches cases for the field names in the data it will return empty strings for the values.
Is this expected behavior? Can the convention be turned off?
Fields need to be exported (declared with an uppercase first letter) or the reflection library cannot edit them. Since the JSON (un)marshaller uses reflection, it cannot read or write unexported fields.
So yes, it is expected, and no, you cannot change it. Sorry.
You can add tags to a field to change the name the marshaller uses:
Model string `json:"model"`
See the documentation for more info on the field tags "encoding/json" supports.
When you unmarshal JSON to []interface{} is there any way to automatically detect the type besides some standard types like bool, int and string?
What I noticed is the following, Let's say I marshal [uuid.UUID, bool] then the JSON I get looks like:
[[234,50,7,116,194,41,64,225,177,151,60,195,60,45,123,106],true]
When I unmarshal it again, I get the types as shown through reflect:
[]interface{}, bool
I don't understand why it picked []interface{}. If it cannot detect it, shouldn't it be at least interface{}?
In any case, my question is, is it possible to unmarshal any type when the target is of type []interface{}? It seems to work for standard types like string, bool, int but for custom types I don't think that's possible, is it? You can define custom JSON marshal/unmarshal methods but that only works if you decode it into a target type so that it can look up which custom marshal/unmarshal methods to use.
You can unmarshal any type into a value of type interface{}. If you use a value of type []interface{}, you can only unmarshal JSON arrays into it, but yes, the elements of the array may be of any type.
Since you're using interface{} or []interface{}, yes, type information is not available, and it's up to the encoding/json package to choose the best it sees fit. For example, for JSON objects it will choose map[string]interface{}. The full list of default types is documented in json.Unmarshal():
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Obviously if your JSON marshaling/unmarshaling logic needs some pre- / postprocessing, the json package will not miraculously find it out. It can know about those only if you unmarshal into values of specific types (which implement json.Unmarshaler). The json package will still be able to unmarshal them to the default types, but custom logic will obviously not run on them.
I post a request to a server and get a reply in JSON format. I'm able to unmarshal it to a struct. Then I need to create a new JSON file with the same data but different JSON tags.
Example:
In the following code, I get {"name":"Sam","age":20} from a server and unmarshal it to the struct Foo:
type Foo struct {
Name string `json:"name"`
Age int `json:"age"`
}
Then I need to change the tag name to employee_name and omit age:
type Bar struct {
Name string `json:"employee_name"`
Age int `json:"-"`
}
After that I send this modified data to another server.
I know I could just create a new Bar and copy all data into it, but there are a lot of fields. I was wondering if there is a way to attach multiple JSON tags like this:
type Foo struct {
Name string `json:"name" json:"employee_name"`
Age int `json:"age" json:"-"`
}
Thanks in advance.
It's not possible. The encoding/json package only handles the json key in struct tags. If the json key is listed multiple times (as in your example), the first occurrence will be used (this is implemented in StructTag.Get()).
Note that this is an implementation restriction of the encoding/json package and not that of Go. One could easily create a similar JSON encoding package supporting either multiple tag keys (e.g. json1, json2) or multiple occurrences of the same key (as in your example).
What is possible though, with 2 identically laid out structs (namin, types and ordering of fields needs to match exactly) is to cast from one to the other. I would be very cautious of doing this though and make sure the 2nd type (bar in your example) is unexported to prevent from being used elsewhere.
https://play.golang.org/p/y8EH1U9_3jN
I want to deserialise an object that includes an array of a some interface Entity:
type Result struct {
Foo int;
Bar []Entity;
};
Entity is an interface that is implemented by a number of struct types. JSON data identifies the struct type with a "type" field in each entity. E.g.
{"type":"t1","field1":1}
{"type":"t2","field2":2,"field3":3}
How would I go about deserialising the Result type in such a way that it correctly populates the array. From what I can see, I have to:
Implement UnmarshalJSON on Result.
Parse Bar as a []*json.RawMessage.
Parse each raw message as map[string]interface{}.
Check "type" field in the raw message.
Create a struct of appropriate type.
Parse the raw message again, this time into the just created struct.
This all sounds very tedious and boring. Is there a better way to do this? Or am I doing it backwards, and there is a more canonical method to handle an array of heterogeneous objects?
I think your process is probably a bit more complicated than it has to be, see http://play.golang.org/p/0gahcMpuQc. A single map[string]interface{} will handle a lot of that for you.
Alternatively, you could make a type like
struct EntityUnion {
Type string
// Fields from t1
// Fields from t2
// ...
}
Unmarshal into that; it will set the Type string and fill in all the fields it can get from the JSON data. Then you just need a small function to copy the fields to the specific type.