Is this possible way to add nested json "as is". The nested json doesn't have any structure and might be different. I need to put the nested json data directly to the root node.
https://play.golang.org/p/MzBt7DLQEpD
type RootJson struct {
NestedJson []byte
AdditionalField string
}
func main() {
nestedJson := []byte("{\"number\": 1, \"string\": \"string\", \"float\": 6.56}")
rootJson := RootJson{nestedJson, "additionalField"}
payload, _ := json.Marshal(&rootJson)
fmt.Println(string(payload))
}
Yes, it's possible. Use the json.RawMessage type which implements custom marshaling / unmarshaling, which "renders" it as-is into the JSON output. It's just a plain byte slice:
type RawMessage []byte
Its value should be the UTF-8 encoded byte sequence of the raw JSON text (exactly what you get when you do a conversion, e.g. []byte("someText")).
type RootJson struct {
NestedJson json.RawMessage
AdditionalField string
}
With this, the output will be (try it on the Go Playground):
{"NestedJson":{"number":1,"string":"string","float":6.56},
"AdditionalField":"additionalField"}
(Indentation added by me.)
Related
I would like to take an arbitrary http.Request and get the body as a json string. I know this involves the json package, but it seems json.Decode needs a specific struct passed in by reference. How can I decode an arbitrary request body (and then stringify the result)?
func RequestBodyJsonString(r *http.Request) string {
}
Use ioutil.ReadAll to get data in byte slice then type conversion to string to get json string
bytedata, err := ioutil.ReadAll(r.Body)
reqBodyString := string(data)
An example in go playground here
Currently, I try to parse JSON to map[string][]interface{}, but unmarshalling returns an error. According to (https://golang.org/pkg/encoding/json/), to unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
-[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
I wonder if golang is able to unmarshal map[string][]interface{}. The following is code snippet. I am new to Golang, thanks for help in advance.
// emailsStr looks like "{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}"
emailsRaw := make(map[string][]*entities.Email)
err := json.Unmarshal([]byte(emailsStr), &emailsRaw)
Error message:
&json.UnmarshalTypeError{Value:"number", Type:(*reflect.rtype)(0x151c7a0), Offset:44, Struct:"", Field:""}
The Go encoding/json package will only unmarshal dynamically to a map[string]interface{}. From there, you will need to use type assertions and casting to pull out the values you want, like so:
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
dynamic := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &dynamic)
firstEmail := dynamic["unknown.0"].([]interface{})[0].(map[string]interface{})["email_address"]
fmt.Println(firstEmail)
}
(https://play.golang.org/p/VEUEIwj3CIC)
Each time, Go's .(<type>) operator is used to assert and cast the dynamic value to a specific type. This particular code will panic if anything happens to be the wrong type at runtime, like if the contents of unknown.0 aren't an array of JSON objects.
The more idiomatic (and robust) way to do this in Go is to annotate a couple structs with json:"" tags and have encoding/json unmarshal into them. This avoids all the nasty brittle .([]interface{}) type casting:
type Email struct {
Email string `json:"email_address"`
}
type EmailsList struct {
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion int `json:"schemaVersion"`
Emails []Email `json:"unknown.0"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
emails := EmailsList{}
json.Unmarshal([]byte(jsonStr), &emails)
fmt.Printf("%+v\n", emails)
}
(https://play.golang.org/p/iS6e0_87P2J)
A better approach will be to use struct for main schema and then use an slice of email struct for fetching the data for email entities get the values from the same according to requirements. Please find the solution below :-
package main
import (
"fmt"
"encoding/json"
)
type Data struct{
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion float64 `json:"schemaVersion"`
EmailEntity []Email `json:"unknown.0"`
}
// Email struct
type Email struct{
EmailAddress string `json:"email_address"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
var dynamic Data
json.Unmarshal([]byte(jsonStr), &dynamic)
fmt.Printf("%#v", dynamic)
}
Background
I'm working with JSON data that must be non-repudiable.
The API that grants me this data also has a service to verify that the data originally came from them.
As best as I can tell, they do this by requiring that the complete JSON they originally sent needs to be supplied to them inside another JSON request, with no byte changes.
Issue
I can't seem to preserve the original JSON!
Because I cannot modify the original JSON, I have carefully preserved it as a json.RawMessage when unmarshalling:
// struct I unmarshal my original data into
type SignedResult struct {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
...
}
// struct I marshal my data back into
type VerifiedSignatureReq {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
}
// ... getData is placeholder for function that gets my data
response := SignedResult{}
x, _ := json.Unmarshal(getData(), &response)
// do some post-processing with SignedResult that does not alter `Raw` or `Signature`
// trouble begins here - x.Raw started off as json.RawMessage...
y := json.Marshal(VerifiedSignatureReq{Raw: x.Raw, Signature: x.Signature}
// but now y.Raw is base64-encoded.
The problem is that []bytes / RawMessages are base64-encoded when marshaled. So I can't use this method, because it completely alters the string.
I'm unsure how to ensure this string is correctly preserved. I had assumed that the json.RawMessage specification in my struct would survive the perils of marshaling an already marshaled instance because it implements the Marshaler interface, but I appear mistaken.
Things I've Tried
My next attempt was to try:
// struct I unmarshal my original data into
type SignedResult struct {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
...
}
// struct I marshal my data back into
type VerifiedSignatureReq {
Raw map[string]interface{} `json:"random"`
Signature string `json:"signature"`
}
// ... getData is placeholder for function that gets my data
response := SignedResult{}
x, _ := json.Unmarshal(getData(), &response)
// do some post-processing with SignedResult that does not alter `Raw` or `Signature`
var object map[string]interface{}
json.Unmarshal(x.Raw, &object)
// now correctly generates the JSON structure.
y := json.Marshal(VerifiedSignatureReq{Raw: object, Signature: x.Signature}
// but now this is not the same JSON string as received!
The issue with this approach is that there are minor byte-wise differences in the spacing between the data. It no longer looks exactly the same when catted to a file.
I cannot use string(x.Raw) either because it escapes certain characters when marshaled with \.
You will need a custom type with its own marshaler, in place of json.RawMessage for your VerifiedSignatureReq struct to use. Example:
type VerifiedSignatureReq {
Raw RawMessage `json:"random"`
Signature string `json:"signature"`
}
type RawMessage []byte
func (m RawMessage) MarshalJSON() ([]byte, error) {
return []byte(m), nil
}
I have this struct with a field of type interface{}. In the process of caching it using memcached (https://github.com/bradfitz/gomemcache), the struct is marshalled into a JSON, which is then unmarshalled back into the struct when retrieved from the cache. The resulting interface{} field inevitably points to an object of type map[string]interface{} (as in, the interface{} field can only type asserted as map[string]interface{}), the marshalling and unmarshalling process not having preserved the type information. Is there any way to save this information in the marshalling process, in such a way that it can be unmarshalled properly? Or do I have to use some other codec or something?
type A struct {
value interface{}
}
type B struct {
name string
id string
}
func main() {
a := A{value: B{name: "hi", id: "12345"}}
cache.Set("a", a) // Marshals 'a' into JSON and stores in cache
result = cache.Get("a") // Retrieves 'a' from cache and unmarshals
fmt.Printf("%s", result.value.(B).name) // Generates error saying that
// map[string]interface{} cannot be type asserted as a 'B' struct
fmt.Printf("%s", result.value.(map[string]interface{})["name"].(string)) // Correctly prints "12345"
}
Short version, no you can't do that, you have few options though.
Change A.Value to use B instead of interface{}.
Add a function to A that converts A.Value from a map to B if it isn't already B.
Use encoding/gob and store the bytes in memcache then convert it back with a function like NewA(b []byte) *A.
For using gob, you have to register each struct with it first before encoding / decoding, example:
func init() {
//where you should register your types, just once
gob.Register(A{})
gob.Register(B{})
}
func main() {
var (
buf bytes.Buffer
enc = gob.NewEncoder(&buf)
dec = gob.NewDecoder(&buf)
val = A{B{"name", "id"}}
r A
)
fmt.Println(enc.Encode(&val))
fmt.Println(dec.Decode(&r))
fmt.Printf("%#v", r)
}
JSON isn't able to encode the depth of type information that you can in Go, so you'll always get back the following basic types when unmarshalling:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
From go docs: http://golang.org/pkg/encoding/json/#Unmarshal
If you have that knowledge about the types you need, you can write some methods to construct the right variables on unmarshalling perhaps?
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}