Using Go's built-in JSON encoding and RSA certificates, encoding an RSA certificate to JSON and then decoding it again fails. Can anyone tell me why?
Sample code:
package main
import (
"math/big"
"bytes"
"crypto/rsa"
"crypto/x509"
"encoding/json"
"fmt"
)
func main() {
x := x509.Certificate{}
N := big.Int{}
N.SetString("22748727785943691703458259255009028279491017474470365445444638136335844588257736864763884308763893296520197853772438093808627310216929382637891614145169115712146316592081285259872752696564531634381437767246277672628285238806746417464073995314135364153811760424176263991818395775316575483628262670505480241278789196525640591586277923027755459881178666634831466031143685993230414451164180917857690755309066589767418739391025091379117132119476119702352443067398453637494721792111385747721853079909232828234315851102595340792201691113238210441101837352511126798300795766862744529103670862211990583319260851455953450220537", 10)
x.PublicKey = &rsa.PublicKey{N:&N, E:65537}
var y x509.Certificate
b := bytes.Buffer{}
err := json.NewEncoder(&b).Encode(x)
if err != nil { fmt.Println(err); return; }
err = json.NewDecoder(&b).Decode(&y)
if err != nil { fmt.Println(err) }
}
This fails in the Decode step with:
json: cannot unmarshal number 22748727785943691703458259255009028279491017474470365445444638136335844588257736864763884308763893296520197853772438093808627310216929382637891614145169115712146316592081285259872752696564531634381437767246277672628285238806746417464073995314135364153811760424176263991818395775316575483628262670505480241278789196525640591586277923027755459881178666634831466031143685993230414451164180917857690755309066589767418739391025091379117132119476119702352443067398453637494721792111385747721853079909232828234315851102595340792201691113238210441101837352511126798300795766862744529103670862211990583319260851455953450220537 into Go struct field Certificate.PublicKey of type float64
Playground example
The problem here is that the Certificate type declares PublicKey to be an interface{} and so the JSON unmarshaller has no hints about what type it should unmarshal the public key into. It tries to guess and gets it wrong; even if the large integer was correctly unmarshalled into a math/big.Int object, the PublicKey field would still be a map[string]interface{}, not a crypto/rsa.PublicKey object and so almost certainly not what you want.
To make this work, you would need to write a custom JSON marshaller / unmarshaller; probably encoding the whole certificate as PEM text would be correct way to store it in the JSON output. In my case, I can get away with just not encoding the certificate to JSON at all.
Related
Following this tutorial I'm trying to read a json file in Golang. It says there are two ways of doing that:
unmarshal the JSON using a set of predefined structs
or unmarshal the JSON using a map[string]interface{}
Since I'll probably have a lot of different json formats I prefer to interpret it on the fly. So I now have the following code:
package main
import (
"fmt"
"os"
"io/ioutil"
"encoding/json"
)
func main() {
// Open our jsonFile
jsonFile, err := os.Open("users.json")
// if we os.Open returns an error then handle it
if err != nil {
fmt.Println(err)
}
fmt.Println("Successfully Opened users.json")
// defer the closing of our jsonFile so that we can parse it later on
defer jsonFile.Close()
byteValue, _ := ioutil.ReadAll(jsonFile)
var result map[string]interface{}
json.Unmarshal([]byte(byteValue), &result)
fmt.Println(result["users"])
fmt.Printf("%T\n", result["users"])
}
This prints out:
Successfully Opened users.json
[map[type:Reader age:23 social:map[facebook:https://facebook.com twitter:https://twitter.com] name:Elliot] map[name:Fraser type:Author age:17 social:map[facebook:https://facebook.com twitter:https://twitter.com]]]
[]interface {}
At this point I don't understand how I can read the age of the first user (23). I tried some variations:
fmt.Println(result["users"][0])
fmt.Println(result["users"][0].age)
But apparently, type interface {} does not support indexing.
Is there a way that I can access the items in the json without defining the structure?
Probably you want
fmt.Println(result["users"].(map[string]interface{})["age"])
or
fmt.Println(result[0].(map[string]interface{})["age"])
As the JSON is a map of maps the type of the leaf nodes is interface{} and so has to be converted to map[string]interface{} in order to lookup a key
Defining a struct is much easier. My top tip for doing this is to use a website that converts JSON to a Go struct definition, like Json-To-Go
Currently, I try to parse JSON to map[string][]interface{}, but unmarshalling returns an error. According to (https://golang.org/pkg/encoding/json/), to unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
-[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
I wonder if golang is able to unmarshal map[string][]interface{}. The following is code snippet. I am new to Golang, thanks for help in advance.
// emailsStr looks like "{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}"
emailsRaw := make(map[string][]*entities.Email)
err := json.Unmarshal([]byte(emailsStr), &emailsRaw)
Error message:
&json.UnmarshalTypeError{Value:"number", Type:(*reflect.rtype)(0x151c7a0), Offset:44, Struct:"", Field:""}
The Go encoding/json package will only unmarshal dynamically to a map[string]interface{}. From there, you will need to use type assertions and casting to pull out the values you want, like so:
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
dynamic := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &dynamic)
firstEmail := dynamic["unknown.0"].([]interface{})[0].(map[string]interface{})["email_address"]
fmt.Println(firstEmail)
}
(https://play.golang.org/p/VEUEIwj3CIC)
Each time, Go's .(<type>) operator is used to assert and cast the dynamic value to a specific type. This particular code will panic if anything happens to be the wrong type at runtime, like if the contents of unknown.0 aren't an array of JSON objects.
The more idiomatic (and robust) way to do this in Go is to annotate a couple structs with json:"" tags and have encoding/json unmarshal into them. This avoids all the nasty brittle .([]interface{}) type casting:
type Email struct {
Email string `json:"email_address"`
}
type EmailsList struct {
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion int `json:"schemaVersion"`
Emails []Email `json:"unknown.0"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
emails := EmailsList{}
json.Unmarshal([]byte(jsonStr), &emails)
fmt.Printf("%+v\n", emails)
}
(https://play.golang.org/p/iS6e0_87P2J)
A better approach will be to use struct for main schema and then use an slice of email struct for fetching the data for email entities get the values from the same according to requirements. Please find the solution below :-
package main
import (
"fmt"
"encoding/json"
)
type Data struct{
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion float64 `json:"schemaVersion"`
EmailEntity []Email `json:"unknown.0"`
}
// Email struct
type Email struct{
EmailAddress string `json:"email_address"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
var dynamic Data
json.Unmarshal([]byte(jsonStr), &dynamic)
fmt.Printf("%#v", dynamic)
}
I am trying to parse a json from a third party software. It returns a json like this
{
"top1/dir1": "10",
"top1/dir2": "20",
"top1/dir3": "30",
"top2/diff_val1": "40"
}
JSONLint says this is a valid json. But I could not figure how I can parse this with golang.
The code I used to parse the json file above (to be clear I took the code from another stackoverflow post).
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
)
type mytype []map[string]string
func main() {
var data mytype
file, err := ioutil.ReadFile("t1.json")
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(file, &data)
if err != nil {
log.Fatal(err)
}
fmt.Println(data)
}
When I do a go run main.go, I get the below error
$ go run main.go
2016/06/19 22:53:57 json: cannot unmarshal object into Go value of type main.mytype
exit status 1
I did try to parse this format with another library - "github.com/Jeffail/gabs", but was unsuccessful. Since this is a valid json, I am pretty sure this can be parsed, but I am not sure how.
There is a Go package with methods for decoding JSON strings.
https://golang.org/pkg/encoding/json/#Unmarshal
Here is an example of usage:
package main
import (
"encoding/json"
"fmt"
)
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", animals)
}
EDIT: As pointed out by Malik, the type of the value whose pointer you pass is wrong. In this case, your type should be map[string]interface{} (preferably, because a JSON field might not store a string) or map[string]string instead of []map[string]string. The brackets at the beginning are wrong: such would be an array of JSON objects.
It's just that you have a small typo in your program. You've declared mytype as a slice of maps, rather than just a map.
Just change:
type mytype []map[string]string
To:
type mytype map[string]string
See https://play.golang.org/p/pZQl8jV5TC for an example.
Jonathan's answer provides a good example of decoding JSON, and links the relevant package. You don't provide much detail on what exactly is going wrong with your parsing, but if I had to take a guess I would say you're probably not creating an appropriate struct to contain the JSON once it is unmarshalled. Because Go is statically typed, it expects data to adhere to explicitly defined formats.
If you don't want to go to the trouble of defining structs, you could just use an empty interface object, which is sort of a catch all in Go. Simply declare a variable with the type []interface{}, and then pass it into the JSON unmarshal function. Hope this helps!
I want to decode a big set of data from a (static-schema) json file. The file contains exclusively numeric data, and keys are all integers. I know how to decode this json into a struct containing fields of map[string]int or map[string]float32 using json.Unmarshal. But I have no interest in string keys, I'd need to convert them to int somehow.
So what I'd like to know is:
Is there a way to achieve this, .ie getting a struct containing fields of map[int]float32 type directly from decoding,
Otherwise how to achieve this after decoding, in a memory efficient manner ?
Thanks
The JSON format only specifies key/value objects with string keys. Because of this, the encoding/json package only supports string keys as well.
The json/encoding documentation states:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
If you want to use encoding/json package and move it over to a map[int]float64, you can do the following (works with float32 as well):
package main
import (
"fmt"
"strconv"
)
func main() {
a := map[string]float64{"1":1, "2":4, "3":9, "5":25}
b := make(map[int]float64, len(a))
for k,v := range a {
if i, err := strconv.Atoi(k); err == nil {
b[i] = v
} else {
// A non integer key
}
}
fmt.Printf("%#v\n", b)
}
Playground
The encoding/json package includes an interface Unmarshaler which has a single method: UnmarshalJSON(data []byte) error.
If you're feeling brave you could implement that for the following:
type IntToFloat map[int]float32
func (itf *IntToFloat) UnmarshalJSON(data []byte) error {
if itf == nil {
return errors.New("Unmarshalling JSON for a null IntToFload")
}
// MAGIC Goes here.
return nil
}
EDIT
Note: http://golang.org/src/pkg/encoding/json/decode.go?s=2221:2269#L53 is where the std library version of Unmarshal comes from.
http://golang.org/pkg/encoding/json/#Unmarshaler is where the interface referenced above comes from.
I am using Sockjs with Go, but when the JavaScript client send json to the server it escapes it, and send's it as a []byte. I'm trying to figure out how to parse the json, so that i can read the data. but I get this error.
json: cannot unmarshal string into Go value of type main.Msg
How can I fix this? html.UnescapeString() has no effect.
val, err := session.ReadMessage()
if err != nil {
break
}
var msg Msg
err = json.Unmarshal(val, &msg)
fmt.Printf("%v", val)
fmt.Printf("%v", err)
type Msg struct {
Channel string
Name string
Msg string
}
//Output
"{\"channel\":\"buu\",\"name\":\"john\", \"msg\":\"doe\"}"
json: cannot unmarshal string into Go value of type main.Msg
You might want to use strconv.Unquote on your JSON string first :)
Here's an example, kindly provided by #gregghz:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Msg struct {
Channel string
Name string
Msg string
}
func main() {
var msg Msg
var val []byte = []byte(`"{\"channel\":\"buu\",\"name\":\"john\", \"msg\":\"doe\"}"`)
s, _ := strconv.Unquote(string(val))
err := json.Unmarshal([]byte(s), &msg)
fmt.Println(s)
fmt.Println(err)
fmt.Println(msg.Channel, msg.Name, msg.Msg)
}
You need to fix this in the code that is generating the JSON.
When it turns out formatted like that, it is being JSON encoded twice. Fix that code that is doing the generating so that it only happens once.
Here's some JavaScript that shows what's going on.
// Start with an object
var object = {"channel":"buu","name":"john", "msg":"doe"};
// serialize once
var json = JSON.stringify(object); // {"channel":"buu","name":"john","msg":"doe"}
// serialize twice
json = JSON.stringify(json); // "{\"channel\":\"buu\",\"name\":\"john\",\"msg\":\"doe\"}"
Sometimes, strconv.Unquote doesn't work.
Heres an example shows the problem and my solution.
(The playground link: https://play.golang.org/p/Ap0cdBgiA05)
Thanks for #Crazy Train's "encodes twice" idea, I just decoded it twice ...
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Wrapper struct {
Data string
}
type Msg struct {
Photo string
}
func main() {
var wrapper Wrapper
var original = `"{\"photo\":\"https:\/\/www.abc.net\/v\/t1.0-1\/p320x320\/123.jpg\"}"`
_, err := strconv.Unquote(original)
fmt.Println(err)
var val []byte = []byte("{\"data\":"+original+"}")
fmt.Println(string(val))
err = json.Unmarshal([]byte(val), &wrapper)
fmt.Println(wrapper.Data)
var msg Msg
err = json.Unmarshal([]byte(wrapper.Data), &msg)
fmt.Println(msg.Photo)
}
As Crazy Train pointed out, it appears that your input is doubly escaped, thus causing the issue. One way to fix this is to make sure that the function session.ReadMessasge() returns proper output that is escaped appropriately. However, if that's not possible, you can always do what x3ro suggested and use the golang function strconv.Unquote.
Here's a playground example of it in action:
http://play.golang.org/p/GTishI0rwe
The data shown in the problem is stringified for some purposes, in some cases you can even have \n in your string representing break of line in your json.
Let's understand the easiest way to unmarshal/deserialize this kind of data using the following example:
Next line shows the data you get from your sources and want to derserialize
stringifiedData := "{\r\n \"a\": \"b\",\r\n \"c\": \"d\"\r\n}"
Now, remove all new lines first
stringifiedData = strings.ReplaceAll(stringifiedData, "\n", "")
Then remove all the extra quotes that are present in your string
stringifiedData = strings.ReplaceAll(stringifiedData, "\\"", "\"")
Let's now convert the string into a byte array
dataInBytes := []byte(stringifiedData)
Before doing unmarshal, let's define structure of our data
jsonObject := &struct{
A string `json:"a"`
C string `json:"c"`
}
Now, you can dersialize your values into jsonObject
json.Unmarshal(dataInBytes, jsonObject)}
You got into infamous escaped string pitfall from JavaScript. Quite often people face (as I did) the same issue in Go, when serializing JSON string with json.Marshal, e.g.:
in := `{"firstName":"John","lastName":"Dow"}`
bytes, err := json.Marshal(in)
json.Marshal escapes double quotes, producing the same issue when you try to unmarshal bytes into struct.
If you faced the issue in Go, have a look at How To Correctly Serialize JSON String In Golang post which describes the issue in details with solution to it.