Fetching json data with struct in Go - json

I am facing a rather simple problem in Go since I am totally new to it. I want to fetch and print data from a REST api. the code I have written:
package main
import (
_"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
)
type Headers struct {
Headers HttpHeaders `json:"headers"`
}
type HttpHeaders struct {
Accept string
Accept_Encoding string `json:"Accept-Encoding"`
Accept_Language string `json:"Accept-Language"`
Connection string
Cookie string
Host string
Referer string
Upgrade_Insecure_Requests bool `json:"Upgrade-Insecure-Requests"`
User_Agent string `json:"User-Agent"`
}
func main() {
url := "http://httpbin.org/headers"
res, err := http.Get(url)
if err != nil {
panic(err)
}
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
var head Headers
err = json.Unmarshal(body, &head)
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", head)
}
After the adjustments I made, the response looks like:
{Headers:{Accept: Accept_Encoding:gzip Accept_Language: Connection:close Cookie: Host:httpbin.org Referer: Upgrade_Insecure_Requests:false User_Agent:}}
I have still some fields without a value and the value in Upgrade_Insecure_requests does not seem to match the one that is returned via the API.
2nd edit:
Removed the spaces in the tags. The response now still looks not good.
{Headers:{Accept: Accept_Encoding:gzip Accept_Language: Connection:close Cookie: Host:httpbin.org Referer: Upgrade_Insecure_Requests:false User_Agent:Go-http-client/1.1}}
The Upgrade_insecure_Requests is still 0 instead of 1 and the other fields are still blank.

Change your last line to
fmt.Printf("%+v\n", head)
to see all of your fields. As you can see all of strings are empty. It's because in Headers struct headers field is not exported (first letter must be capital) and no data will be unmarshaled into it. Also some fields name don't match the data in http://httpbin.org/headers.
For example change Accept_Encoding field in struct as follows:
Accept_Encoding string `json:"Accept-Encoding"`
to read from Accept-Encoding instead of Accept_Encoding.

A couple of issues here, first you must export the "HttpHeaders" field of the "Headers" struct, and second, you must tag the fields with JSON names if their JSON key isn't the same (case insensitive):
type Headers struct {
// The "Headers" member must have a capital "H" so that the
// JSON marshaler knows that it can be de/serialized.
Headers HttpHeaders
}
type HttpHeaders {
Accept string
// Since the JSON key is the same as the field name we don't need a tag.
Accept_Encoding string `json:"Accept-Encoding"`
// Here we need a tag since the member name cannot contain a dash.
// ...
}
Note that the "Headers" field of the "Headers" struct doesn't need a JSON tag because the unmarshaler ignores capitalization when matching JSON document keys to go struct field names. That is, the "Header" field would be populated by the JSON key "header", or "Header", or even "HEADER".

Related

JSON encoding / decoding and RSA certificates

Using Go's built-in JSON encoding and RSA certificates, encoding an RSA certificate to JSON and then decoding it again fails. Can anyone tell me why?
Sample code:
package main
import (
"math/big"
"bytes"
"crypto/rsa"
"crypto/x509"
"encoding/json"
"fmt"
)
func main() {
x := x509.Certificate{}
N := big.Int{}
N.SetString("22748727785943691703458259255009028279491017474470365445444638136335844588257736864763884308763893296520197853772438093808627310216929382637891614145169115712146316592081285259872752696564531634381437767246277672628285238806746417464073995314135364153811760424176263991818395775316575483628262670505480241278789196525640591586277923027755459881178666634831466031143685993230414451164180917857690755309066589767418739391025091379117132119476119702352443067398453637494721792111385747721853079909232828234315851102595340792201691113238210441101837352511126798300795766862744529103670862211990583319260851455953450220537", 10)
x.PublicKey = &rsa.PublicKey{N:&N, E:65537}
var y x509.Certificate
b := bytes.Buffer{}
err := json.NewEncoder(&b).Encode(x)
if err != nil { fmt.Println(err); return; }
err = json.NewDecoder(&b).Decode(&y)
if err != nil { fmt.Println(err) }
}
This fails in the Decode step with:
json: cannot unmarshal number 22748727785943691703458259255009028279491017474470365445444638136335844588257736864763884308763893296520197853772438093808627310216929382637891614145169115712146316592081285259872752696564531634381437767246277672628285238806746417464073995314135364153811760424176263991818395775316575483628262670505480241278789196525640591586277923027755459881178666634831466031143685993230414451164180917857690755309066589767418739391025091379117132119476119702352443067398453637494721792111385747721853079909232828234315851102595340792201691113238210441101837352511126798300795766862744529103670862211990583319260851455953450220537 into Go struct field Certificate.PublicKey of type float64
Playground example
The problem here is that the Certificate type declares PublicKey to be an interface{} and so the JSON unmarshaller has no hints about what type it should unmarshal the public key into. It tries to guess and gets it wrong; even if the large integer was correctly unmarshalled into a math/big.Int object, the PublicKey field would still be a map[string]interface{}, not a crypto/rsa.PublicKey object and so almost certainly not what you want.
To make this work, you would need to write a custom JSON marshaller / unmarshaller; probably encoding the whole certificate as PEM text would be correct way to store it in the JSON output. In my case, I can get away with just not encoding the certificate to JSON at all.

Making HTTP responses with JSON [duplicate]

This question already has an answer here:
json.Unmarshal returning blank structure
(1 answer)
Closed 3 years ago.
I am new to Go, and I am trying to practice with building a simple HTTP server. However I met some problems with JSON responses. I wrote following code, then try postman to send some JSON data. However, my postman always gets an empty response and the content-type is text/plain; charset=utf-8. Then I checked a sample in http://www.alexedwards.net/blog/golang-response-snippets#json. I copied and pasted the sample, and it was working well. But I cannot see any difference between mine and the sample. Can someone give some help?
package main
import (
"encoding/json"
"net/http"
)
type ResponseCommands struct {
key string
value bool
}
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":5432", nil)
}
func handler(rw http.ResponseWriter, req *http.Request) {
responseBody := ResponseCommands{"BackOff", false}
data, err := json.Marshal(responseBody)
if err != nil {
http.Error(rw, err.Error(), http.StatusInternalServerError)
return
}
rw.WriteHeader(200)
rw.Header().Set("Content-Type", "application/json")
rw.Write(data)
}
The main difference is that the variable in the struct are public (exported)
type Profile struct {
Name string
Hobbies []string
}
In your case, they are not (lowercase).
type ResponseCommands struct {
key string
value bool
}
See "Lowercase JSON key names with JSON Marshal in Go".
As VonC already answered correct. Just want to add that IDEA can help with such 'small' problems.
I'm using Gogland and it let me know that json tag cannot be applied to lowercase field.

How to JSON-decode lowercased names into my struct?

I'm starting to go crazy trying to get Go to decode this json request body. Here's a sample request:
curl -X POST -d "{\"username\":\"foo\", \"password\":\"bar\"}" http://localhost:3000/users
And here's my handler:
mux.HandleFunc("/users", func(rw http.ResponseWriter, req *http.Request) {
var body struct {
username string
password string
}
// buf := make([]byte, req.ContentLength)
// req.Body.Read(buf)
// fmt.Println(string(buf))
//
// The above commented out code will correctly print:
// {"username":"foo", "password":"bar"}
err := json.NewDecoder(req.Body).Decode(&body)
if err != nil {
rw.WriteHeader(http.StatusNotAcceptable)
return
}
fmt.Printf("%+v\n", body)
// prints -> {username: password:}
})
Like the comment suggests, I can verify that req.Body is indeed correct -- but for whatever reason, json.NewDecoder(req.Body).Decode(&body) never fills out the fields of body.
Any help would be greatly appreciated!
The problem is that the json decoder does not deal with private struct fields. The fields in your body structs are private.
Rewrite it like so and it will work:
var body struct {
Username string `json:"username"`
Password string `json:"password"`
}
basically the json:"username" is a way to tell the json decoder how to map the json name of the object to the struct name. In this instance, for decoding only, it is not necessary - the json decoder is smart enough to make the translation of the upper/lower case.
But if you use the object to encode json as well, you need it, or you'll have upper case field names in the resulting json.
You can use the json struct tags for a few more useful things, like omitting empty field from encoded json, or skipping fields entirely.
You can read more about the JSON struct tags in the documentation for json.Marshal: http://golang.org/pkg/encoding/json/#Marshal

How to access interface fields on json decode?

I have a json document and I'm using a client which decodes the document in an interface (instead of struct) as below:
var jsonR interface{}
err = json.Unmarshal(res, &jsonR)
How can I access the interface fields? I've read the go doc and blog but my head still can't get it. Their example seem to show only that you can decode the json in an interface but doesn't explain how its fields can be used.
I've tried to use a range loop but it seems the story ends when I reach a map[string]interface. The fields that I need seem to be in the interface.
for k, v := range jsonR {
if k == "topfield" {
fmt.Printf("k is %v, v is %v", k, v)
}
}
The value inside the interface depends on the json structure you're parsing. If you have a json dictionary, the dynamic type of jsonR will be: map[string]interface{}.
Here's an example.
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
a := []byte(`{"topfield": 123}`)
var v interface{}
if err := json.Unmarshal(a, &v); err != nil {
log.Fatalf("unmarshal failed: %s", err)
}
fmt.Printf("value is %v", v.(map[string]interface{})["topfield"])
}
Parsing json like this can be very difficult. The default type of a parse is map[string]interface{}. The Problem arises when you have another complex data structure within the main json(like another list or object). The best way to go about decoding json is defining a struct to hold data. Not only will the values be of the correct type but you can extract the specific data you actually care about.
Your struct can look like this:
type Top struct {
Topfield int `json:"topfield"`
}
which can be decoded like this:
a := []byte(`{"topfield": 123}`)
var data Top
json.Unmarshal(a, &data) //parse the json into data
now you can use regular struct operations to access your data like this:
value := data.Topfield
json which contains more complex data can also be easyli decoded. Perhaps you have a list in your data somewhere, you can use a struct like the following to extract it:
type Data struct {
States []string `json:"states"`
PopulationData []Country `json:"popdata"`
}
type Country struct {
Id int `json:"id"`
LastCensusPop int `json:"lcensuspopulation"`
Gdp float64 `json:"gdp"`
}
such a structure can not only parse list but also parse objects withing fields.

How to unmarshal an escaped JSON string

I am using Sockjs with Go, but when the JavaScript client send json to the server it escapes it, and send's it as a []byte. I'm trying to figure out how to parse the json, so that i can read the data. but I get this error.
json: cannot unmarshal string into Go value of type main.Msg
How can I fix this? html.UnescapeString() has no effect.
val, err := session.ReadMessage()
if err != nil {
break
}
var msg Msg
err = json.Unmarshal(val, &msg)
fmt.Printf("%v", val)
fmt.Printf("%v", err)
type Msg struct {
Channel string
Name string
Msg string
}
//Output
"{\"channel\":\"buu\",\"name\":\"john\", \"msg\":\"doe\"}"
json: cannot unmarshal string into Go value of type main.Msg
You might want to use strconv.Unquote on your JSON string first :)
Here's an example, kindly provided by #gregghz:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Msg struct {
Channel string
Name string
Msg string
}
func main() {
var msg Msg
var val []byte = []byte(`"{\"channel\":\"buu\",\"name\":\"john\", \"msg\":\"doe\"}"`)
s, _ := strconv.Unquote(string(val))
err := json.Unmarshal([]byte(s), &msg)
fmt.Println(s)
fmt.Println(err)
fmt.Println(msg.Channel, msg.Name, msg.Msg)
}
You need to fix this in the code that is generating the JSON.
When it turns out formatted like that, it is being JSON encoded twice. Fix that code that is doing the generating so that it only happens once.
Here's some JavaScript that shows what's going on.
// Start with an object
var object = {"channel":"buu","name":"john", "msg":"doe"};
// serialize once
var json = JSON.stringify(object); // {"channel":"buu","name":"john","msg":"doe"}
// serialize twice
json = JSON.stringify(json); // "{\"channel\":\"buu\",\"name\":\"john\",\"msg\":\"doe\"}"
Sometimes, strconv.Unquote doesn't work.
Heres an example shows the problem and my solution.
(The playground link: https://play.golang.org/p/Ap0cdBgiA05)
Thanks for #Crazy Train's "encodes twice" idea, I just decoded it twice ...
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Wrapper struct {
Data string
}
type Msg struct {
Photo string
}
func main() {
var wrapper Wrapper
var original = `"{\"photo\":\"https:\/\/www.abc.net\/v\/t1.0-1\/p320x320\/123.jpg\"}"`
_, err := strconv.Unquote(original)
fmt.Println(err)
var val []byte = []byte("{\"data\":"+original+"}")
fmt.Println(string(val))
err = json.Unmarshal([]byte(val), &wrapper)
fmt.Println(wrapper.Data)
var msg Msg
err = json.Unmarshal([]byte(wrapper.Data), &msg)
fmt.Println(msg.Photo)
}
As Crazy Train pointed out, it appears that your input is doubly escaped, thus causing the issue. One way to fix this is to make sure that the function session.ReadMessasge() returns proper output that is escaped appropriately. However, if that's not possible, you can always do what x3ro suggested and use the golang function strconv.Unquote.
Here's a playground example of it in action:
http://play.golang.org/p/GTishI0rwe
The data shown in the problem is stringified for some purposes, in some cases you can even have \n in your string representing break of line in your json.
Let's understand the easiest way to unmarshal/deserialize this kind of data using the following example:
Next line shows the data you get from your sources and want to derserialize
stringifiedData := "{\r\n \"a\": \"b\",\r\n \"c\": \"d\"\r\n}"
Now, remove all new lines first
stringifiedData = strings.ReplaceAll(stringifiedData, "\n", "")
Then remove all the extra quotes that are present in your string
stringifiedData = strings.ReplaceAll(stringifiedData, "\\"", "\"")
Let's now convert the string into a byte array
dataInBytes := []byte(stringifiedData)
Before doing unmarshal, let's define structure of our data
jsonObject := &struct{
A string `json:"a"`
C string `json:"c"`
}
Now, you can dersialize your values into jsonObject
json.Unmarshal(dataInBytes, jsonObject)}
You got into infamous escaped string pitfall from JavaScript. Quite often people face (as I did) the same issue in Go, when serializing JSON string with json.Marshal, e.g.:
in := `{"firstName":"John","lastName":"Dow"}`
bytes, err := json.Marshal(in)
json.Marshal escapes double quotes, producing the same issue when you try to unmarshal bytes into struct.
If you faced the issue in Go, have a look at How To Correctly Serialize JSON String In Golang post which describes the issue in details with solution to it.