Why are json package's Decode and Marshal methods used here? - json

In the following example from Web Development with Go by Shiju Varghese, which is for implementing a HTTP server using a new MongoDB session for each HTTP request:
Why is json package's Decode method used in PostCategory function?
Why is json package's Marshal method used in GetCategories function?
At first I thought that Decode in PostCategory and Marshal in GetCategories are opposite to each other, but later I found that there is a Unmarshal method and maybe a Encode one in the json package. So I asked a question earlier.
Here is the program
package main
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
var session *mgo.Session
type (
Category struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Name string
Description string
}
DataStore struct {
session *mgo.Session
}
)
//Close mgo.Session
func (d *DataStore) Close() {
d.session.Close()
}
//Returns a collection from the database.
func (d *DataStore) C(name string) *mgo.Collection {
return d.session.DB("taskdb").C(name)
}
//Create a new DataStore object for each HTTP request
func NewDataStore() *DataStore {
ds := &DataStore{
session: session.Copy(),
}
return ds
}
//Insert a record
func PostCategory(w http.ResponseWriter, r *http.Request) {
var category Category
// Decode the incoming Category json
err := json.NewDecoder(r.Body).Decode(&category)
if err != nil {
panic(err)
}
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
//Insert record
err = c.Insert(&category)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusCreated)
}
//Read all records
func GetCategories(w http.ResponseWriter, r *http.Request) {
var categories []Category
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
iter := c.Find(nil).Iter()
result := Category{}
for iter.Next(&result) {
categories = append(categories, result)
}
w.Header().Set("Content-Type", "application/json")
j, err := json.Marshal(categories)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusOK)
w.Write(j)
}
func main() {
var err error
session, err = mgo.Dial("localhost")
if err != nil {
panic(err)
}
r := mux.NewRouter()
r.HandleFunc("/api/categories", GetCategories).Methods("GET")
r.HandleFunc("/api/categories", PostCategory).Methods("POST")
server := &http.Server{
Addr: ":8080",
Handler: r,
}
log.Println("Listening...")
server.ListenAndServe()
}

I think the main reason for using a json.NewDecoder here is to read directly from response's body (r.Body) here, since NewDecoder takes an io.Reader as an input.
You could have used json.Unmarshal but then you'd have to first read response body into a []byte and pass that value to json.Unmarshal. NewDecoder is more convenient here.

TL;DR — Marshal/Unmarshal take and return byte slices, while Encode/Decode do the same thing, but read the bytes from a stream such as a network connection (readers and writers).
The encoding/json package uses the Encoder and Decoder types to act on streams of data, that is, io.Reader's and io.Writer's. This means that you can take data directly from a network socket (or an HTTP body in this case which implements io.Reader) and transform it to JSON as the bytes come in. Doing it this way, we can go ahead and start processing that JSON as soon as any data is available but before we've received the whole document (on a slow network connection with a big document this could save us a lot of time, and for some streaming protocols with "infinitely sized" document streams this is absolutely necessary!)
Marshal and Unmarshal however operate on byte slices, which means that you have to have the entirety of the JSON document in memory before you can use them. In your example, the author uses Marshal because they already have a []byte slice so there's no point in constructing a buffer using the byte slice, then making an encoder that uses that buffer, then calling encode: Instead they can just let Marshal do that for them.
In reality, Marshal/Unmarshal are just convenience methods on top of Encoders and Decoders. If we look at the source for Unmarshal, we see that under the hood it's just constructing an encoder (or the internal representation of an encoder, but trust me, they're the same thing, if you want proof you can look at the Encode method source and see that it's also creating an encodeState) and then returning the output bytes:
func Marshal(v interface{}) ([]byte, error) {
e := &encodeState{}
err := e.marshal(v)
if err != nil {
return nil, err
}
return e.Bytes(), nil
}

Related

Converting YAML to JSON in Go

I have a config file in YAML format, which I am trying to output as JSON via an http API call. I am unmarshalling using gopkg.in/yaml.v2. Yaml can have non-string keys, which means that the yaml is unmarshalled as map[interface{}]interface{}, which is not supported by Go's JSON marshaller. Therefore I convert to map[string]interface{} before unmarshalling. But I still get: json: unsupported type: map[interface {}]interface" {}. I don't understand. The variable cfy is not map[interface{}]interface{}.
import (
"io/ioutil"
"net/http"
"encoding/json"
"gopkg.in/yaml.v2"
)
func GetConfig(w http.ResponseWriter, r *http.Request) {
cfy := make(map[interface{}]interface{})
f, err := ioutil.ReadFile("config/config.yml")
if err != nil {
// error handling
}
if err := yaml.Unmarshal(f, &cfy); err != nil {
// error handling
}
//convert to a type that json.Marshall can digest
cfj := make(map[string]interface{})
for key, value := range cfy {
switch key := key.(type) {
case string:
cfj[key] = value
}
}
j, err := json.Marshal(cfj)
if err != nil {
// errr handling. We get: "json: unsupported type: map[interface {}]interface" {}
}
w.Header().Set("content-type", "application/json")
w.Write(j)
}
Your solution only converts values at the "top" level. If a value is also a map (nested map), your solution does not convert those.
Also you only "copy" the values with string keys, the rest will be left out of the result map.
Here's a function that recursively converts nested maps:
func convert(m map[interface{}]interface{}) map[string]interface{} {
res := map[string]interface{}{}
for k, v := range m {
switch v2 := v.(type) {
case map[interface{}]interface{}:
res[fmt.Sprint(k)] = convert(v2)
default:
res[fmt.Sprint(k)] = v
}
}
return res
}
Testing it:
m := map[interface{}]interface{}{
1: "one",
"two": 2,
"three": map[interface{}]interface{}{
"3.1": 3.1,
},
}
m2 := convert(m)
data, err := json.Marshal(m2)
if err != nil {
panic(err)
}
fmt.Println(string(data))
Output (try it on the Go Playground):
{"1":"one","three":{"3.1":3.1},"two":2}
Some things to note:
To covert interface{} keys, I used fmt.Sprint() which will handle all types. The switch could have a dedicated string case for keys that are already string values to avoid calling fmt.Sprint(). This is solely for performance reasons, the result will be the same.
The above convert() function does not go into slices. So for example if the map contains a value which is a slice ([]interface{}) which may also contain maps, those will not be converted. For a full solution, see the lib below.
There is a lib github.com/icza/dyno which has an optimized, built-in support for this (disclosure: I'm the author). Using dyno, this is how it would look like:
var m map[interface{}]interface{} = ...
m2 := dyno.ConvertMapI2MapS(m)
dyno.ConvertMapI2MapS() also goes into and converts maps in []interface{} slices.
Also see possible duplicate: Convert yaml to json without struct

How to output golang http json body to website page using html/javascript/jquery

I've got a Golang Website where I want to display 'scores' from my UWP Game using SQLite's Mobile App Quickstart's API called SwaggerUI. I am getting the scores by doing a HTTP GET request. The problem is that the scores output to the Golang console in JSON Format. I want to display the scores onto the actual website. How could I call my golang function in the Frontend in order to do this? The frontend is written in HTML/Javascript/JQuery.
This is my Golang Function that does the HTTP Request to SwaggerUI and outputs to the Golang Console:
func scoresPage(res http.ResponseWriter, req *http.Request) {
//Connecting to SwaggerUI API to get Scores from Azure for UWP Application
req, err := http.NewRequest("GET", os.ExpandEnv("https://brainworksappservice.azurewebsites.net/tables/TodoItem?$select=score"), nil)
if err != nil {
log.Fatal(err)
}
//You have to specify these headers
req.Header.Set("Accept", "application/json")
//If you do not specify what version your API is, you cannot receive the JSON
req.Header.Set("Zumo-Api-Version", "2.0.0")
//Do the request
resp, err := http.DefaultClient.Do(req)
//Error if the request cannot be done
if err != nil {
log.Fatal(err)
}
//You need to close the Body everytime, as if you don't you could leak information
defer resp.Body.Close()
//Read all of the information from the body
body, err := ioutil.ReadAll(resp.Body)
//Error if the info cannot be read
if err != nil {
log.Fatal(err)
}
//Write the JSON to the standard output (the Console)
_, err = os.Stdout.Write(body)
//Error if the info cannot be output to the console
if err != nil {
log.Fatal(err)
}
http.ServeFile(res, req, "Scores.html")
} `
This is the main Function which serves up the website and handles the scores page:
func main() {
http.HandleFunc("/scores", scoresPage)
//serve on the port 8000 forever
http.ListenAndServe(":8000", nil)
}
Assuming, that you don't want to dump the json as is onto you page but instead format it in some way with html and css, then you could first decode the returned body into a slice of structs that mirror the structure of your json. For example like this:
type Score struct {
Id string `json:"id"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
Version string `json:"version"`
Deleted bool `json:"deleted"`
Text string `json:"text"`
Complete bool `json:"complete"`
Score string `json:"score"`
}
scores := []*Score{}
if err := json.Unmarshal(body, &scores); err != nil {
panic(err)
}
fmt.Println(scores[0])
https://play.golang.org/p/m_ySdZulqy
After you've decoded the json you can use Go's template package to loop over the scores and format them as you wish. While you should use the html/template package for rendering html you should check out text/template for the documentation on how to actually program the templates, they have the same interface.
Here's a quick example: https://play.golang.org/p/EYfV-TzoA0
In that example I'm using the template package to parse a string (scoresPage) and output the result to stdout, but you can just as easily parse your Scores.html file with ParseFiles and return the output in the http response by passing the res http.ResponseWriter instead of os.Stdout as the first argument to template.Execute.

Encode Set data structure in Golang

I have a Set data structure implemented in Go with the basic operations like Add, Remove, Difference, Union. I am trying to send a http request using the json encoder to encode the request body which contains the object of the form map[string]Set. The Set data structure is defined below:
type Set map[interface{}]struct{}
func NewSet() Set {
set := make(Set)
return set
}
The encoder looks like this:
func (req *Request) BodyContentInJson (val interface{}) error {
buf := bytes.NewBuffer(nil)
enc := json.NewEncoder(buf)
if err := enc.Encode(val); err != nil {
return err
}
req.Obj = val
req.Body = buf
req.BodySize = int64(buf.Len())
return nil
}
This code fails at
if err := enc.Encode(val); err != nil {
return err
}
giving an error:{"errors":["json: unsupported type: Set"]}. Also, the type of val is map[string]interface{}when I debugged it.
How could I possibly encode and marshal/unmarshal JSON content here using the Go's encoder ?
You could write your own UnmarshalJSON method on the *Set type which would then be used by the json.Encoder to encode the json data into the Set. Here's a simple example https://play.golang.org/p/kx1E-jDu5e.
By the way, the reason you're getting the error is because a map key of type interface{} is not supported by the encoding/json package. (https://github.com/golang/go/blob/master/src/encoding/json/encode.go#L697-L699)

Golang server, how to receive TCP JSON packet?

I'm new to Golang and am using the "Server" code here as a starting point: http://www.golang-book.com/13/index.htm#section7
I've attempted to use JSON instead of Gob decoding (since I am required to write the client in C#), and I'm sending the JSON TCP data client data in a separate script from the code below.
I'm stuck on the part where I'm actually receiving the JSON TCP data and storing it in a variable for it to be decoded. It looks like I can decode it with json.Unmarshal, but I can't find any examples where json.Unmarshal is being used to decode TCP data. I can only find examples where json.Unmarshal is being used to decode JSON strings.
My code is below:
package main
import (
"encoding/json"
"fmt"
"net"
)
type coordinate struct {
X float64 `json:"x"`
Y float64 `json:"y"`
Z float64 `json:"z"`
}
func server() {
// listen on a port
ln, err := net.Listen("tcp", ":9999")
if err != nil {
fmt.Println(err)
return
}
for {
// accept a connection
c, err := ln.Accept()
if err != nil {
fmt.Println(err)
continue
}
// handle the connection
go handleServerConnection(c)
}
}
func handleServerConnection(c net.Conn) {
// receive the message
var msg coordinate
Stuck on the line below. What could I set the rawJSON variable equal to?
err := json.Unmarshal([]byte(rawJSON), &msg)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Received", msg)
}
c.Close()
}
func main() {
go server()
//let the server goroutine run forever
var input string
fmt.Scanln(&input)
}
You can patch a json.Decoder directly to the connection:
func handleServerConnection(c net.Conn) {
// we create a decoder that reads directly from the socket
d := json.NewDecoder(c)
var msg coordinate
err := d.Decode(&msg)
fmt.Println(msg, err)
c.Close()
}

Go lang decode io.read JSON v.s. unmarshal giving different results

I have a handler like this:
type Location struct {
Lat float32 `json:"lat"`
Lon float32 `json:"lon"`
}
func handleJSONLocation(res http.ResponseWriter, req *http.Request) {
b := new(Location)
var bb Location;
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
json.Unmarshal(buf.Bytes(), &bb)
json.NewDecoder(req.Body).Decode(b)
log.Printf("%s %f,%f %f,%f", buf.String(), b.Lat, b.Lon, bb.Lat, bb.Lon);
data, _ := json.Marshal("{'location':'saved'}")
res.Header().Set("Content-Type", "application/json; charset=utf-8")
res.Write(data)
}
I get
2014/04/25 22:05:55 {"lat":42.9635013,"lon":-81.22387210000001} 0.000000,0.000000 42.963501,-81.223869
Why doesn't b object get decoded into?
Regards
When you run the following code, the entire body of the HTTP request is read into buf:
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
At this point, further reads from req.Body will result in an EOF error. So when you attempt to decode into b, you are getting an error, which would be visible if you checked the return value with something like:
if err := json.NewDecoder(req.Body).Decode(b); err != nil {
log.Println("Could not decode body:", err)
}
It would also be wise to check the error on the other function calls you make.
Since you've already read the request body into a memory buffer though, you could create a new bytes.Buffer to pass to APIs that expect an io.Reader (such as json.NewDecoder).