I'm new to Golang and am using the "Server" code here as a starting point: http://www.golang-book.com/13/index.htm#section7
I've attempted to use JSON instead of Gob decoding (since I am required to write the client in C#), and I'm sending the JSON TCP data client data in a separate script from the code below.
I'm stuck on the part where I'm actually receiving the JSON TCP data and storing it in a variable for it to be decoded. It looks like I can decode it with json.Unmarshal, but I can't find any examples where json.Unmarshal is being used to decode TCP data. I can only find examples where json.Unmarshal is being used to decode JSON strings.
My code is below:
package main
import (
"encoding/json"
"fmt"
"net"
)
type coordinate struct {
X float64 `json:"x"`
Y float64 `json:"y"`
Z float64 `json:"z"`
}
func server() {
// listen on a port
ln, err := net.Listen("tcp", ":9999")
if err != nil {
fmt.Println(err)
return
}
for {
// accept a connection
c, err := ln.Accept()
if err != nil {
fmt.Println(err)
continue
}
// handle the connection
go handleServerConnection(c)
}
}
func handleServerConnection(c net.Conn) {
// receive the message
var msg coordinate
Stuck on the line below. What could I set the rawJSON variable equal to?
err := json.Unmarshal([]byte(rawJSON), &msg)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Received", msg)
}
c.Close()
}
func main() {
go server()
//let the server goroutine run forever
var input string
fmt.Scanln(&input)
}
You can patch a json.Decoder directly to the connection:
func handleServerConnection(c net.Conn) {
// we create a decoder that reads directly from the socket
d := json.NewDecoder(c)
var msg coordinate
err := d.Decode(&msg)
fmt.Println(msg, err)
c.Close()
}
Related
I was playing with go recently and stuck with a runtime error, I can't explain. These are my working functions.
type User struct {
Browsers []string `json:"browsers"`
Name string `json:"name"`
Email string `json:"email"`
}
func asyncUserProcJson(wg *sync.WaitGroup, users *[]User, ch chan []byte) {
for buf := range ch {
var mu sync.Mutex
var user User
mu.Lock()
err := json.Unmarshal(buf, &user)
mu.Unlock()
if err != nil {
fmt.Println("json:", err)
wg.Done()
continue
}
*users = append(*users, user)
wg.Done()
}
}
func userProcJson(buf []byte) (User, error) {
var user User
err := json.Unmarshal(buf, &user)
if err != nil {
return User{}, err
}
return user, nil
}
If I do a common - non-concurrent aproach, its works as expected. But if, try to use channel to pass bytes to goroutine... it fails.
type AsyncUserProc func(*sync.WaitGroup, *[]User, chan []byte)
type UserProc func(buf []byte) (User, error)
type SearchParams struct {
out io.Writer
asyncUserProc AsyncUserProc
userProc UserProc
}
func (sp SearchParams) AsyncSearch() []User {
file, err := os.Open(filePath)
if err != nil {
log.Fatalln(err)
}
var Users = make([]User, 0, 1024)
var ch = make(chan []byte)
var wg sync.WaitGroup
go sp.asyncUserProcess(&wg, &Users, ch)
scanner := bufio.NewScanner(file)
for scanner.Scan() {
wg.Add(1)
ch <- scanner.Bytes()
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
close(ch)
wg.Wait()
return Users
}
func (sp SearchParams) Search() []User {
file, err := os.Open(filePath)
if err != nil {
log.Fatalln(err)
}
// json processor
var Users = make([]User, 0, 1024)
scanner := bufio.NewScanner(file)
for scanner.Scan() {
u, err := sp.userProcess(scanner.Bytes())
if err != nil {
log.Panicln(err)
continue
}
Users = append(Users, u)
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
return Users
}
Workflow is the next one:
filePath contains a JSON chunks (each on new line)
Open for reading.
Create a line scanner
(AsyncSearch)
Pass line to channel.
return value of the line from range (blocking operation)
pass to json.Unmarshal
troubles
(Search)
Pass line directly to userProc func
Enjoy result
I am getting a lot (different) errors.
a lot of json unmarshaling error.
index out of range
JSON decoder out of sync - data changing underfoot?
as description of last error:
// phasePanicMsg is used as a panic message when we end up with something that
// shouldn't happen. It can indicate a bug in the JSON decoder, or that
// something is editing the data slice while the decoder executes.
So here is a question: How the bytes slice is modified?
I thought it was blocking operation. What am I missing in language mechanics?
Example of the errors (different each run)
json: invalid character 'i' looking for beginning of value
json: invalid character ':' after top-level value
json: invalid character 'r' looking for beginning of value
panic: runtime error: index out of range
----
json: invalid character '.' after top-level value
json: invalid character 'K' looking for beginning of value
panic: JSON decoder out of sync - data changing underfoot?
Package bufio
import "bufio"
func (*Scanner) Bytes
func (s *Scanner) Bytes() []byte
Bytes returns the most recent token generated by a call to Scan. The
underlying array may point to data that will be overwritten by a
subsequent call to Scan. It does no allocation.
The underlying array may point to data that will be overwritten by a subsequent call to Scan.
I have the following type in Golang:
type Base64Data []byte
In order to support unmarshalling a base64 encoded string to this type, I did the following:
func (b *Base64Data) UnmarshalJSON(data []byte) error {
if len(data) == 0 {
return nil
}
content, err := base64.StdEncoding.DecodeString(string(data[1 : len(data)-1]))
if err != nil {
return err
}
*b = []byte(xml)
return nil
}
Now I also want to be able to marshal and unmarshal it to mongo database, using mgo Golang library.
The problem is that I already have documents there stored as base64 encoded string, so I have to maintain that.
I tried to do the following:
func (b Base64Data) GetBSON() (interface{}, error) {
return base64.StdEncoding.EncodeToString([]byte(b)), nil
}
func (b *Base64DecodedXml) SetBSON(raw bson.Raw) error {
var s string
var err error
if err = raw.Unmarshal(&s); err != nil {
return err
}
*b, err = base64.StdEncoding.DecodeString(s)
return err
}
So that after unmarshaling, the data is already decoded, so I need to encode it back, and return it as a string so it will be written to db as a string (and vice versa)
For that I implemented bson getter and setter, but it seems only the getter is working properly
JSON unmarshaling from base64 encoded string works, as well marshaling it to database. but unmarshling setter seems to not be called at all.
Can anyone suggest what I'm missing, so that I'll be able to properly hold the data decoded in memory, but encoded string type?
This is a test I tried to run:
b := struct {
Value shared.Base64Data `json:"value" bson:"value"`
}{}
s := `{"value": "PHJvb3Q+aGVsbG88L3Jvb3Q+"}`
require.NoError(t, json.Unmarshal([]byte(s), &b))
t.Logf("%v", string(b.Value))
b4, err := bson.Marshal(b)
require.NoError(t, err)
t.Logf("%v", string(b4))
require.NoError(t, bson.Unmarshal(b4, &b))
t.Logf("%v", string(b.Value))
You can't marshal any value with bson.Marshal(), only maps and struct values.
If you want to test it, pass a map, e.g. bson.M to bson.Marshal():
var x = Base64Data{0x01, 0x02, 0x03}
dd, err := bson.Marshal(bson.M{"data": x})
fmt.Println(string(dd), err)
Your code works as-is, and as you intend it to. Try to insert a wrapper value to verify it:
c := sess.DB("testdb").C("testcoll")
var x = Base64Data{0x01, 0x02, 0x03}
if err := c.Insert(bson.M{
"data": x,
}); err != nil {
panic(err)
}
This will save the data as a string, being the Base64 encoded form.
Of course if you want to load it back into a value of type Base64Data, you will also need to define the SetBSON(raw Raw) error method too (bson.Setter interface).
In the following example from Web Development with Go by Shiju Varghese, which is for implementing a HTTP server using a new MongoDB session for each HTTP request:
Why is json package's Decode method used in PostCategory function?
Why is json package's Marshal method used in GetCategories function?
At first I thought that Decode in PostCategory and Marshal in GetCategories are opposite to each other, but later I found that there is a Unmarshal method and maybe a Encode one in the json package. So I asked a question earlier.
Here is the program
package main
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
var session *mgo.Session
type (
Category struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Name string
Description string
}
DataStore struct {
session *mgo.Session
}
)
//Close mgo.Session
func (d *DataStore) Close() {
d.session.Close()
}
//Returns a collection from the database.
func (d *DataStore) C(name string) *mgo.Collection {
return d.session.DB("taskdb").C(name)
}
//Create a new DataStore object for each HTTP request
func NewDataStore() *DataStore {
ds := &DataStore{
session: session.Copy(),
}
return ds
}
//Insert a record
func PostCategory(w http.ResponseWriter, r *http.Request) {
var category Category
// Decode the incoming Category json
err := json.NewDecoder(r.Body).Decode(&category)
if err != nil {
panic(err)
}
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
//Insert record
err = c.Insert(&category)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusCreated)
}
//Read all records
func GetCategories(w http.ResponseWriter, r *http.Request) {
var categories []Category
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
iter := c.Find(nil).Iter()
result := Category{}
for iter.Next(&result) {
categories = append(categories, result)
}
w.Header().Set("Content-Type", "application/json")
j, err := json.Marshal(categories)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusOK)
w.Write(j)
}
func main() {
var err error
session, err = mgo.Dial("localhost")
if err != nil {
panic(err)
}
r := mux.NewRouter()
r.HandleFunc("/api/categories", GetCategories).Methods("GET")
r.HandleFunc("/api/categories", PostCategory).Methods("POST")
server := &http.Server{
Addr: ":8080",
Handler: r,
}
log.Println("Listening...")
server.ListenAndServe()
}
I think the main reason for using a json.NewDecoder here is to read directly from response's body (r.Body) here, since NewDecoder takes an io.Reader as an input.
You could have used json.Unmarshal but then you'd have to first read response body into a []byte and pass that value to json.Unmarshal. NewDecoder is more convenient here.
TL;DR — Marshal/Unmarshal take and return byte slices, while Encode/Decode do the same thing, but read the bytes from a stream such as a network connection (readers and writers).
The encoding/json package uses the Encoder and Decoder types to act on streams of data, that is, io.Reader's and io.Writer's. This means that you can take data directly from a network socket (or an HTTP body in this case which implements io.Reader) and transform it to JSON as the bytes come in. Doing it this way, we can go ahead and start processing that JSON as soon as any data is available but before we've received the whole document (on a slow network connection with a big document this could save us a lot of time, and for some streaming protocols with "infinitely sized" document streams this is absolutely necessary!)
Marshal and Unmarshal however operate on byte slices, which means that you have to have the entirety of the JSON document in memory before you can use them. In your example, the author uses Marshal because they already have a []byte slice so there's no point in constructing a buffer using the byte slice, then making an encoder that uses that buffer, then calling encode: Instead they can just let Marshal do that for them.
In reality, Marshal/Unmarshal are just convenience methods on top of Encoders and Decoders. If we look at the source for Unmarshal, we see that under the hood it's just constructing an encoder (or the internal representation of an encoder, but trust me, they're the same thing, if you want proof you can look at the Encode method source and see that it's also creating an encodeState) and then returning the output bytes:
func Marshal(v interface{}) ([]byte, error) {
e := &encodeState{}
err := e.marshal(v)
if err != nil {
return nil, err
}
return e.Bytes(), nil
}
I have a server that successfully opens a connection with a second server. The second server performs an action and I am trying to get it to reply to the first server with a JSON on the same connection.
package main
import (
"fmt"
"net"
"encoding/json"
)
type NewContainerJSON struct {
Action string `json:"Action"`
ContainerName string `json:"ContainerName"`
BaseServer string `json:"BaseServer"`
CMS string `json:"CMS"`
WebsiteName string `json:"WebsiteName"`
DBrootPWD string `json:"DBrootPWD"`
DBadminUname string `json:"DBadminUname"`
DBadminPWD string `json:"DBadminPWD"`
}
func main() {
service := "127.0.0.1:8081"
tcpAddr, err := net.ResolveTCPAddr("tcp", service)
checkError(err)
listener, err := net.ListenTCP("tcp", tcpAddr)
checkError(err)
conn, err := listener.Accept()
checkError(err)
decoder := json.NewDecoder(conn)
encoder := json.NewEncoder(conn)
var b NewContainerJSON
err = decoder.Decode(&b)
checkError(err)
fmt.Println(b.Action)
if b.Action == "createNew" {
fmt.Println("This works")
resp := []byte("And here's our repomse")
conn.Write(resp)
c := NewContainerJSON {
Action: "createdNewContainer",
ContainerName: "Test",
BaseServer: "Test",
CMS: "Test",
WebsiteName: "Test",
DBrootPWD: "Test",
DBadminUname: "Test",
DBadminPWD: "Test",
}
encoder := json.NewEncoder(conn)
if err := encoder.Encode(c); err != nil {
fmt.Println("encode.Encode error: ", err)
}
conn.Write(c)
}
}
func checkError(err error) {
if err != nil {
fmt.Println("An error occurred: ", err.Error())
}
}
I get following error on the line conn.Write(c)
cannot use c (type NewContainerJSON) as type []byte in argument to conn.Write
Two questions:
1: What exactly is this error saying? It seems to be complaining that 'c' cannot be used as a Byte when using the conn.Write function but shouldn't the json.Encoder convert the JSON to a format the conn.Write can use?
2: How exactly can I return a JSON back to the first server using the open connection?
The encoder writes the JSON encoding of c to conn on this line:
if err := encoder.Encode(c); err != nil {
That's all you need to do. Delete the call to conn.Write(c).
The error message is telling you that the value of c cannot be used as the argument to Write because of a type mismatch. A NewContainerJSON is not a []byte.
You first write a string to the connection by
resp := []byte("And here's our repomse")
conn.Write(resp)
This will make it error-prone on the client side. You'll need to read exactly the same amount of data before employ the json decoder on this connection.
If a connection is used for json communication, all the messages on this stream should be json.
So if you want send a message to notify, encode that message too:
encoder.Encode(string(resp))
I am trying to develop a REST service with net/http.
The service receives a JSON structure containing all the input parameters. I wonder if there is an easier and shorter way to implement the following:
func call(w http.ResponseWriter, r *http.Request) {
if err := r.ParseForm(); err != nil {
fmt.Printf("Error parsing request %s\n", err)
}
var buf []byte
buf = make([]byte, 256)
var n, err = r.Body.Read(buf)
var decoded map[string]interface{}
err = json.Unmarshal(buf[:n], &decoded)
if err != nil {
fmt.Printf("Error decoding json: %s\n", err)
}
var uid = decoded["uid"]
...
}
As you can see it requires quite a number of lines just to get to the extraction of the first parameter. Any ideas?
You don't need to call r.ParseForm if the body of the request will contain a JSON structure and you don't need any URL parameters.
You don't need the buffer either; you can use:
decoder := json.NewDecoder(r.Body)
And then:
error := decoder.Decode(decoded)
Putting it all together:
func call(w http.ResponseWriter, r *http.Request) {
values := make(map[string]interface{})
if error := json.NewDecoder(r.Body).Decode(&values); error != nil {
panic(error)
}
uid := values["uid"].(int)
}
It would be much nicer, though, if you could formally define the structure of the input that you're expecting in a struct type:
type UpdateUserInformationRequest struct {
UserId int `json:"uid"`
// other fields...
}
And use an instance of this struct instead of a more general map.