Replying with a JSON on an open TCP socket - json

I have a server that successfully opens a connection with a second server. The second server performs an action and I am trying to get it to reply to the first server with a JSON on the same connection.
package main
import (
"fmt"
"net"
"encoding/json"
)
type NewContainerJSON struct {
Action string `json:"Action"`
ContainerName string `json:"ContainerName"`
BaseServer string `json:"BaseServer"`
CMS string `json:"CMS"`
WebsiteName string `json:"WebsiteName"`
DBrootPWD string `json:"DBrootPWD"`
DBadminUname string `json:"DBadminUname"`
DBadminPWD string `json:"DBadminPWD"`
}
func main() {
service := "127.0.0.1:8081"
tcpAddr, err := net.ResolveTCPAddr("tcp", service)
checkError(err)
listener, err := net.ListenTCP("tcp", tcpAddr)
checkError(err)
conn, err := listener.Accept()
checkError(err)
decoder := json.NewDecoder(conn)
encoder := json.NewEncoder(conn)
var b NewContainerJSON
err = decoder.Decode(&b)
checkError(err)
fmt.Println(b.Action)
if b.Action == "createNew" {
fmt.Println("This works")
resp := []byte("And here's our repomse")
conn.Write(resp)
c := NewContainerJSON {
Action: "createdNewContainer",
ContainerName: "Test",
BaseServer: "Test",
CMS: "Test",
WebsiteName: "Test",
DBrootPWD: "Test",
DBadminUname: "Test",
DBadminPWD: "Test",
}
encoder := json.NewEncoder(conn)
if err := encoder.Encode(c); err != nil {
fmt.Println("encode.Encode error: ", err)
}
conn.Write(c)
}
}
func checkError(err error) {
if err != nil {
fmt.Println("An error occurred: ", err.Error())
}
}
I get following error on the line conn.Write(c)
cannot use c (type NewContainerJSON) as type []byte in argument to conn.Write
Two questions:
1: What exactly is this error saying? It seems to be complaining that 'c' cannot be used as a Byte when using the conn.Write function but shouldn't the json.Encoder convert the JSON to a format the conn.Write can use?
2: How exactly can I return a JSON back to the first server using the open connection?

The encoder writes the JSON encoding of c to conn on this line:
if err := encoder.Encode(c); err != nil {
That's all you need to do. Delete the call to conn.Write(c).
The error message is telling you that the value of c cannot be used as the argument to Write because of a type mismatch. A NewContainerJSON is not a []byte.

You first write a string to the connection by
resp := []byte("And here's our repomse")
conn.Write(resp)
This will make it error-prone on the client side. You'll need to read exactly the same amount of data before employ the json decoder on this connection.
If a connection is used for json communication, all the messages on this stream should be json.
So if you want send a message to notify, encode that message too:
encoder.Encode(string(resp))

Related

Is there any method similar to Golang's Scan() method (used for SQL) for Elasticsearch?

I am new to both Go and ES. I want to convert the following piece of code:
query = `SELECT roll_no, name, school FROM students where roll_no = $1`
err = db.QueryRowContext(ctx, query, roll_no).Scan(&Student.ID, &Student.Name, &Student.School)
into something like the following Elasticsearch query:
str = fmt.Sprintf(`{"query": { "match": { "roll_no" : %d } } }`, roll_no)
b := []byte(str)
// calls retrieve method, which is shown below
I am connecting to ES using HTTP calls, but the following code is showing http: panic serving [::1]:5574: EOF error while parsing.
func retrieve(url string, b []byte) ([]byte, error) {
request, _ := http.NewRequest("GET", url, bytes.NewBuffer(b))
request.Header.Set("Content-Type", "application/json; charset=UTF-8")
client := &http.Client{}
response, error := client.Do(request)
if error != nil {
panic(error)
}
defer response.Body.Close()
body, _ := ioutil.ReadAll(response.Body)
s := Student{}
error = json.NewDecoder(response.Body).Decode(&s)
if error != nil {
panic(error)
}
fmt.Printf("Student: %v", s)
return body, error
}
Is there anyway I can store it into an object by parsing?
How can I use Golang's Scan() method (normally used for SQL) in Elasticsearch?
Not at all.
Package database/sql needs a database driver and there isn't one for ES.

JSON decoding error in Go: "invalid character '\\' looking for beginning of object key string"

I'm using the vpp package of MicroMDM but noticed that initializing a VPP client, which was working before, has stopped working. The error I get occurs in the client's GetClientContext() method (https://github.com/micromdm/micromdm/blob/ea5c0a3865e87f5de04a58dc2c67b5fa6f6fd7de/vpp/clientconfigsrv.go#L99-L117), which I've modified with a fmt.Println() call to print the context:
// GetClientContext gets the values that determine which mdm server is associated with a VPP account token
func (c *Client) GetClientContext() (*ClientContext, error) {
// Get the ClientConfigSrv info
clientConfigSrv, err := c.GetClientConfigSrv()
if err != nil {
return nil, errors.Wrap(err, "get ClientContext request")
}
// Get the ClientContext string
var context = clientConfigSrv.ClientContext
fmt.Println(context)
// Convert the string to a ClientContext type
var clientContext ClientContext
err = json.NewDecoder(strings.NewReader(context)).Decode(&clientContext)
if err != nil {
return nil, errors.Wrap(err, "decode ClientContext")
}
return &clientContext, nil
}
where ClientContext is the following struct (https://github.com/micromdm/micromdm/blob/ea5c0a3865e87f5de04a58dc2c67b5fa6f6fd7de/vpp/clientconfigsrv.go#L10-L14):
// Contains information that associates your particular mdm server to a VPP account token
type ClientContext struct {
HostName string `json:"hostname"`
GUID string `json:"guid"`
}
when I run a script that invokes vpp.NewClient() (which, in turn, invokes this method), I see the following output:
> go run main.go getVPPAssets
INFO[0000] Using config file: .vpp-client.env
{\"hostname\":\"https://your.server.com\",\"guid\":\"7c66e635-ef07-4e26-86a3-c953e4e8f16d\"}
FATA[0000] Failed to initialize VPP client: GetClientContext request: decode ClientContext: invalid character '\\' looking for beginning of object key string
exit status 1
Note that the line with
{\"hostname\":\"https://your.server.com\",\"guid\":\"7c66e635-ef07-4e26-86a3-c953e4e8f16d\"}
is produced by the fmt.Println() statement I added. I've tried to reproduce this error with the following minimal example:
package main
import (
"encoding/json"
"fmt"
"strings"
"github.com/sirupsen/logrus"
)
type ClientContext struct {
HostName string `json:"hostname"`
GUID string `json:"guid"`
}
func main() {
context := "{\"hostname\":\"https://your.server.com\",\"guid\":\"7c66e635-ef07-4e26-86a3-c953e4e8f16d\"}"
var clientContext ClientContext
if err := json.NewDecoder(strings.NewReader(context)).Decode(&clientContext); err != nil {
logrus.Fatalf("decode ClientContext: %v", err)
}
fmt.Printf("%+v\n", clientContext)
}
However, this example decodes the clientConfigSrv.ClientContext string into a ClientContext correctly.
I believe there is something about the escaping of quotes that has changed in the response returned by Apple's VPP service and is causing this decoder to choke; any ideas how to fix this error?

Parsing JSON concurrently - panic of runtime error (decoding related)

I was playing with go recently and stuck with a runtime error, I can't explain. These are my working functions.
type User struct {
Browsers []string `json:"browsers"`
Name string `json:"name"`
Email string `json:"email"`
}
func asyncUserProcJson(wg *sync.WaitGroup, users *[]User, ch chan []byte) {
for buf := range ch {
var mu sync.Mutex
var user User
mu.Lock()
err := json.Unmarshal(buf, &user)
mu.Unlock()
if err != nil {
fmt.Println("json:", err)
wg.Done()
continue
}
*users = append(*users, user)
wg.Done()
}
}
func userProcJson(buf []byte) (User, error) {
var user User
err := json.Unmarshal(buf, &user)
if err != nil {
return User{}, err
}
return user, nil
}
If I do a common - non-concurrent aproach, its works as expected. But if, try to use channel to pass bytes to goroutine... it fails.
type AsyncUserProc func(*sync.WaitGroup, *[]User, chan []byte)
type UserProc func(buf []byte) (User, error)
type SearchParams struct {
out io.Writer
asyncUserProc AsyncUserProc
userProc UserProc
}
func (sp SearchParams) AsyncSearch() []User {
file, err := os.Open(filePath)
if err != nil {
log.Fatalln(err)
}
var Users = make([]User, 0, 1024)
var ch = make(chan []byte)
var wg sync.WaitGroup
go sp.asyncUserProcess(&wg, &Users, ch)
scanner := bufio.NewScanner(file)
for scanner.Scan() {
wg.Add(1)
ch <- scanner.Bytes()
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
close(ch)
wg.Wait()
return Users
}
func (sp SearchParams) Search() []User {
file, err := os.Open(filePath)
if err != nil {
log.Fatalln(err)
}
// json processor
var Users = make([]User, 0, 1024)
scanner := bufio.NewScanner(file)
for scanner.Scan() {
u, err := sp.userProcess(scanner.Bytes())
if err != nil {
log.Panicln(err)
continue
}
Users = append(Users, u)
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
return Users
}
Workflow is the next one:
filePath contains a JSON chunks (each on new line)
Open for reading.
Create a line scanner
(AsyncSearch)
Pass line to channel.
return value of the line from range (blocking operation)
pass to json.Unmarshal
troubles
(Search)
Pass line directly to userProc func
Enjoy result
I am getting a lot (different) errors.
a lot of json unmarshaling error.
index out of range
JSON decoder out of sync - data changing underfoot?
as description of last error:
// phasePanicMsg is used as a panic message when we end up with something that
// shouldn't happen. It can indicate a bug in the JSON decoder, or that
// something is editing the data slice while the decoder executes.
So here is a question: How the bytes slice is modified?
I thought it was blocking operation. What am I missing in language mechanics?
Example of the errors (different each run)
json: invalid character 'i' looking for beginning of value
json: invalid character ':' after top-level value
json: invalid character 'r' looking for beginning of value
panic: runtime error: index out of range
----
json: invalid character '.' after top-level value
json: invalid character 'K' looking for beginning of value
panic: JSON decoder out of sync - data changing underfoot?
Package bufio
import "bufio"
func (*Scanner) Bytes
func (s *Scanner) Bytes() []byte
Bytes returns the most recent token generated by a call to Scan. The
underlying array may point to data that will be overwritten by a
subsequent call to Scan. It does no allocation.
The underlying array may point to data that will be overwritten by a subsequent call to Scan.

Why are json package's Decode and Marshal methods used here?

In the following example from Web Development with Go by Shiju Varghese, which is for implementing a HTTP server using a new MongoDB session for each HTTP request:
Why is json package's Decode method used in PostCategory function?
Why is json package's Marshal method used in GetCategories function?
At first I thought that Decode in PostCategory and Marshal in GetCategories are opposite to each other, but later I found that there is a Unmarshal method and maybe a Encode one in the json package. So I asked a question earlier.
Here is the program
package main
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
var session *mgo.Session
type (
Category struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Name string
Description string
}
DataStore struct {
session *mgo.Session
}
)
//Close mgo.Session
func (d *DataStore) Close() {
d.session.Close()
}
//Returns a collection from the database.
func (d *DataStore) C(name string) *mgo.Collection {
return d.session.DB("taskdb").C(name)
}
//Create a new DataStore object for each HTTP request
func NewDataStore() *DataStore {
ds := &DataStore{
session: session.Copy(),
}
return ds
}
//Insert a record
func PostCategory(w http.ResponseWriter, r *http.Request) {
var category Category
// Decode the incoming Category json
err := json.NewDecoder(r.Body).Decode(&category)
if err != nil {
panic(err)
}
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
//Insert record
err = c.Insert(&category)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusCreated)
}
//Read all records
func GetCategories(w http.ResponseWriter, r *http.Request) {
var categories []Category
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
iter := c.Find(nil).Iter()
result := Category{}
for iter.Next(&result) {
categories = append(categories, result)
}
w.Header().Set("Content-Type", "application/json")
j, err := json.Marshal(categories)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusOK)
w.Write(j)
}
func main() {
var err error
session, err = mgo.Dial("localhost")
if err != nil {
panic(err)
}
r := mux.NewRouter()
r.HandleFunc("/api/categories", GetCategories).Methods("GET")
r.HandleFunc("/api/categories", PostCategory).Methods("POST")
server := &http.Server{
Addr: ":8080",
Handler: r,
}
log.Println("Listening...")
server.ListenAndServe()
}
I think the main reason for using a json.NewDecoder here is to read directly from response's body (r.Body) here, since NewDecoder takes an io.Reader as an input.
You could have used json.Unmarshal but then you'd have to first read response body into a []byte and pass that value to json.Unmarshal. NewDecoder is more convenient here.
TL;DR — Marshal/Unmarshal take and return byte slices, while Encode/Decode do the same thing, but read the bytes from a stream such as a network connection (readers and writers).
The encoding/json package uses the Encoder and Decoder types to act on streams of data, that is, io.Reader's and io.Writer's. This means that you can take data directly from a network socket (or an HTTP body in this case which implements io.Reader) and transform it to JSON as the bytes come in. Doing it this way, we can go ahead and start processing that JSON as soon as any data is available but before we've received the whole document (on a slow network connection with a big document this could save us a lot of time, and for some streaming protocols with "infinitely sized" document streams this is absolutely necessary!)
Marshal and Unmarshal however operate on byte slices, which means that you have to have the entirety of the JSON document in memory before you can use them. In your example, the author uses Marshal because they already have a []byte slice so there's no point in constructing a buffer using the byte slice, then making an encoder that uses that buffer, then calling encode: Instead they can just let Marshal do that for them.
In reality, Marshal/Unmarshal are just convenience methods on top of Encoders and Decoders. If we look at the source for Unmarshal, we see that under the hood it's just constructing an encoder (or the internal representation of an encoder, but trust me, they're the same thing, if you want proof you can look at the Encode method source and see that it's also creating an encodeState) and then returning the output bytes:
func Marshal(v interface{}) ([]byte, error) {
e := &encodeState{}
err := e.marshal(v)
if err != nil {
return nil, err
}
return e.Bytes(), nil
}

Golang server, how to receive TCP JSON packet?

I'm new to Golang and am using the "Server" code here as a starting point: http://www.golang-book.com/13/index.htm#section7
I've attempted to use JSON instead of Gob decoding (since I am required to write the client in C#), and I'm sending the JSON TCP data client data in a separate script from the code below.
I'm stuck on the part where I'm actually receiving the JSON TCP data and storing it in a variable for it to be decoded. It looks like I can decode it with json.Unmarshal, but I can't find any examples where json.Unmarshal is being used to decode TCP data. I can only find examples where json.Unmarshal is being used to decode JSON strings.
My code is below:
package main
import (
"encoding/json"
"fmt"
"net"
)
type coordinate struct {
X float64 `json:"x"`
Y float64 `json:"y"`
Z float64 `json:"z"`
}
func server() {
// listen on a port
ln, err := net.Listen("tcp", ":9999")
if err != nil {
fmt.Println(err)
return
}
for {
// accept a connection
c, err := ln.Accept()
if err != nil {
fmt.Println(err)
continue
}
// handle the connection
go handleServerConnection(c)
}
}
func handleServerConnection(c net.Conn) {
// receive the message
var msg coordinate
Stuck on the line below. What could I set the rawJSON variable equal to?
err := json.Unmarshal([]byte(rawJSON), &msg)
if err != nil {
fmt.Println(err)
} else {
fmt.Println("Received", msg)
}
c.Close()
}
func main() {
go server()
//let the server goroutine run forever
var input string
fmt.Scanln(&input)
}
You can patch a json.Decoder directly to the connection:
func handleServerConnection(c net.Conn) {
// we create a decoder that reads directly from the socket
d := json.NewDecoder(c)
var msg coordinate
err := d.Decode(&msg)
fmt.Println(msg, err)
c.Close()
}