How to match request to response while proxying packets? - mysql

I'm working on MySQL proxy with browser based GUI.
Core of an app looks like:
//Trying to connect to server
rightConn, err := net.Dial("tcp", l.rightAddr)
if err != nil {
return
}
defer rightConn.Close()
wg.Add(2)
//Start passing packets from client to server
go func() {
defer wg.Done()
l.Pipe(leftConn, rightConn)
}()
//Start passing packets from server to client
go func() {
defer wg.Done()
l.Pipe(rightConn, leftConn)
}()
wg.Wait()
And here's definition of Pipe:
func (l *Lottip) Pipe(right, left net.Conn) {
for {
pkt, err := mysql.ReadPacket(right)
if err != nil {
break
}
if _, err = mysql.WritePacket(pkt, left); err != nil {
break
}
}
}
The reason i'm using my custom proxy function instead of
go io.Copy(left, right)
go io.Copy(right, left)
is i have to parse each MySQL's Request/Response packet and prepare it for further processing.
As we all know MySQL client can send a lot of queries and get responses in random order.
The problem is i cannot get how to match request to it's response. While i'm piping using 2 goroutines(1 per direction) i can easily read server response but i don't know to what request it responding.
I've examined request/response between MySQL client and server via Wireshark and found no mention for packet_id or query_id or similar

Related

Go concurrent TCP server hangs when JSON is sent as the response but works with plain string

I am trying to implement a concurrent TCP server in Go and found this great explanatory article in linode where it clearly explains with a sample code. The sample code snippets for the client and server and included below.
Concurrent TCP server where for each TCP client a new go-routine is created.
package main
import (
"bufio"
"fmt"
"net"
"os"
"strconv"
"strings"
)
var count = 0
func handleConnection(c net.Conn) {
fmt.Print(".")
for {
netData, err := bufio.NewReader(c).ReadString('\n')
if err != nil {
fmt.Println(err)
return
}
temp := strings.TrimSpace(string(netData))
if temp == "STOP" {
break
}
fmt.Println(temp)
counter := strconv.Itoa(count) + "\n"
c.Write([]byte(string(counter)))
}
c.Close()
}
func main() {
arguments := os.Args
if len(arguments) == 1 {
fmt.Println("Please provide a port number!")
return
}
PORT := ":" + arguments[1]
l, err := net.Listen("tcp4", PORT)
if err != nil {
fmt.Println(err)
return
}
defer l.Close()
for {
c, err := l.Accept()
if err != nil {
fmt.Println(err)
return
}
go handleConnection(c)
count++
}
}
TCP client code snippet
package main
import (
"bufio"
"fmt"
"net"
"os"
"strings"
)
func main() {
arguments := os.Args
if len(arguments) == 1 {
fmt.Println("Please provide host:port.")
return
}
CONNECT := arguments[1]
c, err := net.Dial("tcp", CONNECT)
if err != nil {
fmt.Println(err)
return
}
for {
reader := bufio.NewReader(os.Stdin)
fmt.Print(">> ")
text, _ := reader.ReadString('\n')
fmt.Fprintf(c, text+"\n")
message, _ := bufio.NewReader(c).ReadString('\n')
fmt.Print("->: " + message)
if strings.TrimSpace(string(text)) == "STOP" {
fmt.Println("TCP client exiting...")
return
}
}
}
The above concurrent TCP server and client works without any issue. The issue comes when I change the TCP server to send a JSON response instead of the text response. When I change the line:
counter := strconv.Itoa(count) + "\n"
c.Write([]byte(string(counter)))
to
res, err := json.Marshal(IdentitySuccess{MessageType: "newidentity", Approved: "approved"})
if err != nil {
fmt.Printf("Error: %v", err)
}
c.Write(res)
the server hangs and does not send any response back to client. The strange thing is that when I shut down the server forcefully with Ctrl+C, the server sends the response to the client. Any idea about this strange behavior? It's like the server holds the response and sends it when it exists.
That socket tutorial, just as so many other broken-by-design socket tutorials, doesn't explain at all what an application protocol is or why you need it. All it says is:
In this example, you have implemented an unofficial protocol that is based on TCP.
This "unofficial" protocol is as rudimentary as it gets: messages are separated by newline characters (\n).
You should not be using sockets like that in any environment, apart from learning the basics about sockets.
You need an application protocol to frame messages (so your client and server can recognise partial and concatenated messages).
So the short answer: send a \n after your JSON. The long answer: don't use barebones sockets like this, use an application protocol, such as HTTP.
take care to data races. You are writing and reading the counter variable from different routines without synchronization mechanisms. There is no benign data races.
Your implementation wont hit yet, because you are not testing simultaneous clients queries.
Enable the race detector by building your program using the -race flag, like this go run -race . / go build -race .
I have fixed the data race using the atomic package functions.
In below code, i have adjusted your code to use a bufio.Scanner instead of bufio.Reader, only for demonstration purposes.
input := bufio.NewScanner(src)
output := bufio.NewScanner(c)
for input.Scan() {
text := input.Text()
fmt.Fprintf(c, "%v\n", text)
isEOT := text == "STOP"
if !output.Scan() {
fmt.Fprintln(os.Stderr, output.Err())
return
}
message := output.Text()
fmt.Print("->: " + message)
if isEOT {
fmt.Println("All messages sent...")
return
}
}
I also have adjusted the main sequence to simulate 2 consecutive clients, using a predefined buffer input that I reset along the way.
input := `hello
world!
STOP
nopnop`
test := strings.NewReader(input)
go serve(arguments[1])
test.Reset(input)
query(arguments[1], test)
test.Reset(input)
query(arguments[1], test)
I added a simple retrier into your client, it helps us writing simple code.
c, err := net.Dial("tcp", addr)
for {
if err != nil {
fmt.Fprintln(os.Stderr, err)
<-time.After(time.Second)
c, err = net.Dial("tcp", addr)
continue
}
break
}
The overall program is assembled into one file, not really good to read the output, but easier to transport around and execute.
https://play.golang.org/p/keKQsKA3fAw
In below example I demonstrate how you can use a json marshaller / unmarshaller to exchange structured data.
input := bufio.NewScanner(src)
dst := json.NewEncoder(c)
output := json.NewDecoder(c)
for input.Scan() {
text := input.Text()
isEOT := text == "STOP"
err = dst.Encode(text)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
var tmp interface{}
err = output.Decode(&tmp)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
fmt.Printf("->: %v\n", tmp)
if isEOT {
fmt.Println("All messages sent...")
return
}
}
But ! Beware, this last version is sensible to malicious users. Unlike bufio.Scanner or bufio.Reader it does not check the amount of data read on the wire. So it can possibly accumulate data until OOM.
This is particularly true for the server side of the thing, in
defer c.Close()
defer atomic.AddUint64(&count, ^uint64(0))
input := json.NewDecoder(c)
output := json.NewEncoder(c)
fmt.Print(".")
for {
var netData interface{}
input.Decode(&netData)
fmt.Printf("%v", netData)
count := atomic.LoadUint64(&count)
output.Encode(count)
if x, ok := netData.(string); ok && x == "STOP" {
break
}
}
https://play.golang.org/p/LpIu4ofpm9e
In your last piece of code, as answered by CodeCaster, don't forget to frame your messages using the appropriate delimiter.

How to return specific JSON data by an ID parameter in Golang server

I am working on a project I've tasked to myself.
I'm trying to set up a Golang server that displays JSON data from a remote json file based on a parameter (ID) passed by the user.
I can find plenty of guides that show how to consume ALL the data from the json API, but I don't know how to pass a parameter to the function so it only returns the required data.
Here is what I have so far:
Setting up the basic server and routes
func main() {
//Initialises basic router and endpoints
r := mux.NewRouter()
r.HandleFunc("/", home).Methods("GET")
r.HandleFunc("/games/all", getAll).Methods("GET")
r.HandleFunc("/games/{id}", getGame).Methods("POST")
r.HandleFunc("/games/report", getReport).Methods("GET")
fmt.Println("Listening on port 8080")
http.ListenAndServe(":8080", r)
}
This is my getAll method that sends the GET request (Using the pokedex api for now until i host my json file).
func getAll(w http.ResponseWriter, r *http.Request) {
response, err := http.Get("http://pokeapi.co/api/v2/pokedex/kanto/")
if err != nil {
fmt.Print(err.Error())
os.Exit(1)
}
responseData, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(responseData))
}
Have also built the structs for the data. But i cant, for the life of me, find a way to pass the parameter to do the getGame method to handle that route. Please, if anybody could point me in the right direction.

Golang library (or code) needed for client JSON-RPC2.0 over websocket

I have to add
JSON-RPC2.0
over websockets, to my
Golang
Client process
to communicate with another server. Unfortunately, none of the requirements can be changed.
I have libs and example code that offer parts of the solution but I have been unable to put it all together.
I have looked at Ethereum (I cannot find a client example) Ethereum rpc
A stackoverflow question that dials tcp.
This dials TCP
net.rpc not 2.0, dials tcp
over http net/rpc and http
This one looked promising but I cannot make my client work.
type Params struct {
Name string `json:"name"`
}
func main() {
h := NewRPCHandler()
h.Register("some_handler", dummySuccessHandler)
ts := http.Server{Handler: h}
defer ts.Close()
conn, _, err := websocket.DefaultDialer.Dial("ws://127.0.0.1:8080/ws_rpc", nil)
if err != nil {
log.Printf("Error dialing %v", err)
}
client := jsonrpc.NewClient(conn.UnderlyingConn())
par := &Params{"newuser"}
req, err := json.Marshal(&par)
if err != nil {
log.Printf("Bad marshal %#v", err)
}
var resp Result
err = client.Call("some_handler", &req, &resp)
I tried with/without marshaling but I'm clutching at straws here.

Create a new JSON file for every GET request

Background story: I'm retrieving details using a GET method. I managed to get the program to parse the output given by the server into a JSON file titled "output.json".
Problem: Every time I make different requests, the output will overwrite any previous content in the output.json file. Is it possible to make a new JSON file for every request?
I am rather new to GoLang and any help will be very useful :)
Note: I am only showing the method that is being used to call the API, if need to, I will show the rest of my code.
Here is my method used:
func render(endpoint string, w http.ResponseWriter, r *http.Request) {
// check if request has cookie set
if cookie, err := r.Cookie(COOKIE_NAME); err != nil {
// else redirect to OAuth Authorization EP
redirectToOAuth(w, r, endpoint)
} else {
session := cookie.Value
accessToken := sessions[session]
// pipe api endpoint
ep := fmt.Sprintf("%s/%s", fidorConfig.FidorApiUrl, endpoint)
if api_req, err := http.NewRequest("GET", ep, nil); err != nil {
w.WriteHeader(500)
w.Write([]byte(err.Error()))
} else {
api_req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", accessToken))
api_req.Header.Set("Accept", "application/vnd.fidor.de; version=1,text/json")
client := &http.Client{}
if api_resp, err := client.Do(api_req); err != nil {
w.WriteHeader(500)
w.Write([]byte(err.Error()))
} else {
if api_resp.StatusCode == 401 { // token probably expired
handleLogout(w, r, endpoint)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(api_resp.StatusCode)
defer api_resp.Body.Close()
out, err := os.Create("output.json")
if err != nil {
// panic?
}
defer out.Close()
io.Copy(out, api_resp.Body)
}
}
}
}
If you want to append time in your filename (like #Florian suggested), you can do something like this when you are creating file:
out, err := os.Create(fmt.Sprintf("output-%d.json", time.Now().Unix()))
// filename => output-1257894000.json
Here time.Now().Unix() returns the number of seconds elapsed since January 1, 1970 UTC (aka Unix time). So each time it will create different json file.
More info about time: https://golang.org/pkg/time/#Time.Unix
If you don't want your file to be overwritten you can either give the file a different name (for example by appending the current time using time.Now()) or you can append the output to the file, so the output.json contains all json files.

golang http+jsonrpc access from web page

I've used Go's net/rpc and net/rpc/jsonrpc packages a bit to perform connections between Go processes, however I'm wondering if there is a way to connect to an HTTP JSONRPC server using only the server tooling from the standard library (not that I have a problem writing my own, just don't want to do it if I don't have to).
This is the basic server setup I have:
arith := new(server.Arith)
server := rpc.NewServer()
server.Register(arith)
server.HandleHTTP(rpc.DefaultRPCPath, rpc.DefaultDebugPath)
listener, e := net.Listen("tcp", ":4321")
if e != nil {
log.Fatal("listen error:", e)
}
defer listener.Close()
http.Serve(listener, http.DefaultServeMux)
And I'd like to be able to be hitting this from a web page or a simple command line CURL call - just a regular POST.
However, this line: http://golang.org/src/net/rpc/server.go?s=20445:20475#L670 appears to indicate that it expects an HTTP client to issue a CONNECT and then directly write the JSON RPC request to the stream and receive the reply back the same way. I don't know if this is even possible from a browser, but it certainly is not as common or compatible as a simple POST.
Is there a way to start a JSON RPC server that I can just POST to using good ol' XMLHttpRequest ?
EDIT: Crap - the above is not even using the jsonrpc stuff - this is probably trying to use Gob, but whatever - the problem is the same - the code in src/net/rpc/server.go is not going to handle POSTs, so this route overall isn't going to work regardless of server codec.
FWIW, I got this working by making a simple HTTP handler that adapts the HTTP request/response to a ServerCodec. Seems to work like a charm.
Here's the working code as a test:
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"log"
"net"
"net/http"
"net/rpc"
"net/rpc/jsonrpc"
"testing"
)
// adapt HTTP connection to ReadWriteCloser
type HttpConn struct {
in io.Reader
out io.Writer
}
func (c *HttpConn) Read(p []byte) (n int, err error) { return c.in.Read(p) }
func (c *HttpConn) Write(d []byte) (n int, err error) { return c.out.Write(d) }
func (c *HttpConn) Close() error { return nil }
// our service
type CakeBaker struct{}
func (cb *CakeBaker) BakeIt(n int, msg *string) error {
*msg = fmt.Sprintf("your cake has been bacon (%d)", n)
return nil
}
func TestHTTPServer(t *testing.T) {
fmt.Printf("TestHTTPServer\n")
cb := &CakeBaker{}
server := rpc.NewServer()
server.Register(cb)
listener, e := net.Listen("tcp", ":4321")
if e != nil {
log.Fatal("listen error:", e)
}
defer listener.Close()
go http.Serve(listener, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/bake-me-a-cake" {
serverCodec := jsonrpc.NewServerCodec(&HttpConn{in: r.Body, out: w})
w.Header().Set("Content-type", "application/json")
w.WriteHeader(200)
err := server.ServeRequest(serverCodec)
if err != nil {
log.Printf("Error while serving JSON request: %v", err)
http.Error(w, "Error while serving JSON request, details have been logged.", 500)
return
}
}
}))
resp, err := http.Post("http://localhost:4321/bake-me-a-cake", "application/json", bytes.NewBufferString(
`{"jsonrpc":"2.0","id":1,"method":"CakeBaker.BakeIt","params":[10]}`,
))
if err != nil {
panic(err)
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
fmt.Printf("returned JSON: %s\n", string(b))
}
a RPC framework shoud have language supports list, I not used json-rpc, but it should support javascript language by this link. you need add one of the javascript client sdk listed there.