golang http+jsonrpc access from web page - json

I've used Go's net/rpc and net/rpc/jsonrpc packages a bit to perform connections between Go processes, however I'm wondering if there is a way to connect to an HTTP JSONRPC server using only the server tooling from the standard library (not that I have a problem writing my own, just don't want to do it if I don't have to).
This is the basic server setup I have:
arith := new(server.Arith)
server := rpc.NewServer()
server.Register(arith)
server.HandleHTTP(rpc.DefaultRPCPath, rpc.DefaultDebugPath)
listener, e := net.Listen("tcp", ":4321")
if e != nil {
log.Fatal("listen error:", e)
}
defer listener.Close()
http.Serve(listener, http.DefaultServeMux)
And I'd like to be able to be hitting this from a web page or a simple command line CURL call - just a regular POST.
However, this line: http://golang.org/src/net/rpc/server.go?s=20445:20475#L670 appears to indicate that it expects an HTTP client to issue a CONNECT and then directly write the JSON RPC request to the stream and receive the reply back the same way. I don't know if this is even possible from a browser, but it certainly is not as common or compatible as a simple POST.
Is there a way to start a JSON RPC server that I can just POST to using good ol' XMLHttpRequest ?
EDIT: Crap - the above is not even using the jsonrpc stuff - this is probably trying to use Gob, but whatever - the problem is the same - the code in src/net/rpc/server.go is not going to handle POSTs, so this route overall isn't going to work regardless of server codec.

FWIW, I got this working by making a simple HTTP handler that adapts the HTTP request/response to a ServerCodec. Seems to work like a charm.
Here's the working code as a test:
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"log"
"net"
"net/http"
"net/rpc"
"net/rpc/jsonrpc"
"testing"
)
// adapt HTTP connection to ReadWriteCloser
type HttpConn struct {
in io.Reader
out io.Writer
}
func (c *HttpConn) Read(p []byte) (n int, err error) { return c.in.Read(p) }
func (c *HttpConn) Write(d []byte) (n int, err error) { return c.out.Write(d) }
func (c *HttpConn) Close() error { return nil }
// our service
type CakeBaker struct{}
func (cb *CakeBaker) BakeIt(n int, msg *string) error {
*msg = fmt.Sprintf("your cake has been bacon (%d)", n)
return nil
}
func TestHTTPServer(t *testing.T) {
fmt.Printf("TestHTTPServer\n")
cb := &CakeBaker{}
server := rpc.NewServer()
server.Register(cb)
listener, e := net.Listen("tcp", ":4321")
if e != nil {
log.Fatal("listen error:", e)
}
defer listener.Close()
go http.Serve(listener, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/bake-me-a-cake" {
serverCodec := jsonrpc.NewServerCodec(&HttpConn{in: r.Body, out: w})
w.Header().Set("Content-type", "application/json")
w.WriteHeader(200)
err := server.ServeRequest(serverCodec)
if err != nil {
log.Printf("Error while serving JSON request: %v", err)
http.Error(w, "Error while serving JSON request, details have been logged.", 500)
return
}
}
}))
resp, err := http.Post("http://localhost:4321/bake-me-a-cake", "application/json", bytes.NewBufferString(
`{"jsonrpc":"2.0","id":1,"method":"CakeBaker.BakeIt","params":[10]}`,
))
if err != nil {
panic(err)
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
fmt.Printf("returned JSON: %s\n", string(b))
}

a RPC framework shoud have language supports list, I not used json-rpc, but it should support javascript language by this link. you need add one of the javascript client sdk listed there.

Related

Go json.NewDecoder().Decode() doesn't seem to respect context deadline

I have a Golang program with a context deadline set. I am sending an HTTP request, and expected to see a deadline exceeded error when Im reading the body.
It seems that when I read the response body with ioutil.ReadAll then that read method will get interrupted (?) and return the appropriate error (context.DeadlineExceeded).
However if I read the response body with json.NewDecoder(resp.Body).Decode then the error returned is nil (instead of context.DeadlineExceeded). My full code is below. Is this a bug in json.NewDecoder(resp.Body).Decode?
package main
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"time"
)
var url string = "http://ip.jsontest.com/"
func main() {
readDoesntFail()
readFails()
}
type IpResponse struct {
Ip string
}
func readDoesntFail() {
ctx, _ := context.WithTimeout(context.Background(), time.Second*5)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
ipResponse := new(IpResponse)
time.Sleep(time.Second * 6)
fmt.Println("before reading response body, context error is:", ctx.Err())
err = json.NewDecoder(resp.Body).Decode(ipResponse)
if err != nil {
panic(err)
}
fmt.Println("Expected panic but there was none")
}
func readFails() {
ctx, _ := context.WithTimeout(context.Background(), time.Second*5)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
time.Sleep(time.Second * 6)
fmt.Println("before reading response body, context error is:", ctx.Err())
_, err = ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println("received expected error", err)
}
}
The net/http package may use buffers to process requests. This means the incoming response body may be read and buffered partly or entirely before you read it, so an expiring context may not prevent you to finish reading the body. And this is exactly what happens.
Let's modify your example to fire up a test HTTP server which deliberately delays the response (partly):
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s := []byte(`{"ip":"12.34.56.78"}`)
w.Write(s[:10])
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
time.Sleep(time.Second * 6)
w.Write(s[10:])
}))
defer ts.Close()
url = ts.URL
readDoesntFail()
readFails()
This test server sends a similar JSON object to that of ip.jsontest.com's response. But it only sends 10 bytes body, then flushes it, then sleeps 6 seconds on purpose before sending the rest, "allowing" the client to time out.
Now let's see what happens if we call readDoesntFail():
before reading response body, context error is: context deadline exceeded
panic: Get "http://127.0.0.1:38230": context deadline exceeded
goroutine 1 [running]:
main.readDoesntFail()
/tmp/sandbox721114198/prog.go:46 +0x2b4
main.main()
/tmp/sandbox721114198/prog.go:28 +0x93
Try it on the Go Playground.
In your example json.Decoder.Decode() reads already buffered data, so the expired context plays no role here. In my example json.Decoder.Decode() tries to read from the connection because the data isn't yet buffered (it can't be as it hasn't been sent yet), so once the context expires, further reading from the connection returns a deadline exceeded error.

How can I send request payload data using a JSON file in GO?

I'm really new to coding and Golang itself.
I would like to know how can I send request Payload data using a JSON file in GO?
I mean, I have a post request and the JSON file and I would like to put it into the request body but I am coming across some errors.
The request is working when I use an alternative HTTP client.
Depending on the nature of the HTTP request, you may be able to use an existing client package. Eg, JSON RPC.
Here is an example if you would like to understand how to make a request using the standard library. This example also demonstrates using context to set timeouts for client requests:
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"time"
)
func main() {
ctx := context.Background()
var client http.Client
reqCtx, cancel := context.WithTimeout(ctx, time.Minute)
defer cancel()
err := deleteEntry(reqCtx, &client, 42)
fmt.Println(err)
}
func deleteEntry(ctx context.Context, client *http.Client, entryID int) error {
payload := &struct {
EntryID int `json:"entry_id"`
Method string `json:"method"`
}{
EntryID: entryID,
Method: "delete",
}
buf, err := json.Marshal(payload)
if err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, "POST", "http://localhost/example", bytes.NewReader(buf))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return err
}
// Note: Response body must always be closed.
// Response body data (if any) should be consumed before closure, otherwise the
// the client connection may not be reused.
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("request failed with %s", resp.Status)
}
return nil
}
I'd recommend reading through the net/http documentation to gain a better understanding. In particular:
http.Request
http.Response

Go concurrent TCP server hangs when JSON is sent as the response but works with plain string

I am trying to implement a concurrent TCP server in Go and found this great explanatory article in linode where it clearly explains with a sample code. The sample code snippets for the client and server and included below.
Concurrent TCP server where for each TCP client a new go-routine is created.
package main
import (
"bufio"
"fmt"
"net"
"os"
"strconv"
"strings"
)
var count = 0
func handleConnection(c net.Conn) {
fmt.Print(".")
for {
netData, err := bufio.NewReader(c).ReadString('\n')
if err != nil {
fmt.Println(err)
return
}
temp := strings.TrimSpace(string(netData))
if temp == "STOP" {
break
}
fmt.Println(temp)
counter := strconv.Itoa(count) + "\n"
c.Write([]byte(string(counter)))
}
c.Close()
}
func main() {
arguments := os.Args
if len(arguments) == 1 {
fmt.Println("Please provide a port number!")
return
}
PORT := ":" + arguments[1]
l, err := net.Listen("tcp4", PORT)
if err != nil {
fmt.Println(err)
return
}
defer l.Close()
for {
c, err := l.Accept()
if err != nil {
fmt.Println(err)
return
}
go handleConnection(c)
count++
}
}
TCP client code snippet
package main
import (
"bufio"
"fmt"
"net"
"os"
"strings"
)
func main() {
arguments := os.Args
if len(arguments) == 1 {
fmt.Println("Please provide host:port.")
return
}
CONNECT := arguments[1]
c, err := net.Dial("tcp", CONNECT)
if err != nil {
fmt.Println(err)
return
}
for {
reader := bufio.NewReader(os.Stdin)
fmt.Print(">> ")
text, _ := reader.ReadString('\n')
fmt.Fprintf(c, text+"\n")
message, _ := bufio.NewReader(c).ReadString('\n')
fmt.Print("->: " + message)
if strings.TrimSpace(string(text)) == "STOP" {
fmt.Println("TCP client exiting...")
return
}
}
}
The above concurrent TCP server and client works without any issue. The issue comes when I change the TCP server to send a JSON response instead of the text response. When I change the line:
counter := strconv.Itoa(count) + "\n"
c.Write([]byte(string(counter)))
to
res, err := json.Marshal(IdentitySuccess{MessageType: "newidentity", Approved: "approved"})
if err != nil {
fmt.Printf("Error: %v", err)
}
c.Write(res)
the server hangs and does not send any response back to client. The strange thing is that when I shut down the server forcefully with Ctrl+C, the server sends the response to the client. Any idea about this strange behavior? It's like the server holds the response and sends it when it exists.
That socket tutorial, just as so many other broken-by-design socket tutorials, doesn't explain at all what an application protocol is or why you need it. All it says is:
In this example, you have implemented an unofficial protocol that is based on TCP.
This "unofficial" protocol is as rudimentary as it gets: messages are separated by newline characters (\n).
You should not be using sockets like that in any environment, apart from learning the basics about sockets.
You need an application protocol to frame messages (so your client and server can recognise partial and concatenated messages).
So the short answer: send a \n after your JSON. The long answer: don't use barebones sockets like this, use an application protocol, such as HTTP.
take care to data races. You are writing and reading the counter variable from different routines without synchronization mechanisms. There is no benign data races.
Your implementation wont hit yet, because you are not testing simultaneous clients queries.
Enable the race detector by building your program using the -race flag, like this go run -race . / go build -race .
I have fixed the data race using the atomic package functions.
In below code, i have adjusted your code to use a bufio.Scanner instead of bufio.Reader, only for demonstration purposes.
input := bufio.NewScanner(src)
output := bufio.NewScanner(c)
for input.Scan() {
text := input.Text()
fmt.Fprintf(c, "%v\n", text)
isEOT := text == "STOP"
if !output.Scan() {
fmt.Fprintln(os.Stderr, output.Err())
return
}
message := output.Text()
fmt.Print("->: " + message)
if isEOT {
fmt.Println("All messages sent...")
return
}
}
I also have adjusted the main sequence to simulate 2 consecutive clients, using a predefined buffer input that I reset along the way.
input := `hello
world!
STOP
nopnop`
test := strings.NewReader(input)
go serve(arguments[1])
test.Reset(input)
query(arguments[1], test)
test.Reset(input)
query(arguments[1], test)
I added a simple retrier into your client, it helps us writing simple code.
c, err := net.Dial("tcp", addr)
for {
if err != nil {
fmt.Fprintln(os.Stderr, err)
<-time.After(time.Second)
c, err = net.Dial("tcp", addr)
continue
}
break
}
The overall program is assembled into one file, not really good to read the output, but easier to transport around and execute.
https://play.golang.org/p/keKQsKA3fAw
In below example I demonstrate how you can use a json marshaller / unmarshaller to exchange structured data.
input := bufio.NewScanner(src)
dst := json.NewEncoder(c)
output := json.NewDecoder(c)
for input.Scan() {
text := input.Text()
isEOT := text == "STOP"
err = dst.Encode(text)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
var tmp interface{}
err = output.Decode(&tmp)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
fmt.Printf("->: %v\n", tmp)
if isEOT {
fmt.Println("All messages sent...")
return
}
}
But ! Beware, this last version is sensible to malicious users. Unlike bufio.Scanner or bufio.Reader it does not check the amount of data read on the wire. So it can possibly accumulate data until OOM.
This is particularly true for the server side of the thing, in
defer c.Close()
defer atomic.AddUint64(&count, ^uint64(0))
input := json.NewDecoder(c)
output := json.NewEncoder(c)
fmt.Print(".")
for {
var netData interface{}
input.Decode(&netData)
fmt.Printf("%v", netData)
count := atomic.LoadUint64(&count)
output.Encode(count)
if x, ok := netData.(string); ok && x == "STOP" {
break
}
}
https://play.golang.org/p/LpIu4ofpm9e
In your last piece of code, as answered by CodeCaster, don't forget to frame your messages using the appropriate delimiter.

Printing decoded JSON in Golang

I am very new to Go / programming in general - having just picked it up whilst messing about creating my own crypto currency portfolio web site.
I am struggling printing to the web server output. If I used Printf - it prints to console but as soon as I use Fprintf to print to the web app, I get a number of errors which I can't seem to solve.
Could someone walk me through it?
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
)
type Obsidian []struct {
PriceUsd string `json:"price_usd"`
PriceBtc string `json:"price_btc"`
}
func webserver(w http.ResponseWriter, r *http.Request) {
url := "https://api.coinmarketcap.com/v1/ticker/obsidian/"
req, err := http.NewRequest("GET", url, nil)
if err != nil {
log.Fatal("NewRequest: ", err)
return
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal("Do: ", err)
return
}
defer resp.Body.Close()
var record Obsidian
if err := json.NewDecoder(resp.Body).Decode(&record); err != nil {
log.Println(err)
}
fmt.Printf("%+v", record)
}
func main() {
http.HandleFunc("/test", webserver)
http.ListenAndServe(":8001", nil)
}
I have tried to replace:
fmt.Printf("%+v", record)
with:
fmt.Fprintf("%+v", record)
and receive the following errors:
./test.go:54:21: cannot use "%+v" (type string) as type io.Writer in argument to fmt.Fprintf:
string does not implement io.Writer (missing Write method)
./test.go:54:21: cannot use record (type Obsidian) as type string in argument to fmt.Fprintf
Thanks to #MiloChrisstiansen
fmt.Fprintf(w, "%+v", record)
You could also use
w.Write([]byte(record))

How to properly call JSON-RPC in Go?

I've been trying various configurations in order to call a simple JSON-RPC server for Bitcoin in Go, but didn't manage to get anywhere.
In Python, the entire code looks like:
from jsonrpc import ServiceProxy
access = ServiceProxy("http://user:pass#127.0.0.1:8332")
print access.getinfo()
But in Go, I seem to be bumping into erros like "too many colons in address", or "no such host". I've tried using both of the packages rpc and rpc/jsonrpc, using methods Dial and DialHTTP, using various network parameters and still can't get anywhere.
So, how do I properly call a JSON-RPC server in Go?
The jsonrpc package doesn't support json-rpc over HTTP at the moment. So, you can't use that, sorry.
But the jsonrpc specification is quite simple and it's probably quite easy to write your own jsonrpchttp (oh, I hope you know a better name) package.
I was able to call "getinfo" succesfully using the following (horrible) code:
package main
import (
"encoding/json"
"io/ioutil"
"log"
"net/http"
"strings"
)
func main() {
data, err := json.Marshal(map[string]interface{}{
"method": "getinfo",
"id": 1,
"params": []interface{}{},
})
if err != nil {
log.Fatalf("Marshal: %v", err)
}
resp, err := http.Post("http://bob:secret#127.0.0.1:8332",
"application/json", strings.NewReader(string(data)))
if err != nil {
log.Fatalf("Post: %v", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("ReadAll: %v", err)
}
result := make(map[string]interface{})
err = json.Unmarshal(body, &result)
if err != nil {
log.Fatalf("Unmarshal: %v", err)
}
log.Println(result)
}
Maybe you can clean it up a bit by implementing the rpc.ClientCodec interface (see jsonrpc/client.go for an example). Then you can take advantage of Go's rpc package.