I have a JSON object of type []byte that I created from a struct using json.Marshal. I want to GZip the JSON before posting it to my endpoint. The following does not work:
gz := gzip.NewWriter(myJSON)
because []byte does not implement io.Writer.
Is there some very simple way I can do this once I already have my JSON created?
Compress to a buffer and post that buffer.
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
gz.Write(myJSON)
gz.Close()
Because a *bytes.Buffer statisifies the io.Reader interface, you can use the buffer directly when creating the request.
req, err := http.NewRequest("POST", url, &buf)
if err != nil {
// handle error
}
req.Header.Set("Content-Encoding", "gzip")
req.Header.Set("Content-Type", "application/json; charset=utf-8")
The best option is to stream your JSON marshaling to your gzip writer:
func compressJSON(w io.Writer, i interface{}) error {
gz := gzip.NewWriter(w)
if err := json.NewEncoder(gz).Encode(i); err != nil {
return err
}
return gz.Close()
}
This has the advantage of not buffering your json in memory temporarily, so it will be faster and use less RAM.
If you need this as an io.Reader, such as for the body of an HTTP request, you can use a pipe to do the conversion:
r, w := io.Pipe()
go func() {
err := compressJSON(w, someObject)
w.CloseWithError(err)
}()
req, err := http.NewRequest("POST", "http://example.com/", r)
Related
I am encountering a problem while using ristretto cache. Indeed, i have a little api that should return me a value stored in my ristretto cache as json.
The problem is that when i call my function, the return is the json encoded in base64 and i just can't find the way to decode it.
Here is the code i have:
Part 1: the code for initializing my ristretto cache:
func InitCache() {
var err error
ristrettoCache, err = ristretto.NewCache(&ristretto.Config{
NumCounters: 3000,
MaxCost: 1e6,
BufferItems: 64,
})
if err != nil {
panic(err)
}
}
Part 2: Putting my values in cache:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, b, 1)
}
Part 3: getting the value from cache
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
json.NewEncoder(w).Encode(value)
}
The result i have when i make a call to my api is:
"eyJTeW1ib2wiOiJic2NDUllQVE8iLCJBZGRyIjoiMHgyQmNBMUFlM0U1MjQ0NzMyM0IzRWE0NzA4QTNkMTg1ODRDYWY4NWE3IiwiTHBzIjpbeyJTeW1ib2xUb2tlbiI6IkZFRyIsIlRva2VuQWRkciI6IjB4YWNGQzk1NTg1RDgwQWI2MmY2N0ExNEM1NjZDMWI3YTQ5RmU5MTE2NyIsIkxwQWRkciI6IjB4NDU5ZTJlMjQ4NGNlMDU2MWRmNTJiYzFlNjkxMzkyNDA2M2JhZDM5MCJ9LHsiU3ltYm9sVG9rZW4iOiJmQk5CIiwiVG9rZW5BZGRyIjoiMHg4N2IxQWNjRTZhMTk1OEU1MjIyMzNBNzM3MzEzQzA4NjU1MWE1Yzc2IiwiTHBBZGRyIjoiMHg3OGM2NzkzZGMxMDY1OWZlN2U0YWJhMTQwMmI5M2Y2ODljOGY0YzI3In1dfQ=="
But i want the base64 decoded version...
If I change the value b to be string when i insert it in cache like so:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, string(b), 1)
}
When i get the response, i get the stringified json like this:
"{"Symbol":"bscCRYPTO","Addr":"0x2BcA1Ae3E52447323B..."
And i can't find a way to get out of this string :/
Anyone would know how i could get the real json please?
Thank you in advance and i wish u a good day!
From my comments, I meant, in this line, value is most likely of type []byte (or []uint8 - which is the same thing)
value, found := ristrettoCache.Get(key)
JSON encoding a []byte will implicitly base64 the output - since JSON is text-based.
json.NewEncoder(w).Encode(value) // <- value is of type []byte
Inspecting the base64 you posted (https://play.golang.org/p/NAVS4qRfDM2) the underlying binary-bytes are already encoded in JSON - so no extra json.Encode is needed.
Just output the raw-bytes in your handler - and set the content-type to application/json:
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
// json.NewEncoder(w).Encode(value) // not this
w.Header().Set("Content-Type", "application/json")
if bs, ok := value.([]byte); ok {
_, err := w.Write(bs) //raw bytes (already encoded in JSON)
// check 'err'
} else {
// error unexpected type behind interface{}
}
}
If I create a struct, how do I send it with a HTTP GET request to a web server endpoint?
package main
import (
"encoding/json"
"net/http"
)
type Payload struct {
Endpoint string `json:"endpoint"`
Data map[string]interface{} `json:"data"`
}
/*
eg.
{"endpoint":"some-service", "data": {"userID": "abc123"}}
*/
func main() {
http.HandleFunc("/service", func(w http.ResponseWriter, r *http.Request) {
decoder := json.NewDecoder(r.Body)
var p Payload
err := decoder.Decode(&p)
if err != nil {
panic(err)
}
// How to attach 'p' ?
resp, err := http.Get("www.example.com/" + p.Endpoint) // Add "data": p.Data
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
// handle response here
})
http.ListenAndServe(":8080", nil)
}
The endpoint receiving this data would ideally interpret it as JSON.
HTTP GET requests do not allow a request body.
If you must do it with a GET, basically you have 2 options: add the data as a query parameter, or send it in an HTTP Header field.
Note that both the URL and header fields have length limits, so if you want to "attach" a long JSON text, it might fail. To send arbitrary data, you should use another method, e.g. POST.
Example adding it as a query param:
u, err := url.Parse("http://www.example.com")
if err != nil {
panic(err)
}
params := url.Values{}
params.Add("data", `{"a":1,"b":"c"}`)
u.RawQuery = params.Encode()
// use u.String() as the request URL
Example sending it in a Header field:
req, err := http.NewRequest("GET", "http://example.com", nil)
// ...
req.Header.Add("X-Data", `{"a":1,"b":"c"}`)
resp, err := client.Do(req)
Currently the way to convert an object to json and gzip it is:
jsonBytes, _ := json.Marshal(payload)
//gzip json
var body bytes.Buffer
g := gzip.NewWriter(&body)
g.Write(jsonBytes)
g.Close()
This results in an intermediate large byte buffer jsonBytes, whose only purpose is to be then converted into gzipped buffer.
Is there any way to stream the marshalling of the payload object so it comes out gzipped in the first place?
Yes, you may use json.Encoder to stream the JSON output, and similarly json.Decoder to decode a streamed JSON input. They take any io.Writer and io.Reader to write the JSON result to / read from, including gzip.Writer and gzip.Reader.
For example:
var body bytes.Buffer
w := gzip.NewWriter(&body)
enc := json.NewEncoder(w)
payload := map[string]interface{}{
"one": 1, "two": 2,
}
if err := enc.Encode(payload); err != nil {
panic(err)
}
if err := w.Close(); err != nil {
panic(err)
}
To verify that it works, this is how we can decode it:
r, err := gzip.NewReader(&body)
if err != nil {
panic(err)
}
dec := json.NewDecoder(r)
payload = nil
if err := dec.Decode(&payload); err != nil {
panic(err)
}
fmt.Println("Decoded:", payload)
Which will output (try it on the Go Playground):
Decoded: map[one:1 two:2]
I have a data encryption function that returns a [] byte. Of course, what has been encrypted must be decrypted (through another function) in another micro-service.
The problem is created when I send the []byte via JSON: the []byte is transformed into a string and then when I go to read the JSON through the call, the result is no longer the same.
I have to be able to pass the original []byte, created by the encryption function, through JSON or otherwise pass the []byte through a call like the one you can see below. Another possibility is to change the decryption function, but I have not succeeded.
caller function
func Dati_mono(c *gin.Context) {
id := c.Param("id")
oracle, err := http.Get("http://XXXX/"+id)
if err != nil {
panic(err)
}
defer oracle.Body.Close()
oJSON, err := ioutil.ReadAll(oracle.Body)
if err != nil {
panic(err)
}
oracleJSON := security.Decrypt(oJSON, keyEn)
c.JSON(http.StatusOK, string(oJSON))
}
function that is called with the url
func Dati(c *gin.Context) {
var (
person Person
result mapstring.Dati_Plus
mmap []map[string]interface{}
)
rows, err := db.DBConor.Query("SELECT COD_DIPENDENTE, MATRICOLA, COGNOME FROM ANDIP021_K")
if err != nil {
fmt.Print(err.Error())
}
for rows.Next() {
err = rows.Scan(&person.COD_DIPENDENTE, &person.MATRICOLA, &person.COGNOME)
ciao := structs.Map(&person)
mmap = append(mmap, ciao)
}
defer rows.Close()
result = mapstring.Dati_Plus{
len(mmap),
mmap,
}
jsonEn := []byte(mapstring.Dati_PlustoStr(result))
keyEn := []byte(key)
cipherjson, err := security.Encrypt(jsonEn, keyEn)
if err != nil {
log.Fatal(err)
}
c.JSON(http.StatusOK, cipherjson)
}
encryption and decryption functions
func Encrypt(json []byte, key []byte) (string, error) {
k, err := aes.NewCipher(key)
if err != nil {
return "nil", err
}
gcm, err := cipher.NewGCM(k)
if err != nil {
return "nil", err
}
nonce := make([]byte, gcm.NonceSize())
if _, err = io.ReadFull(rand.Reader, nonce); err != nil {
return "nil", err
}
return gcm.Seal(nonce, nonce, json, nil), nil
}
func Decrypt(cipherjson []byte, key []byte) ([]byte, error) {
k, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
gcm, err := cipher.NewGCM(k)
if err != nil {
return nil, err
}
nonceSize := gcm.NonceSize()
if len(cipherjson) < nonceSize {
return nil, errors.New("cipherjson too short")
}
nonce, cipherjson := cipherjson[:nonceSize], cipherjson[nonceSize:]
return gcm.Open(nil, nonce, cipherjson, nil)
}
Everything works, the problem is created when I print cipherjson in c.JSON (): the []byte is translated into a string.
At the time it is taken and read by the calling function it is read as string and ioutil.ReadAll () creates the [] byte of the read string.
Instead I must be able to pass to the Decryot function the return of the Encrypt function used in the called function.
I hope I was clear, thanks in advance
You are not decoding the response before decrypting. In other words, you are handing the JSON encoding of the ciphertext to Decrypt. That is obviously not going to do what you want. To recover the plaintext you have to precisely undo all of the operations of the encryption and encoding in reverse order.
Either decode before decrypting, or don't JSON encode on the server. For instance:
oJSON, err := ioutil.ReadAll(oracle.Body)
if err != nil {
panic(err)
}
var ciphertext string
if err := json.Unmarshal(oJSON, &ciphertext); err != nil {
// TODO: handle error
}
oracleJSON := security.Decrypt(ciphertext, keyEn)
Although it is unclear why you even go through the trouble of JSON encoding in the first place. You might as well just write the ciphertext directly. If you really want to encode the ciphertext, you should not convert it to a string. The ciphertext is just a bunch of random bytes, not remotely resembling a UTF-8 encoded string, so don't treat it like one. encoding/json uses the base64 encoding for byte slices automatically, which is a much cleaner (and probably shorter) representation of the ciphertext than tons of unicode escape sequences.
Independent of the encoding you choose (if any), your Encrypt function is broken.
// The plaintext and dst must overlap exactly or not at all. To reuse
// plaintext's storage for the encrypted output, use plaintext[:0] as dst.
Seal(dst, nonce, plaintext, additionalData []byte) []byte
The first argument is the destination for the encryption. If you don't need to retain the plaintext, pass json[:0]; otherwise pass nil.
Also, Decrypt expects the ciphertext to be prefixed by the nonce, but Encrypt doesn't prepend it.
I tried working with Apiary and made a universal template to send JSON to mock server and have this code:
package main
import (
"encoding/json"
"fmt"
"github.com/jmcvetta/napping"
"log"
"net/http"
)
func main() {
url := "http://restapi3.apiary.io/notes"
fmt.Println("URL:>", url)
s := napping.Session{}
h := &http.Header{}
h.Set("X-Custom-Header", "myvalue")
s.Header = h
var jsonStr = []byte(`
{
"title": "Buy cheese and bread for breakfast."
}`)
var data map[string]json.RawMessage
err := json.Unmarshal(jsonStr, &data)
if err != nil {
fmt.Println(err)
}
resp, err := s.Post(url, &data, nil, nil)
if err != nil {
log.Fatal(err)
}
fmt.Println("response Status:", resp.Status())
fmt.Println("response Headers:", resp.HttpResponse().Header)
fmt.Println("response Body:", resp.RawText())
}
This code doesn't send JSON properly, but I don't know why. The JSON string can be different in every call. I can't use Struct for this.
I'm not familiar with napping, but using Golang's net/http package works fine (playground):
func main() {
url := "http://restapi3.apiary.io/notes"
fmt.Println("URL:>", url)
var jsonStr = []byte(`{"title":"Buy cheese and bread for breakfast."}`)
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr))
req.Header.Set("X-Custom-Header", "myvalue")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println("response Status:", resp.Status)
fmt.Println("response Headers:", resp.Header)
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println("response Body:", string(body))
}
you can just use post to post your json.
values := map[string]string{"username": username, "password": password}
jsonValue, _ := json.Marshal(values)
resp, err := http.Post(authAuthenticatorUrl, "application/json", bytes.NewBuffer(jsonValue))
If you already have a struct.
import (
"bytes"
"encoding/json"
"io"
"net/http"
"os"
)
// .....
type Student struct {
Name string `json:"name"`
Address string `json:"address"`
}
// .....
body := &Student{
Name: "abc",
Address: "xyz",
}
payloadBuf := new(bytes.Buffer)
json.NewEncoder(payloadBuf).Encode(body)
req, _ := http.NewRequest("POST", url, payloadBuf)
client := &http.Client{}
res, e := client.Do(req)
if e != nil {
return e
}
defer res.Body.Close()
fmt.Println("response Status:", res.Status)
// Print the body to the stdout
io.Copy(os.Stdout, res.Body)
Full gist.
In addition to standard net/http package, you can consider using my GoRequest which wraps around net/http and make your life easier without thinking too much about json or struct. But you can also mix and match both of them in one request! (you can see more details about it in gorequest github page)
So, in the end your code will become like follow:
func main() {
url := "http://restapi3.apiary.io/notes"
fmt.Println("URL:>", url)
request := gorequest.New()
titleList := []string{"title1", "title2", "title3"}
for _, title := range titleList {
resp, body, errs := request.Post(url).
Set("X-Custom-Header", "myvalue").
Send(`{"title":"` + title + `"}`).
End()
if errs != nil {
fmt.Println(errs)
os.Exit(1)
}
fmt.Println("response Status:", resp.Status)
fmt.Println("response Headers:", resp.Header)
fmt.Println("response Body:", body)
}
}
This depends on how you want to achieve. I made this library because I have the same problem with you and I want code that is shorter, easy to use with json, and more maintainable in my codebase and production system.
Example post request for http or https
//Encode the data
postBody, _ := json.Marshal(map[string]string{
"name": "Test",
"email": "Test#Test.com",
})
responseBody := bytes.NewBuffer(postBody)
//Leverage Go's HTTP Post function to make request
resp, err := http.Post("https://postman-echo.com/post", "application/json", responseBody)
//Handle Error
if err != nil {
log.Fatalf("An Error Occured %v", err)
}
defer resp.Body.Close()
//Read the response body
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}
sb := string(body)
log.Printf(sb)
Use io.Pipe for large request bodies as mentioned in another answer. This approach avoids building the entire request body in memory by streaming the data from the JSON encoder to the network.
This answer builds on the other answer by showing how to handle errors. Always handle errors!
Use the pipe's CloseWithError function to propagate encoding errors back to error returned from http.Post.
Handle the error returned from http.Post
Close the response body.
Here's the code:
r, w := io.Pipe()
go func() {
w.CloseWithError(json.NewEncoder(w).Encode(data))
}()
// Ensure that read side of pipe is closed. This
// unblocks goroutine in scenario where http.Post
// errors out before reading the entire request body.
defer r.Close()
resp, err := http.Post(url, r)
if err != nil {
// Adjust error handling here to meet application requrirements.
log.Fatal(err)
}
defer resp.Body.Close()
// Use the response here.
If you have a lot of data to send, you can use a pipe:
package main
import (
"encoding/json"
"io"
"net/http"
)
func main() {
m := map[string]int{"SNG_ID": 75498415}
r, w := io.Pipe()
go func() {
json.NewEncoder(w).Encode(m)
w.Close()
}()
http.Post("https://stackoverflow.com", "application/json", r)
}
https://golang.org/pkg/io#Pipe
if you want to do it like that, you need to use this map for unmarshalling json string.
var data map[string]interface{}
but if you need to change the json each time and to make initialization of your requst body more convenient, you can use this map for creating json body.
var bodyJsonMap map[string]interface{}{
"key1": val1,
"key2": val2,
...
}
Then marshal it to a json-string.