I am encountering a problem while using ristretto cache. Indeed, i have a little api that should return me a value stored in my ristretto cache as json.
The problem is that when i call my function, the return is the json encoded in base64 and i just can't find the way to decode it.
Here is the code i have:
Part 1: the code for initializing my ristretto cache:
func InitCache() {
var err error
ristrettoCache, err = ristretto.NewCache(&ristretto.Config{
NumCounters: 3000,
MaxCost: 1e6,
BufferItems: 64,
})
if err != nil {
panic(err)
}
}
Part 2: Putting my values in cache:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, b, 1)
}
Part 3: getting the value from cache
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
json.NewEncoder(w).Encode(value)
}
The result i have when i make a call to my api is:
"eyJTeW1ib2wiOiJic2NDUllQVE8iLCJBZGRyIjoiMHgyQmNBMUFlM0U1MjQ0NzMyM0IzRWE0NzA4QTNkMTg1ODRDYWY4NWE3IiwiTHBzIjpbeyJTeW1ib2xUb2tlbiI6IkZFRyIsIlRva2VuQWRkciI6IjB4YWNGQzk1NTg1RDgwQWI2MmY2N0ExNEM1NjZDMWI3YTQ5RmU5MTE2NyIsIkxwQWRkciI6IjB4NDU5ZTJlMjQ4NGNlMDU2MWRmNTJiYzFlNjkxMzkyNDA2M2JhZDM5MCJ9LHsiU3ltYm9sVG9rZW4iOiJmQk5CIiwiVG9rZW5BZGRyIjoiMHg4N2IxQWNjRTZhMTk1OEU1MjIyMzNBNzM3MzEzQzA4NjU1MWE1Yzc2IiwiTHBBZGRyIjoiMHg3OGM2NzkzZGMxMDY1OWZlN2U0YWJhMTQwMmI5M2Y2ODljOGY0YzI3In1dfQ=="
But i want the base64 decoded version...
If I change the value b to be string when i insert it in cache like so:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, string(b), 1)
}
When i get the response, i get the stringified json like this:
"{"Symbol":"bscCRYPTO","Addr":"0x2BcA1Ae3E52447323B..."
And i can't find a way to get out of this string :/
Anyone would know how i could get the real json please?
Thank you in advance and i wish u a good day!
From my comments, I meant, in this line, value is most likely of type []byte (or []uint8 - which is the same thing)
value, found := ristrettoCache.Get(key)
JSON encoding a []byte will implicitly base64 the output - since JSON is text-based.
json.NewEncoder(w).Encode(value) // <- value is of type []byte
Inspecting the base64 you posted (https://play.golang.org/p/NAVS4qRfDM2) the underlying binary-bytes are already encoded in JSON - so no extra json.Encode is needed.
Just output the raw-bytes in your handler - and set the content-type to application/json:
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
// json.NewEncoder(w).Encode(value) // not this
w.Header().Set("Content-Type", "application/json")
if bs, ok := value.([]byte); ok {
_, err := w.Write(bs) //raw bytes (already encoded in JSON)
// check 'err'
} else {
// error unexpected type behind interface{}
}
}
Related
I am new to go and I am trying to build a little weather app using OpenWeatherMap
and the go-package by briandowns.
I have no problem with reading the current weather
but I have trouble processing the results of the forecast methods.
func main() {
apiKey := "XXXX"
w, err := owm.NewForecast("5", "C", "en", apiKey)
if err != nil {
log.Fatal(err)
}
w.DailyByName("London", 1)
data := w.ForecastWeatherJson
fmt.Println(data)
}
where the apiKey needs to be replaced by a valid one (which one can get for free upon registration).
My problem is to extract the information from the ForecastWeatherJson.
It is defined as:
type ForecastWeatherJson interface {
Decode(r io.Reader) error
}
in the forecast.go file.
With Decode defined as:
func (f *Forecast5WeatherData) Decode(r io.Reader) error {
if err := json.NewDecoder(r).Decode(&f); err != nil {
return err
}
return nil
}
in forecast5.go.
I really do not know where to start as I did not find a documented example which showed processing the data except for other languages (so I guess it s a go specific problem).
I saw how it can be done in e.g. python but in the go case the return type is not clear to me.
Any hints or links to examples are appreciated.
Data that you need are already decoded in you w param, but you need to type assert to correct Weather type. In your case because you are using type=5 you should use owm.Forecast5WeatherData. Then your main will look like this.
func main() {
apiKey := "XXXX"
w, err := owm.NewForecast("5", "C", "en", apiKey)
if err != nil {
log.Fatal(err)
}
w.DailyByName("London", 3)
if val, ok := w.ForecastWeatherJson.(*owm.Forecast5WeatherData); ok {
fmt.Println(val)
fmt.Println(val.City)
fmt.Println(val.Cnt)
}
}
I have a data encryption function that returns a [] byte. Of course, what has been encrypted must be decrypted (through another function) in another micro-service.
The problem is created when I send the []byte via JSON: the []byte is transformed into a string and then when I go to read the JSON through the call, the result is no longer the same.
I have to be able to pass the original []byte, created by the encryption function, through JSON or otherwise pass the []byte through a call like the one you can see below. Another possibility is to change the decryption function, but I have not succeeded.
caller function
func Dati_mono(c *gin.Context) {
id := c.Param("id")
oracle, err := http.Get("http://XXXX/"+id)
if err != nil {
panic(err)
}
defer oracle.Body.Close()
oJSON, err := ioutil.ReadAll(oracle.Body)
if err != nil {
panic(err)
}
oracleJSON := security.Decrypt(oJSON, keyEn)
c.JSON(http.StatusOK, string(oJSON))
}
function that is called with the url
func Dati(c *gin.Context) {
var (
person Person
result mapstring.Dati_Plus
mmap []map[string]interface{}
)
rows, err := db.DBConor.Query("SELECT COD_DIPENDENTE, MATRICOLA, COGNOME FROM ANDIP021_K")
if err != nil {
fmt.Print(err.Error())
}
for rows.Next() {
err = rows.Scan(&person.COD_DIPENDENTE, &person.MATRICOLA, &person.COGNOME)
ciao := structs.Map(&person)
mmap = append(mmap, ciao)
}
defer rows.Close()
result = mapstring.Dati_Plus{
len(mmap),
mmap,
}
jsonEn := []byte(mapstring.Dati_PlustoStr(result))
keyEn := []byte(key)
cipherjson, err := security.Encrypt(jsonEn, keyEn)
if err != nil {
log.Fatal(err)
}
c.JSON(http.StatusOK, cipherjson)
}
encryption and decryption functions
func Encrypt(json []byte, key []byte) (string, error) {
k, err := aes.NewCipher(key)
if err != nil {
return "nil", err
}
gcm, err := cipher.NewGCM(k)
if err != nil {
return "nil", err
}
nonce := make([]byte, gcm.NonceSize())
if _, err = io.ReadFull(rand.Reader, nonce); err != nil {
return "nil", err
}
return gcm.Seal(nonce, nonce, json, nil), nil
}
func Decrypt(cipherjson []byte, key []byte) ([]byte, error) {
k, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
gcm, err := cipher.NewGCM(k)
if err != nil {
return nil, err
}
nonceSize := gcm.NonceSize()
if len(cipherjson) < nonceSize {
return nil, errors.New("cipherjson too short")
}
nonce, cipherjson := cipherjson[:nonceSize], cipherjson[nonceSize:]
return gcm.Open(nil, nonce, cipherjson, nil)
}
Everything works, the problem is created when I print cipherjson in c.JSON (): the []byte is translated into a string.
At the time it is taken and read by the calling function it is read as string and ioutil.ReadAll () creates the [] byte of the read string.
Instead I must be able to pass to the Decryot function the return of the Encrypt function used in the called function.
I hope I was clear, thanks in advance
You are not decoding the response before decrypting. In other words, you are handing the JSON encoding of the ciphertext to Decrypt. That is obviously not going to do what you want. To recover the plaintext you have to precisely undo all of the operations of the encryption and encoding in reverse order.
Either decode before decrypting, or don't JSON encode on the server. For instance:
oJSON, err := ioutil.ReadAll(oracle.Body)
if err != nil {
panic(err)
}
var ciphertext string
if err := json.Unmarshal(oJSON, &ciphertext); err != nil {
// TODO: handle error
}
oracleJSON := security.Decrypt(ciphertext, keyEn)
Although it is unclear why you even go through the trouble of JSON encoding in the first place. You might as well just write the ciphertext directly. If you really want to encode the ciphertext, you should not convert it to a string. The ciphertext is just a bunch of random bytes, not remotely resembling a UTF-8 encoded string, so don't treat it like one. encoding/json uses the base64 encoding for byte slices automatically, which is a much cleaner (and probably shorter) representation of the ciphertext than tons of unicode escape sequences.
Independent of the encoding you choose (if any), your Encrypt function is broken.
// The plaintext and dst must overlap exactly or not at all. To reuse
// plaintext's storage for the encrypted output, use plaintext[:0] as dst.
Seal(dst, nonce, plaintext, additionalData []byte) []byte
The first argument is the destination for the encryption. If you don't need to retain the plaintext, pass json[:0]; otherwise pass nil.
Also, Decrypt expects the ciphertext to be prefixed by the nonce, but Encrypt doesn't prepend it.
I have the following type in Golang:
type Base64Data []byte
In order to support unmarshalling a base64 encoded string to this type, I did the following:
func (b *Base64Data) UnmarshalJSON(data []byte) error {
if len(data) == 0 {
return nil
}
content, err := base64.StdEncoding.DecodeString(string(data[1 : len(data)-1]))
if err != nil {
return err
}
*b = []byte(xml)
return nil
}
Now I also want to be able to marshal and unmarshal it to mongo database, using mgo Golang library.
The problem is that I already have documents there stored as base64 encoded string, so I have to maintain that.
I tried to do the following:
func (b Base64Data) GetBSON() (interface{}, error) {
return base64.StdEncoding.EncodeToString([]byte(b)), nil
}
func (b *Base64DecodedXml) SetBSON(raw bson.Raw) error {
var s string
var err error
if err = raw.Unmarshal(&s); err != nil {
return err
}
*b, err = base64.StdEncoding.DecodeString(s)
return err
}
So that after unmarshaling, the data is already decoded, so I need to encode it back, and return it as a string so it will be written to db as a string (and vice versa)
For that I implemented bson getter and setter, but it seems only the getter is working properly
JSON unmarshaling from base64 encoded string works, as well marshaling it to database. but unmarshling setter seems to not be called at all.
Can anyone suggest what I'm missing, so that I'll be able to properly hold the data decoded in memory, but encoded string type?
This is a test I tried to run:
b := struct {
Value shared.Base64Data `json:"value" bson:"value"`
}{}
s := `{"value": "PHJvb3Q+aGVsbG88L3Jvb3Q+"}`
require.NoError(t, json.Unmarshal([]byte(s), &b))
t.Logf("%v", string(b.Value))
b4, err := bson.Marshal(b)
require.NoError(t, err)
t.Logf("%v", string(b4))
require.NoError(t, bson.Unmarshal(b4, &b))
t.Logf("%v", string(b.Value))
You can't marshal any value with bson.Marshal(), only maps and struct values.
If you want to test it, pass a map, e.g. bson.M to bson.Marshal():
var x = Base64Data{0x01, 0x02, 0x03}
dd, err := bson.Marshal(bson.M{"data": x})
fmt.Println(string(dd), err)
Your code works as-is, and as you intend it to. Try to insert a wrapper value to verify it:
c := sess.DB("testdb").C("testcoll")
var x = Base64Data{0x01, 0x02, 0x03}
if err := c.Insert(bson.M{
"data": x,
}); err != nil {
panic(err)
}
This will save the data as a string, being the Base64 encoded form.
Of course if you want to load it back into a value of type Base64Data, you will also need to define the SetBSON(raw Raw) error method too (bson.Setter interface).
I am trying to develop a REST service with net/http.
The service receives a JSON structure containing all the input parameters. I wonder if there is an easier and shorter way to implement the following:
func call(w http.ResponseWriter, r *http.Request) {
if err := r.ParseForm(); err != nil {
fmt.Printf("Error parsing request %s\n", err)
}
var buf []byte
buf = make([]byte, 256)
var n, err = r.Body.Read(buf)
var decoded map[string]interface{}
err = json.Unmarshal(buf[:n], &decoded)
if err != nil {
fmt.Printf("Error decoding json: %s\n", err)
}
var uid = decoded["uid"]
...
}
As you can see it requires quite a number of lines just to get to the extraction of the first parameter. Any ideas?
You don't need to call r.ParseForm if the body of the request will contain a JSON structure and you don't need any URL parameters.
You don't need the buffer either; you can use:
decoder := json.NewDecoder(r.Body)
And then:
error := decoder.Decode(decoded)
Putting it all together:
func call(w http.ResponseWriter, r *http.Request) {
values := make(map[string]interface{})
if error := json.NewDecoder(r.Body).Decode(&values); error != nil {
panic(error)
}
uid := values["uid"].(int)
}
It would be much nicer, though, if you could formally define the structure of the input that you're expecting in a struct type:
type UpdateUserInformationRequest struct {
UserId int `json:"uid"`
// other fields...
}
And use an instance of this struct instead of a more general map.
I have a handler like this:
type Location struct {
Lat float32 `json:"lat"`
Lon float32 `json:"lon"`
}
func handleJSONLocation(res http.ResponseWriter, req *http.Request) {
b := new(Location)
var bb Location;
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
json.Unmarshal(buf.Bytes(), &bb)
json.NewDecoder(req.Body).Decode(b)
log.Printf("%s %f,%f %f,%f", buf.String(), b.Lat, b.Lon, bb.Lat, bb.Lon);
data, _ := json.Marshal("{'location':'saved'}")
res.Header().Set("Content-Type", "application/json; charset=utf-8")
res.Write(data)
}
I get
2014/04/25 22:05:55 {"lat":42.9635013,"lon":-81.22387210000001} 0.000000,0.000000 42.963501,-81.223869
Why doesn't b object get decoded into?
Regards
When you run the following code, the entire body of the HTTP request is read into buf:
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
At this point, further reads from req.Body will result in an EOF error. So when you attempt to decode into b, you are getting an error, which would be visible if you checked the return value with something like:
if err := json.NewDecoder(req.Body).Decode(b); err != nil {
log.Println("Could not decode body:", err)
}
It would also be wise to check the error on the other function calls you make.
Since you've already read the request body into a memory buffer though, you could create a new bytes.Buffer to pass to APIs that expect an io.Reader (such as json.NewDecoder).