I have a handler like this:
type Location struct {
Lat float32 `json:"lat"`
Lon float32 `json:"lon"`
}
func handleJSONLocation(res http.ResponseWriter, req *http.Request) {
b := new(Location)
var bb Location;
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
json.Unmarshal(buf.Bytes(), &bb)
json.NewDecoder(req.Body).Decode(b)
log.Printf("%s %f,%f %f,%f", buf.String(), b.Lat, b.Lon, bb.Lat, bb.Lon);
data, _ := json.Marshal("{'location':'saved'}")
res.Header().Set("Content-Type", "application/json; charset=utf-8")
res.Write(data)
}
I get
2014/04/25 22:05:55 {"lat":42.9635013,"lon":-81.22387210000001} 0.000000,0.000000 42.963501,-81.223869
Why doesn't b object get decoded into?
Regards
When you run the following code, the entire body of the HTTP request is read into buf:
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
At this point, further reads from req.Body will result in an EOF error. So when you attempt to decode into b, you are getting an error, which would be visible if you checked the return value with something like:
if err := json.NewDecoder(req.Body).Decode(b); err != nil {
log.Println("Could not decode body:", err)
}
It would also be wise to check the error on the other function calls you make.
Since you've already read the request body into a memory buffer though, you could create a new bytes.Buffer to pass to APIs that expect an io.Reader (such as json.NewDecoder).
Related
I am encountering a problem while using ristretto cache. Indeed, i have a little api that should return me a value stored in my ristretto cache as json.
The problem is that when i call my function, the return is the json encoded in base64 and i just can't find the way to decode it.
Here is the code i have:
Part 1: the code for initializing my ristretto cache:
func InitCache() {
var err error
ristrettoCache, err = ristretto.NewCache(&ristretto.Config{
NumCounters: 3000,
MaxCost: 1e6,
BufferItems: 64,
})
if err != nil {
panic(err)
}
}
Part 2: Putting my values in cache:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, b, 1)
}
Part 3: getting the value from cache
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
json.NewEncoder(w).Encode(value)
}
The result i have when i make a call to my api is:
"eyJTeW1ib2wiOiJic2NDUllQVE8iLCJBZGRyIjoiMHgyQmNBMUFlM0U1MjQ0NzMyM0IzRWE0NzA4QTNkMTg1ODRDYWY4NWE3IiwiTHBzIjpbeyJTeW1ib2xUb2tlbiI6IkZFRyIsIlRva2VuQWRkciI6IjB4YWNGQzk1NTg1RDgwQWI2MmY2N0ExNEM1NjZDMWI3YTQ5RmU5MTE2NyIsIkxwQWRkciI6IjB4NDU5ZTJlMjQ4NGNlMDU2MWRmNTJiYzFlNjkxMzkyNDA2M2JhZDM5MCJ9LHsiU3ltYm9sVG9rZW4iOiJmQk5CIiwiVG9rZW5BZGRyIjoiMHg4N2IxQWNjRTZhMTk1OEU1MjIyMzNBNzM3MzEzQzA4NjU1MWE1Yzc2IiwiTHBBZGRyIjoiMHg3OGM2NzkzZGMxMDY1OWZlN2U0YWJhMTQwMmI5M2Y2ODljOGY0YzI3In1dfQ=="
But i want the base64 decoded version...
If I change the value b to be string when i insert it in cache like so:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, string(b), 1)
}
When i get the response, i get the stringified json like this:
"{"Symbol":"bscCRYPTO","Addr":"0x2BcA1Ae3E52447323B..."
And i can't find a way to get out of this string :/
Anyone would know how i could get the real json please?
Thank you in advance and i wish u a good day!
From my comments, I meant, in this line, value is most likely of type []byte (or []uint8 - which is the same thing)
value, found := ristrettoCache.Get(key)
JSON encoding a []byte will implicitly base64 the output - since JSON is text-based.
json.NewEncoder(w).Encode(value) // <- value is of type []byte
Inspecting the base64 you posted (https://play.golang.org/p/NAVS4qRfDM2) the underlying binary-bytes are already encoded in JSON - so no extra json.Encode is needed.
Just output the raw-bytes in your handler - and set the content-type to application/json:
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
// json.NewEncoder(w).Encode(value) // not this
w.Header().Set("Content-Type", "application/json")
if bs, ok := value.([]byte); ok {
_, err := w.Write(bs) //raw bytes (already encoded in JSON)
// check 'err'
} else {
// error unexpected type behind interface{}
}
}
I was playing with go recently and stuck with a runtime error, I can't explain. These are my working functions.
type User struct {
Browsers []string `json:"browsers"`
Name string `json:"name"`
Email string `json:"email"`
}
func asyncUserProcJson(wg *sync.WaitGroup, users *[]User, ch chan []byte) {
for buf := range ch {
var mu sync.Mutex
var user User
mu.Lock()
err := json.Unmarshal(buf, &user)
mu.Unlock()
if err != nil {
fmt.Println("json:", err)
wg.Done()
continue
}
*users = append(*users, user)
wg.Done()
}
}
func userProcJson(buf []byte) (User, error) {
var user User
err := json.Unmarshal(buf, &user)
if err != nil {
return User{}, err
}
return user, nil
}
If I do a common - non-concurrent aproach, its works as expected. But if, try to use channel to pass bytes to goroutine... it fails.
type AsyncUserProc func(*sync.WaitGroup, *[]User, chan []byte)
type UserProc func(buf []byte) (User, error)
type SearchParams struct {
out io.Writer
asyncUserProc AsyncUserProc
userProc UserProc
}
func (sp SearchParams) AsyncSearch() []User {
file, err := os.Open(filePath)
if err != nil {
log.Fatalln(err)
}
var Users = make([]User, 0, 1024)
var ch = make(chan []byte)
var wg sync.WaitGroup
go sp.asyncUserProcess(&wg, &Users, ch)
scanner := bufio.NewScanner(file)
for scanner.Scan() {
wg.Add(1)
ch <- scanner.Bytes()
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
close(ch)
wg.Wait()
return Users
}
func (sp SearchParams) Search() []User {
file, err := os.Open(filePath)
if err != nil {
log.Fatalln(err)
}
// json processor
var Users = make([]User, 0, 1024)
scanner := bufio.NewScanner(file)
for scanner.Scan() {
u, err := sp.userProcess(scanner.Bytes())
if err != nil {
log.Panicln(err)
continue
}
Users = append(Users, u)
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
return Users
}
Workflow is the next one:
filePath contains a JSON chunks (each on new line)
Open for reading.
Create a line scanner
(AsyncSearch)
Pass line to channel.
return value of the line from range (blocking operation)
pass to json.Unmarshal
troubles
(Search)
Pass line directly to userProc func
Enjoy result
I am getting a lot (different) errors.
a lot of json unmarshaling error.
index out of range
JSON decoder out of sync - data changing underfoot?
as description of last error:
// phasePanicMsg is used as a panic message when we end up with something that
// shouldn't happen. It can indicate a bug in the JSON decoder, or that
// something is editing the data slice while the decoder executes.
So here is a question: How the bytes slice is modified?
I thought it was blocking operation. What am I missing in language mechanics?
Example of the errors (different each run)
json: invalid character 'i' looking for beginning of value
json: invalid character ':' after top-level value
json: invalid character 'r' looking for beginning of value
panic: runtime error: index out of range
----
json: invalid character '.' after top-level value
json: invalid character 'K' looking for beginning of value
panic: JSON decoder out of sync - data changing underfoot?
Package bufio
import "bufio"
func (*Scanner) Bytes
func (s *Scanner) Bytes() []byte
Bytes returns the most recent token generated by a call to Scan. The
underlying array may point to data that will be overwritten by a
subsequent call to Scan. It does no allocation.
The underlying array may point to data that will be overwritten by a subsequent call to Scan.
I have the following type in Golang:
type Base64Data []byte
In order to support unmarshalling a base64 encoded string to this type, I did the following:
func (b *Base64Data) UnmarshalJSON(data []byte) error {
if len(data) == 0 {
return nil
}
content, err := base64.StdEncoding.DecodeString(string(data[1 : len(data)-1]))
if err != nil {
return err
}
*b = []byte(xml)
return nil
}
Now I also want to be able to marshal and unmarshal it to mongo database, using mgo Golang library.
The problem is that I already have documents there stored as base64 encoded string, so I have to maintain that.
I tried to do the following:
func (b Base64Data) GetBSON() (interface{}, error) {
return base64.StdEncoding.EncodeToString([]byte(b)), nil
}
func (b *Base64DecodedXml) SetBSON(raw bson.Raw) error {
var s string
var err error
if err = raw.Unmarshal(&s); err != nil {
return err
}
*b, err = base64.StdEncoding.DecodeString(s)
return err
}
So that after unmarshaling, the data is already decoded, so I need to encode it back, and return it as a string so it will be written to db as a string (and vice versa)
For that I implemented bson getter and setter, but it seems only the getter is working properly
JSON unmarshaling from base64 encoded string works, as well marshaling it to database. but unmarshling setter seems to not be called at all.
Can anyone suggest what I'm missing, so that I'll be able to properly hold the data decoded in memory, but encoded string type?
This is a test I tried to run:
b := struct {
Value shared.Base64Data `json:"value" bson:"value"`
}{}
s := `{"value": "PHJvb3Q+aGVsbG88L3Jvb3Q+"}`
require.NoError(t, json.Unmarshal([]byte(s), &b))
t.Logf("%v", string(b.Value))
b4, err := bson.Marshal(b)
require.NoError(t, err)
t.Logf("%v", string(b4))
require.NoError(t, bson.Unmarshal(b4, &b))
t.Logf("%v", string(b.Value))
You can't marshal any value with bson.Marshal(), only maps and struct values.
If you want to test it, pass a map, e.g. bson.M to bson.Marshal():
var x = Base64Data{0x01, 0x02, 0x03}
dd, err := bson.Marshal(bson.M{"data": x})
fmt.Println(string(dd), err)
Your code works as-is, and as you intend it to. Try to insert a wrapper value to verify it:
c := sess.DB("testdb").C("testcoll")
var x = Base64Data{0x01, 0x02, 0x03}
if err := c.Insert(bson.M{
"data": x,
}); err != nil {
panic(err)
}
This will save the data as a string, being the Base64 encoded form.
Of course if you want to load it back into a value of type Base64Data, you will also need to define the SetBSON(raw Raw) error method too (bson.Setter interface).
In the following example from Web Development with Go by Shiju Varghese, which is for implementing a HTTP server using a new MongoDB session for each HTTP request:
Why is json package's Decode method used in PostCategory function?
Why is json package's Marshal method used in GetCategories function?
At first I thought that Decode in PostCategory and Marshal in GetCategories are opposite to each other, but later I found that there is a Unmarshal method and maybe a Encode one in the json package. So I asked a question earlier.
Here is the program
package main
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
var session *mgo.Session
type (
Category struct {
Id bson.ObjectId `bson:"_id,omitempty"`
Name string
Description string
}
DataStore struct {
session *mgo.Session
}
)
//Close mgo.Session
func (d *DataStore) Close() {
d.session.Close()
}
//Returns a collection from the database.
func (d *DataStore) C(name string) *mgo.Collection {
return d.session.DB("taskdb").C(name)
}
//Create a new DataStore object for each HTTP request
func NewDataStore() *DataStore {
ds := &DataStore{
session: session.Copy(),
}
return ds
}
//Insert a record
func PostCategory(w http.ResponseWriter, r *http.Request) {
var category Category
// Decode the incoming Category json
err := json.NewDecoder(r.Body).Decode(&category)
if err != nil {
panic(err)
}
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
//Insert record
err = c.Insert(&category)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusCreated)
}
//Read all records
func GetCategories(w http.ResponseWriter, r *http.Request) {
var categories []Category
ds := NewDataStore()
defer ds.Close()
//Getting the mgo.Collection
c := ds.C("categories")
iter := c.Find(nil).Iter()
result := Category{}
for iter.Next(&result) {
categories = append(categories, result)
}
w.Header().Set("Content-Type", "application/json")
j, err := json.Marshal(categories)
if err != nil {
panic(err)
}
w.WriteHeader(http.StatusOK)
w.Write(j)
}
func main() {
var err error
session, err = mgo.Dial("localhost")
if err != nil {
panic(err)
}
r := mux.NewRouter()
r.HandleFunc("/api/categories", GetCategories).Methods("GET")
r.HandleFunc("/api/categories", PostCategory).Methods("POST")
server := &http.Server{
Addr: ":8080",
Handler: r,
}
log.Println("Listening...")
server.ListenAndServe()
}
I think the main reason for using a json.NewDecoder here is to read directly from response's body (r.Body) here, since NewDecoder takes an io.Reader as an input.
You could have used json.Unmarshal but then you'd have to first read response body into a []byte and pass that value to json.Unmarshal. NewDecoder is more convenient here.
TL;DR — Marshal/Unmarshal take and return byte slices, while Encode/Decode do the same thing, but read the bytes from a stream such as a network connection (readers and writers).
The encoding/json package uses the Encoder and Decoder types to act on streams of data, that is, io.Reader's and io.Writer's. This means that you can take data directly from a network socket (or an HTTP body in this case which implements io.Reader) and transform it to JSON as the bytes come in. Doing it this way, we can go ahead and start processing that JSON as soon as any data is available but before we've received the whole document (on a slow network connection with a big document this could save us a lot of time, and for some streaming protocols with "infinitely sized" document streams this is absolutely necessary!)
Marshal and Unmarshal however operate on byte slices, which means that you have to have the entirety of the JSON document in memory before you can use them. In your example, the author uses Marshal because they already have a []byte slice so there's no point in constructing a buffer using the byte slice, then making an encoder that uses that buffer, then calling encode: Instead they can just let Marshal do that for them.
In reality, Marshal/Unmarshal are just convenience methods on top of Encoders and Decoders. If we look at the source for Unmarshal, we see that under the hood it's just constructing an encoder (or the internal representation of an encoder, but trust me, they're the same thing, if you want proof you can look at the Encode method source and see that it's also creating an encodeState) and then returning the output bytes:
func Marshal(v interface{}) ([]byte, error) {
e := &encodeState{}
err := e.marshal(v)
if err != nil {
return nil, err
}
return e.Bytes(), nil
}
I am trying to develop a REST service with net/http.
The service receives a JSON structure containing all the input parameters. I wonder if there is an easier and shorter way to implement the following:
func call(w http.ResponseWriter, r *http.Request) {
if err := r.ParseForm(); err != nil {
fmt.Printf("Error parsing request %s\n", err)
}
var buf []byte
buf = make([]byte, 256)
var n, err = r.Body.Read(buf)
var decoded map[string]interface{}
err = json.Unmarshal(buf[:n], &decoded)
if err != nil {
fmt.Printf("Error decoding json: %s\n", err)
}
var uid = decoded["uid"]
...
}
As you can see it requires quite a number of lines just to get to the extraction of the first parameter. Any ideas?
You don't need to call r.ParseForm if the body of the request will contain a JSON structure and you don't need any URL parameters.
You don't need the buffer either; you can use:
decoder := json.NewDecoder(r.Body)
And then:
error := decoder.Decode(decoded)
Putting it all together:
func call(w http.ResponseWriter, r *http.Request) {
values := make(map[string]interface{})
if error := json.NewDecoder(r.Body).Decode(&values); error != nil {
panic(error)
}
uid := values["uid"].(int)
}
It would be much nicer, though, if you could formally define the structure of the input that you're expecting in a struct type:
type UpdateUserInformationRequest struct {
UserId int `json:"uid"`
// other fields...
}
And use an instance of this struct instead of a more general map.