Golang json marshal and encoding give weird output - json

I am trying to customise error message for my db query . Following is what I am doing first I create struct Errormessage . Next if there is error in db.query I do this marshaling then encoding and return. But I end up getting this output "e30=" on my postman testing. What could be wrong I check and followed few examples are showing this mechanism ?
error1 := Errormessage{"Error in select"}
error1_enc,errEn := json.Marshal(error1)
if errEn != nil {
// if error is not nil
// print error
fmt.Println(errEn)
}
json.NewEncoder(w).Encode(error1_enc)
return
/
/ declaring a struct
type Errormessage struct{
// defining struct variables
errormessage string
}
func checkExistUser(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
fmt.Println("File Name :", r.FormValue("email"))
result, err := db.Query("SELECT * from userDetailsss")
if err != nil {
//http.Error(w, err, 500)
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(400)
fmt.Println(err)
error1 := Errormessage{"Error in select"}
error1_enc,errEn := json.Marshal(error1)
if errEn != nil {
// if error is not nil
// print error
fmt.Println(errEn)
}
json.NewEncoder(w).Encode(error1_enc)
return
//panic(err.Error())
}
// This part is how my db is defined and opened
var db *sql.DB
var err error
func main() {
db, err = sql.Open("mysql", "******##tcp(127.0.0.1:3306)/****")
if err != nil {
panic(err.Error())
}
defer db.Close()
router := mux.NewRouter()
router.HandleFunc("/", DoHealthCheck).Methods("POST")
router.HandleFunc("/checkExistUser", checkExistUser).Methods("POST")
log.Fatal(http.ListenAndServe(":8080", router))
}

There are two issues with your code:
You are json encoding the already json encoded error. This means that you are json encoding raw json bytes, which is the reason for the weird output.
Your Errormessage struct's field is unexported. Unexported fields will not be encoded by the encoding/json package.
To fix #1 you can do:
func checkExistUser(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
fmt.Println("File Name :", r.FormValue("email"))
result, err := db.Query("SELECT * from userDetailsss")
if err != nil {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(400)
// use only Encode, no need to call json.Marshal
if err := json.NewEncoder(w).Encode(Errormessage{"Error in select"}); err != nil {
log.Println("failed to send reposnse:", err)
}
return
}
// ...
}
To fix #2 you can do:
type Errormessage struct {
// export the field, i.e. change it to start with an upper case letter
Errormessage string `json:"errormessage"`
}

Related

Exporting JSON into single file from loop function

I wrote some code which hits one public API and saves the JSON output in a file. But the data is storing line by line into the file instead of a single JSON format.
For eg.
Current Output:
{"ip":"1.1.1.1", "Country":"US"}
{"ip":"8.8.8.8", "Country":"IN"}
Desired Output:
[
{"ip":"1.1.1.1", "Country":"US"},
{"ip":"8.8.8.8", "Country":"IN"}
]
I know this should be pretty simple and i am missing out something.
My Current Code is:
To read IP from file and hit the API one by one on each IP.
func readIPfromFile(filename string, outFile string, timeout int) {
data := jsonIn{}
//open input file
jsonFile, err := os.Open(filename) //open input file
...
...
jsonData := bufio.NewScanner(jsonFile)
for jsonData.Scan() {
// marshal json data & check for logs
if err := json.Unmarshal(jsonData.Bytes(), &data); err != nil {
log.Fatal(err)
}
//save to file
url := fmt.Sprintf("http://ipinfo.io/%s", data.Host)
GetGeoIP(url, outFile, timeout)
}
}
To make HTTP Request with custom request header and call write to file function.
func GetGeoIP(url string, outFile string, timeout int) {
geoClient := http.Client{
Timeout: time.Second * time.Duration(timeout), // Timeout after 5 seconds
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("accept", "application/json")
res, getErr := geoClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
if res.Body != nil {
defer res.Body.Close()
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
jsonout := jsonOut{}
jsonErr := json.Unmarshal(body, &jsonout)
if jsonErr != nil {
log.Fatal(jsonErr)
}
file, _ := json.Marshal(jsonout)
write2file(outFile, file)
}
To Write data to file:
func write2file(outFile string, file []byte) {
f, err := os.OpenFile(outFile, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
log.Fatal(err)
}
defer f.Close()
if _, err = f.WriteString(string(file)); err != nil {
log.Fatal(err)
}
if _, err = f.WriteString("\n"); err != nil {
log.Fatal(err)
}
I know, i can edit f.WriteString("\n"); to f.WriteString(","); to add comma but still adding [] in the file is challenging for me.
First, please do not invent a new way of json marshaling, just use golang built-in encoding/json or other library on github.
Second, if you want to create a json string that represents an array of object, you need to create the array of objects in golang and marshal it into string (or more precisely, into array of bytes)
I create a simple as below, but please DIY if possible.
https://go.dev/play/p/RR_ok-fUTb_4

Chi empty http.Request.Body in render.Bind

I am using github.com/pressly/chi to build this simple program where I try to decode some JSON from the http.Request.Body:
package main
import (
"encoding/json"
"fmt"
"net/http"
"github.com/pressly/chi"
"github.com/pressly/chi/render"
)
type Test struct {
Name string `json:"name"`
}
func (p *Test) Bind(r *http.Request) error {
err := json.NewDecoder(r.Body).Decode(p)
if err != nil {
return err
}
return nil
}
func main() {
r := chi.NewRouter()
r.Post("/products", func(w http.ResponseWriter, r *http.Request) {
var p Test
// err := render.Bind(r, &p)
err := json.NewDecoder(r.Body).Decode(&p)
if err != nil {
panic(err)
}
fmt.Println(p)
})
http.ListenAndServe(":8080", r)
}
When I don't use render.Bind() (from "github.com/pressly/chi/render"), it works as expected.
However, when I uncomment the line err := render.Bind(r, &p) and I comment the line err := json.NewDecoder(r.Body).Decode(&p), it panics with EOF :
2017/06/20 22:26:39 http: panic serving 127.0.0.1:39696: EOF
and thus the json.Decode() fails.
Am I doing something wrong or is the http.Request.Body is already read somewhere else before render.Bind() is called?
render.Bind's purpose is to perform decode and execute Bind(r) to do post decode operations.
For eg.:
type Test struct {
Name string `json:"name"`
}
func (p *Test) Bind(r *http.Request) error {
// At this point, Decode is already done by `chi`
p.Name = p.Name + " after decode"
return nil
}
If you have to do only JSON decode no other actions needs to be done after decode with respect to decoded values. Just use:
// Use Directly JSON decoder of std pkg
err := json.NewDecoder(r.Body).Decode(&p)
OR
// Use wrapper method from chi DecodeJSON
err := render.DecodeJSON(r.Body, &p)

Golang Encode/Decode base64 with json post doesn't work

I build a client and a server in golang both are using this functions to encrypt/decrypt
func encrypt(text []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
b := base64.StdEncoding.EncodeToString(text)
ciphertext := make([]byte, aes.BlockSize+len(b))
iv := ciphertext[:aes.BlockSize]
if _, err := io.ReadFull(rand.Reader, iv); err != nil {
return nil, err
}
cfb := cipher.NewCFBEncrypter(block, iv)
cfb.XORKeyStream(ciphertext[aes.BlockSize:], []byte(b))
return ciphertext, nil
}
func decrypt(text []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
if len(text) < aes.BlockSize {
return nil, errors.New("ciphertext too short")
}
iv := text[:aes.BlockSize]
text = text[aes.BlockSize:]
cfb := cipher.NewCFBDecrypter(block, iv)
cfb.XORKeyStream(text, text)
data, err := base64.StdEncoding.DecodeString(string(text))
if err != nil {
return nil, err
}
return data, nil
}
so yeah I make a normal post request
url := "https://"+configuration.Server+configuration.Port+"/get"
// TODO maybe bugs rest here
ciphertext, err := encrypt([]byte(*getUrl))
if err != nil {
fmt.Println("Error: " + err.Error())
}
fmt.Println(string(ciphertext))
values := map[string]interface{}{"url": *getUrl, "urlCrypted": ciphertext}
jsonValue, _ := json.Marshal(values)
jsonStr := bytes.NewBuffer(jsonValue)
req, err := http.NewRequest("POST", url, jsonStr)
and the servercode is as following
requestContent := getRequestContentFromRequest(req)
url := requestContent["url"].(string)
undecryptedUrl := requestContent["urlCrypted"].(string)
decryptedurl, err := decrypt([]byte(undecryptedUrl))
if err != nil {
fmt.Println("Error: " + err.Error())
}
fmt.Println(decryptedurl)
where getRequestContentFromRequest is as following
func getRequestContentFromRequest(req *http.Request)
map[string]interface{} {
buf := new(bytes.Buffer)
buf.ReadFrom(req.Body)
data := buf.Bytes()
var requestContent map[string]interface{}
err := json.Unmarshal(data, &requestContent)
if err != nil {
fmt.Println(err)
}
return requestContent
}
Now to the problem.
If I encrypt my string in the client and decrypt it direct after that everything is fine.
But, when I send the encrypted string to the server and try to decrypt it with literrally the same function as in the client, the decrypt function throws an error.
Error: illegal base64 data at input byte 0
I think the Problem is the unmarshalling of the JSON.
Thanks for help.
P.S.
Repos are
github.com/BelphegorPrime/goSafeClient and github.com/BelphegorPrime/goSafe
UPDATE
Example JSON
{"url":"facebook2.com","urlCrypted":"/}\ufffd\ufffd\ufffdgP\ufffdN뼞\ufffd\u0016\ufffd)\ufffd\ufffd\ufffdy\u001c\u000f\ufffd\ufffd\ufffdep\ufffd\rY\ufffd\ufffd$\ufffd\ufffd"}
UPDATE2
I made a playground here
The problem is that you encode in base64 twice. The first time in the encrypt function and the second time during the JSON marshalling. byte slices are automatically converted into base64 strings by the encoding/json marshaller.
The solution is to decode the base64 string before calling decrypt.
Example on the Go PlayGround
EDIT
Working solution here

How to unmarshall JSON into a value created with reflection?

package controllers
import (
"encoding/json"
"errors"
"io"
"io/ioutil"
"reflect"
)
func GetTypeFromReq(c *App, ty interface{}) (interface{}, error) {
//get the type we are going to marshall into
item := reflect.ValueOf(ty)
//define and set the error that we will be returning to null
var retErr error
retErr = nil
//extract the body from the request and defer closing of the body
body, err := ioutil.ReadAll(io.LimitReader(c.Request.Body, 1048576))
defer c.Request.Body.Close()
//handle errors and unmarshal our data
if err != nil {
retErr = errors.New("Failed to Read body: " + err.Error())
} else if err = json.Unmarshal(body, &item); err != nil {
retErr = errors.New("Unmarshal Failed: " + err.Error())
}
return item, retErr
}
I am trying to pass a type and a request into a function, then inside that function unMarshall the request into a variable and return it.
I assume my approach is wrong because when i try to do this:
inter, err := GetTypeFromReq(&c, models.User{})
if err != nil {
revel.ERROR.Println(err.Error())
}
user := inter.(models.User)
I get the error "interface conversion: interface {} is reflect.Value, not models.User"
any tips on how to approach this?
Here's how to modify the the function to make it work as expected:
func GetTypeFromReq(c *App, ty interface{}) (interface{}, error) {
// Allocate new value with same type as ty
v := reflect.New(reflect.TypeOf(ty))
//define and set the error that we will be returning to null
var retErr error
retErr = nil
//extract the body from the request and defer closing of the body
body, err := ioutil.ReadAll(io.LimitReader(c.Request.Body, 1048576))
defer c.Request.Body.Close()
//handle errors and unmarshal our data
if err != nil {
retErr = errors.New("Failed to Read body: " + err.Error())
} else if err = json.Unmarshal(body, v.Interface()); err != nil {
retErr = errors.New("Unmarshal Failed: " + err.Error())
}
// v holds a pointer, call Elem() to get the value.
return v.Elem().Interface(), retErr
}
Note the calls to Interface() to get a reflect.Value's current value.
Here's an approach that avoids reflection and type assertions:
func GetFromReq(c *App, item interface{}) error {
//extract the body from the request and defer closing of the body
body, err := ioutil.ReadAll(io.LimitReader(c.Request.Body, 1048576))
defer c.Request.Body.Close()
//handle errors and unmarshal our data
if err != nil {
retErr = errors.New("Failed to Read body: " + err.Error())
} else if err = json.Unmarshal(body, item); err != nil {
retErr = errors.New("Unmarshal Failed: " + err.Error())
}
return retErr
}
Use it like this:
var user models.User
err := GetFromReq(&c, &user)
if err != nil {
revel.ERROR.Println(err.Error())
}
Use a JSON decoder to simplify the code:
func GetFromReq(c *App, item interface{}) error {
defer c.Request.Body.Close()
return json.NewDecoder(io.LimitReader(c.Request.Body, 1048576)).Deocode(item)
}
If c.Request is a *http.Request and c.Response is an http.ResponseWriter, then write the function as:
func GetFromReq(c *App, item interface{}) error {
return json.NewDecoder(http.MaxBytesReaer(c.Response, c.Request.Body, 1048576)).Deocode(item)
}
There's no need to close the request body in the net/http server. Use MaxBytesReader instead of io.LimitReader to prevents clients from accidentally or maliciously sending a large request and wasting server resources.
Modify code of the last line: change user := inter.(models.User) to user := inter.Interface().(models.User),have a try!
"interface conversion: interface {} is reflect.Value, not models.User"
pretty straight forward about the message error. That your item is reflect.Value it is not models.User.
so I think in your code you can change the item to models.User.
But I assume that your are tying to create the function that will work with all type of your models, in this case models.User{}.
Your approach is expensive since it is using interface. you could convert the incoming request directly like this:
func GetTypeFromReq(c *App, ty models.User) (models.User, error) {
//get the type we are going to marshall into
var item models.User
//define and set the error that we will be returning to nil
var retErr error // this var if the value not define then it is nil. Because error is interface
//extract the body from the request and defer closing of the body
body, err := ioutil.ReadAll(io.LimitReader(c.Request.Body, 1048576))
defer c.Request.Body.Close()
//handle errors and unmarshal our data
if err != nil {
retErr = errors.New("Failed to Read body: " + err.Error())
} else if err = json.Unmarshal(body, &item); err != nil {
retErr = errors.New("Unmarshal Failed: " + err.Error())
}
return item, retErr
}
if your body has the same structure as your model it will give you the value, if not then it is error.
Note that you need to be careful when using interface. you can see some guideline in this article. Use an interface:
When users of the API need to provide an implementation detail.
When API’s have multiple implementations they need to maintain internally.
When parts of the API that can change have been identified and require decoupling.
Your function convert the value of your models.User to interface, and then return the interface value. that's why it's expensive.

Golang json query from bitcoin api returns invalid character

Something tells me I'm not understanding json correctly. I'm trying to grab some data off http://api.bitcoincharts.com/v1/trades.csv?symbol=rockUSD, but my Unmarshal seems to not be able to read the json data. I'm a fresh beginner to golang (and json as well), and I'm wondering how I am able to skip that wrong character error I'm making.
My error:
invalid character ',' after top-level value
panic: invalid character ',' after top-level value
My code:
package main
import ("fmt"
"net/http"
"io/ioutil"
"encoding/json"
)
type Prices struct {
Data string
}
func main() {
url := "http://api.bitcoincharts.com/v1/trades.csv?symbol=rockUSD"
httpresp, err := http.Get(url)
if err != nil{
fmt.Println(err)
panic(err)
}
defer httpresp.Body.Close()
htmldata, err := ioutil.ReadAll(httpresp.Body)
if err != nil{
fmt.Println(err)
panic (err)
}
var jsonData []Prices
err = json.Unmarshal([]byte(htmldata), &jsonData)
if err != nil {
fmt.Println(err)
panic (err)
}
fmt.Println(jsonData)
}
That is NOT json data at all, you'd have to write a custom parser.
Example:
.........
data := readData(httpresp.Body)
........
func readData(r io.Reader) (out [][3]float64) {
br := bufio.NewScanner(r)
for br.Scan() {
parts := strings.Split(br.Text(), ",")
if len(parts) != 3 {
continue
}
var fparts [3]float64
for i, p := range parts {
// bad idea to ignore errors, but it's left as exercise for the reader.
fparts[i], _ = strconv.ParseFloat(p, 64)
}
out = append(out, fparts)
}
return
}
playground