I am new to go and backend development. I am trying to use a MySQL database with go to create a REST API.
func getUsers(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306/test)")
if err != nil {
panic(err.Error())
}
results, err := db.Query("Select * from users")
if err != nil {
panic(err.Error())
}
for results.Next() {
var user User
err = results.Scan(&user.FirstName)
if err != nil {
panic(err.Error)
}
fmt.Println(user.FirstName)
}
//scores is an array that i have already created just to return dummy data
json.NewEncoder(w).Encode(scores)
}
I get this error:
http: panic serving [::1]:54508: invalid DSN: network address not terminated (missing closing brace)
goroutine 5 [running]:
net/http.(*conn).serve.func1(0xc42009abe0)
/usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0x129cea0, 0xc420010ec0)
/usr/local/go/src/runtime/panic.go:505 +0x229
main.getUsers(0x134bca0, 0xc42011e000, 0xc42011c200)
/Users/tushar/go/src/github.com/tushar/jump/main.go:62 +0x33f
net/http.HandlerFunc.ServeHTTP(0x1326230, 0x134bca0, 0xc42011e000, 0xc42011c200)
/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc420110000, 0x134bca0, 0xc42011e000, 0xc42011c200)
/Users/tushar/go/src/github.com/gorilla/mux/mux.go:162 +0xed
net/http.serverHandler.ServeHTTP(0xc42008aea0, 0x134bca0, 0xc42011e000, 0xc42011c000)
/usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc42009abe0, 0x134bf60, 0xc420062240)
/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x27b
I am not able to understand the issue.
It works fine when there is no DB query.
EDIT 1:
editted sql.Open to change closing braces to root:mypassword#tcp(127.0.0.1:3306)/test
works fine when querying
but i get panic while inserting in db
func insertUsers(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306)/test")
if err != nil {
panic(err.Error())
}
insert, err := db.Query("Insert into users (personId,firstName,lastName) values(20,tushar,saha)")
if err != nil {
panic(err.Error)
}
defer insert.Close()
}
this is the error 1
2018/06/23 11:54:10 http: panic serving [::1]:54802: 0x126a8b0
goroutine 19 [running]:
net/http.(*conn).serve.func1(0xc4200aebe0)
/usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0x129bb20, 0xc42016a020)
/usr/local/go/src/runtime/panic.go:505 +0x229
main.insertUsers(0x134bc80, 0xc42013c000, 0xc420138200)
/Users/tushar/go/src/github.com/tushar/jump/main.go:50 +0x15d
net/http.HandlerFunc.ServeHTTP(0x1326218, 0x134bc80, 0xc42013c000, 0xc420138200)
/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc42012c000, 0x134bc80, 0xc42013c000, 0xc420138200)
/Users/tushar/go/src/github.com/gorilla/mux/mux.go:162 +0xed
net/http.serverHandler.ServeHTTP(0xc420095040, 0x134bc80, 0xc42013c000, 0xc420138000)
/usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc4200aebe0, 0x134bf40, 0xc42009a200)
/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x27b
The DSN (i.e. second) argument to the sql.Open() method is malformed.
Only the address portion (127.0.0.1:3306) is supposed to be surrounded by parentheses. You've included the database name within the parentheses as well.
Try this instead:
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306)/test")
The documented format is:
[username[:password]#][protocol[(address)]]/dbname[?param1=value1&...¶mN=valueN]
In response to "EDIT 1"
err.Error() is a method.
In your error handling block for the insert query, you're not calling the method. Instead you're printing out it's memory location.
if err != nil {
panic(err.Error)
}
If you actually call the method (similar to how you have in the error handling block for the sql.Open()), you'll print out the error message which can point you towards what's wrong.
if err != nil {
panic(err.Error())
}
Related
I have a Golang program with a context deadline set. I am sending an HTTP request, and expected to see a deadline exceeded error when Im reading the body.
It seems that when I read the response body with ioutil.ReadAll then that read method will get interrupted (?) and return the appropriate error (context.DeadlineExceeded).
However if I read the response body with json.NewDecoder(resp.Body).Decode then the error returned is nil (instead of context.DeadlineExceeded). My full code is below. Is this a bug in json.NewDecoder(resp.Body).Decode?
package main
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"time"
)
var url string = "http://ip.jsontest.com/"
func main() {
readDoesntFail()
readFails()
}
type IpResponse struct {
Ip string
}
func readDoesntFail() {
ctx, _ := context.WithTimeout(context.Background(), time.Second*5)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
ipResponse := new(IpResponse)
time.Sleep(time.Second * 6)
fmt.Println("before reading response body, context error is:", ctx.Err())
err = json.NewDecoder(resp.Body).Decode(ipResponse)
if err != nil {
panic(err)
}
fmt.Println("Expected panic but there was none")
}
func readFails() {
ctx, _ := context.WithTimeout(context.Background(), time.Second*5)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
time.Sleep(time.Second * 6)
fmt.Println("before reading response body, context error is:", ctx.Err())
_, err = ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println("received expected error", err)
}
}
The net/http package may use buffers to process requests. This means the incoming response body may be read and buffered partly or entirely before you read it, so an expiring context may not prevent you to finish reading the body. And this is exactly what happens.
Let's modify your example to fire up a test HTTP server which deliberately delays the response (partly):
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s := []byte(`{"ip":"12.34.56.78"}`)
w.Write(s[:10])
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
time.Sleep(time.Second * 6)
w.Write(s[10:])
}))
defer ts.Close()
url = ts.URL
readDoesntFail()
readFails()
This test server sends a similar JSON object to that of ip.jsontest.com's response. But it only sends 10 bytes body, then flushes it, then sleeps 6 seconds on purpose before sending the rest, "allowing" the client to time out.
Now let's see what happens if we call readDoesntFail():
before reading response body, context error is: context deadline exceeded
panic: Get "http://127.0.0.1:38230": context deadline exceeded
goroutine 1 [running]:
main.readDoesntFail()
/tmp/sandbox721114198/prog.go:46 +0x2b4
main.main()
/tmp/sandbox721114198/prog.go:28 +0x93
Try it on the Go Playground.
In your example json.Decoder.Decode() reads already buffered data, so the expired context plays no role here. In my example json.Decoder.Decode() tries to read from the connection because the data isn't yet buffered (it can't be as it hasn't been sent yet), so once the context expires, further reading from the connection returns a deadline exceeded error.
I wrote some code which hits one public API and saves the JSON output in a file. But the data is storing line by line into the file instead of a single JSON format.
For eg.
Current Output:
{"ip":"1.1.1.1", "Country":"US"}
{"ip":"8.8.8.8", "Country":"IN"}
Desired Output:
[
{"ip":"1.1.1.1", "Country":"US"},
{"ip":"8.8.8.8", "Country":"IN"}
]
I know this should be pretty simple and i am missing out something.
My Current Code is:
To read IP from file and hit the API one by one on each IP.
func readIPfromFile(filename string, outFile string, timeout int) {
data := jsonIn{}
//open input file
jsonFile, err := os.Open(filename) //open input file
...
...
jsonData := bufio.NewScanner(jsonFile)
for jsonData.Scan() {
// marshal json data & check for logs
if err := json.Unmarshal(jsonData.Bytes(), &data); err != nil {
log.Fatal(err)
}
//save to file
url := fmt.Sprintf("http://ipinfo.io/%s", data.Host)
GetGeoIP(url, outFile, timeout)
}
}
To make HTTP Request with custom request header and call write to file function.
func GetGeoIP(url string, outFile string, timeout int) {
geoClient := http.Client{
Timeout: time.Second * time.Duration(timeout), // Timeout after 5 seconds
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("accept", "application/json")
res, getErr := geoClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
if res.Body != nil {
defer res.Body.Close()
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
jsonout := jsonOut{}
jsonErr := json.Unmarshal(body, &jsonout)
if jsonErr != nil {
log.Fatal(jsonErr)
}
file, _ := json.Marshal(jsonout)
write2file(outFile, file)
}
To Write data to file:
func write2file(outFile string, file []byte) {
f, err := os.OpenFile(outFile, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
log.Fatal(err)
}
defer f.Close()
if _, err = f.WriteString(string(file)); err != nil {
log.Fatal(err)
}
if _, err = f.WriteString("\n"); err != nil {
log.Fatal(err)
}
I know, i can edit f.WriteString("\n"); to f.WriteString(","); to add comma but still adding [] in the file is challenging for me.
First, please do not invent a new way of json marshaling, just use golang built-in encoding/json or other library on github.
Second, if you want to create a json string that represents an array of object, you need to create the array of objects in golang and marshal it into string (or more precisely, into array of bytes)
I create a simple as below, but please DIY if possible.
https://go.dev/play/p/RR_ok-fUTb_4
I am trying to customise error message for my db query . Following is what I am doing first I create struct Errormessage . Next if there is error in db.query I do this marshaling then encoding and return. But I end up getting this output "e30=" on my postman testing. What could be wrong I check and followed few examples are showing this mechanism ?
error1 := Errormessage{"Error in select"}
error1_enc,errEn := json.Marshal(error1)
if errEn != nil {
// if error is not nil
// print error
fmt.Println(errEn)
}
json.NewEncoder(w).Encode(error1_enc)
return
/
/ declaring a struct
type Errormessage struct{
// defining struct variables
errormessage string
}
func checkExistUser(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
fmt.Println("File Name :", r.FormValue("email"))
result, err := db.Query("SELECT * from userDetailsss")
if err != nil {
//http.Error(w, err, 500)
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(400)
fmt.Println(err)
error1 := Errormessage{"Error in select"}
error1_enc,errEn := json.Marshal(error1)
if errEn != nil {
// if error is not nil
// print error
fmt.Println(errEn)
}
json.NewEncoder(w).Encode(error1_enc)
return
//panic(err.Error())
}
// This part is how my db is defined and opened
var db *sql.DB
var err error
func main() {
db, err = sql.Open("mysql", "******##tcp(127.0.0.1:3306)/****")
if err != nil {
panic(err.Error())
}
defer db.Close()
router := mux.NewRouter()
router.HandleFunc("/", DoHealthCheck).Methods("POST")
router.HandleFunc("/checkExistUser", checkExistUser).Methods("POST")
log.Fatal(http.ListenAndServe(":8080", router))
}
There are two issues with your code:
You are json encoding the already json encoded error. This means that you are json encoding raw json bytes, which is the reason for the weird output.
Your Errormessage struct's field is unexported. Unexported fields will not be encoded by the encoding/json package.
To fix #1 you can do:
func checkExistUser(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
fmt.Println("File Name :", r.FormValue("email"))
result, err := db.Query("SELECT * from userDetailsss")
if err != nil {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(400)
// use only Encode, no need to call json.Marshal
if err := json.NewEncoder(w).Encode(Errormessage{"Error in select"}); err != nil {
log.Println("failed to send reposnse:", err)
}
return
}
// ...
}
To fix #2 you can do:
type Errormessage struct {
// export the field, i.e. change it to start with an upper case letter
Errormessage string `json:"errormessage"`
}
I have a gin application that receives a post request containing a csv file which I want to read without saving it. I'm stuck here trying to read from the post request with the following error message: cannot use file (variable of type *multipart.FileHeader) as io.Reader value in argument to csv.NewReader: missing method Read
file, err := c.FormFile("file")
if err != nil {
errList["Invalid_body"] = "Unable to get request"
c.JSON(http.StatusUnprocessableEntity, gin.H{
"status": http.StatusUnprocessableEntity,
"error": errList,
})
}
r := csv.NewReader(file) // <= Error message
records, err := r.ReadAll()
for _, record := range records {
fmt.Println(record)
}
Is there a good example that I could use?
first read the file and header
csvPartFile, csvHeader, openErr := r.FormFile("file")
if openErr != nil {
// handle error
}
then read the lines from the file
csvLines, readErr := csv.NewReader(csvPartFile).ReadAll()
if readErr != nil {
//handle error
}
you can go through the lines looping through the records
for _, line := range csvLines {
fmt.Println(line)
}
As other answers have mentioned, you should Open() it first.
The latest version of gin.Context.FromFile(string) seems to return only two values.
This worked for me:
func (c *gin.Context) {
file_ptr, err := c.FormFile("file")
if err != nil {
log.Println(err.Error())
c.Status(http.StatusUnprocessableEntity)
return
}
log.Println(file_ptr.Filename)
file, err := file_ptr.Open()
if err != nil {
log.Println(err.Error())
c.Status(http.StatusUnprocessableEntity)
return
}
defer file.Close()
records, err := csv.NewReader(file).ReadAll()
if err != nil {
log.Println(err.Error())
c.Status(http.StatusUnprocessableEntity)
return
}
for _, line := range records {
fmt.Println(line)
}
}
Issue description
I used DB.Begin() to start a transaction and DB.Commit() or DB.Rollback() to end transaction, but I found some connections in the status of "SLEEP" for quite a long time (> 2 hours). And this is not expected. So do I miss something to make sure the connections are all to be freed?
Example code
mgr.statsConn, err = sql.Open("mysql", statsConnStr)
// ......
tx, err := mgr.statsConn.Begin()
if err != nil {
LogError(FAIL_CRITICAL, DB, "Begin transaction failed:%v", err)
return err
}
defer tx.Rollback()
// ...
// multiple Exec clauses just like:
rst, err = tx.Exec(sqlStr)
if err != nil {
LogInfo("Update xxx error. Sql:%s, Error:%v", sqlStr, err)
return err
}
// ...
if err := tx.Commit(); err != nil {
LogError(FAIL_CRITICAL, DB, "Commit error:%v", err)
return err
}
sql.Open() will be called only once of the entire lifetime of the process. and mgr.statsConn will be used everytime when opening a transaction
I'm wondering the right position for the tx.Rollback(). If error happens, is it necessary to call tx.Rollback()? Will tx be non-nill then?
Configuration
Go version:1.7.5
Server version: MYSQL 5.5.37-enterprise-commercial-advanced-log