Issue with connections pool on mysql - mysql

I'm having some issues with the api I'm developing: sometimes (yes, not always) when I make a request to the golang server from my angular app, it gaves me this error: "sql: database is closed" when I'm tring to execute "QueryContext", but I figured that it happens more frequently on a func that request a larger data from database (200 record top).
Is there a way to check if the connection is still open\valid? shouldn't golang's connection pool do it automatically? (I have other api more "light" in the same server with the same database and everything work smootly)
Is there any MySql setting I should change?(mysql has defaullt settings)
golang version: 1.16,
Mysql 8.0.17
Hre is an example of my code:
on package database.go
func OpenConnection() (*sql.DB, error)
{
connection, err = sql.Open("mysql", "root#/my_database")
if err != nil {
log.Println("Error opening the connection with the database")
return nil, err
}
return connection, nil
}
On main.go
func main() {
---
http.HandleFunc("/apicall1", customFunc)
http.HandleFunc("/apicall2", customFunc)
http.HandleFunc("/apicall3", customFunc)
}
func customFunc(w http.ResponseWriter, r *http.Request) {
conn, err := database.OpenConnection()
if err != nil {
//handle error 500 response
}
defer conn.Close()
switch(r.URL.Path) {
case "url1": my_package.Func1(conn)
case "url2": my_package.Func2(conn)
case "url3": my_package.Func3(conn)
...
default: //handle not found response
}
}

You need to configure your sql.DB for better performance from your pool and to avoid hitting max limit of your db allowed connection, here as an example I have made max 100 connection and idle 25 you can change it as per the load. Also it's a nice idea to ping db after making successful connection, just to ensure connection is success.
func OpenConnection() (*sql.DB, error)
{
connection, err = sql.Open("mysql", "root#/my_database")
if err != nil {
log.Println("Error opening the connection with the database")
return nil, err
}
connection.DB().SetMaxIdleConns(25)
connection.DB().SetMaxOpenConns(100)
dberr = connection.DB().Ping()
if dberr != nil {
log.Println("failed to ping db repository : %s", dberr)
return nil, dberr
}
return connection, nil
}
Reference - https://www.alexedwards.net/blog/configuring-sqldb

Related

How to set up loop in database until it connects to mysqlDB

I have init funcion in my golang program. I need to make it so the connection loops until it connects to database if it cant connect it idles then try's again and again for 60 seconds if it cant connect at end then it exist out. Also, have environment variable that overrides the default 60 seconds and user can put their own time for until it connects. I have my code below im looking and theres no solid solution in web .
var override string = os.Getenv("OVERRIDE")
func dsn() string {
return fmt.Sprintf("%s:%s#tcp(%s:%s)/%s", username, password, ipdb, portdb, dbName)
func init() {
var db, err = sql.Open("mysql", dsn())
if err != nil {
fmt.Println(err.Error())
} else {
fmt.Printf("Connection established to MYSQL server\n%s\n", dsn())
}
// Checking if table exists in database
rows, table_check := db.Query("select * from " + "Url" + ";")
if table_check == nil {
fmt.Println("\nTable exists in the Database")
count := 0
for rows.Next() {
count++
}
fmt.Printf("Number of records are %d \n", count)
} else {
fmt.Printf("\nTable does not exist in Database %s", table_check)
_, status := db.Exec("CREATE TABLE Url (LongUrl varchar(255),ShortUrl varchar(32));")
if status == nil {
fmt.Println("\n New table created in the Database")
} else {
fmt.Printf("\n The table was not created %s", status)
}
}
defer db.Close()
}
func main(){
......
}
You could use a switch with a timer channel.
For every 500 ms the connection is attempted, and if in 60 seconds the switch has not been resolved then the timeout channel exits the loop.
For example:
func main() {
doneCh := make(chan bool)
db, err := sql.Open("mysql", dsn())
if err != nil {
fmt.Errorf("Could not open connection: %w", err)
os.Exit(1)
}
// Start polling for connection
go func() {
ticker := time.NewTicker(500 * time.Millisecond)
timeoutTimer := time.After(time.Second * 60)
for {
select {
case <-timeoutTimer:
err = errors.New("connection to db timed out")
doneCh <- true
return
case <-ticker.C:
// Simulate some strange wait ... :)
if rand.Intn(10) == 2 {
err = db.Ping()
if err == nil {
doneCh <- true
return
}
}
fmt.Println("Retrying connection ... ")
}
}
}()
// Start waiting for timeout or success
fmt.Println("Waiting for DB connection ...")
<-doneCh
if err != nil {
fmt.Printf("Could not connect to DB: %v \n", err)
os.Exit(1)
}
defer db.Close()
// Do some work
fmt.Printf("Done! Do work now. %+v", db.Stats())
}
Note
Handle your errors as necessary. The sql.Ping() may return you some specific error which will indicate to you that you should retry.
Some errors may not be worth retrying with.
For timers and channels see:
Timers
Channels
Edit: Supposedly simpler way without channels and using a while loop
func main() {
db, err := sql.Open("mysql", dsn())
if err != nil {
fmt.Errorf("Could not open connection: %w", err)
os.Exit(1)
}
defer db.Close()
timeOut := time.Second * 60
idleDuration := time.Millisecond * 500
start := time.Now()
for time.Since(start) < timeOut {
if err = db.Ping(); err == nil {
break
}
time.Sleep(idleDuration)
fmt.Println("Retrying to connect ...")
}
if time.Since(start) > timeOut {
fmt.Printf("Could not connect to DB, timeout after %s", timeOut)
os.Exit(1)
}
// Do some work
fmt.Printf("Done! \nDo work now. %+v", db.Stats())
}

Error while creating connection with phpmyadmin using golang "commands out of sync. Did you run multiple statements at once?"

I'm facing some issue while fetching the data from the MySQL database using golang below is my code and the error that I'm facing
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
func ConnectMsqlDb() (db *sql.DB, err error) {
db, err = sql.Open("mysql", fmt.Sprintf("%s:%s#tcp(%s:"+sqlDbPort+")/"+sqlDB, sqlUserName, sqlPassword, dbServerIP))
if err != nil {
return nil, err
}
//defer db.Close()
err = db.Ping()
if err != nil {
return nil, err
}
return db, nil
}
func GetSqlData() (err error, data interface{}) {
db, err := ConnectMsqlDb()
if err != nil {
// here it will returning me the error
return err, nil
}
rows, err := db.Query("SELECT * FROM users")
if err != nil {
return err, nil
}
for rows.Next() {
}
defer db.Close()
fmt.Println(rows)
return err, rows
}
func main() {
err, data := GetSqlData()
fmt.Println("data", data, "err", err)
}
error
data commands out of sync. Did you run multiple statements at once?
Can anyone tell me why I'm facing this issue
If the error is comming while opening a connection to mysqld , it could be possible that MySQL server (mysqld) is blocking the host from connecting to it. It means that mysqld has received many connection requests from the given host that were interrupted in the middle.
Read why? To confirm you could see the DB's logs as well. So, a way to unblock it is to flush the cached hosts using the MySQL CLI:
mysql> FLUSH HOSTS;
And then try again to connect.
Plus, give this answer a read as well. Might help.
You can check the state of the socket used by mysqld using (For Linux):
netstat -nt
Check if there's any previously hanging connection from the host to mysqld. If yes, then kill it and try again.

Invalid DSN: network address not terminated (missing closing brace)

I am new to go and backend development. I am trying to use a MySQL database with go to create a REST API.
func getUsers(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306/test)")
if err != nil {
panic(err.Error())
}
results, err := db.Query("Select * from users")
if err != nil {
panic(err.Error())
}
for results.Next() {
var user User
err = results.Scan(&user.FirstName)
if err != nil {
panic(err.Error)
}
fmt.Println(user.FirstName)
}
//scores is an array that i have already created just to return dummy data
json.NewEncoder(w).Encode(scores)
}
I get this error:
http: panic serving [::1]:54508: invalid DSN: network address not terminated (missing closing brace)
goroutine 5 [running]:
net/http.(*conn).serve.func1(0xc42009abe0)
/usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0x129cea0, 0xc420010ec0)
/usr/local/go/src/runtime/panic.go:505 +0x229
main.getUsers(0x134bca0, 0xc42011e000, 0xc42011c200)
/Users/tushar/go/src/github.com/tushar/jump/main.go:62 +0x33f
net/http.HandlerFunc.ServeHTTP(0x1326230, 0x134bca0, 0xc42011e000, 0xc42011c200)
/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc420110000, 0x134bca0, 0xc42011e000, 0xc42011c200)
/Users/tushar/go/src/github.com/gorilla/mux/mux.go:162 +0xed
net/http.serverHandler.ServeHTTP(0xc42008aea0, 0x134bca0, 0xc42011e000, 0xc42011c000)
/usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc42009abe0, 0x134bf60, 0xc420062240)
/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x27b
I am not able to understand the issue.
It works fine when there is no DB query.
EDIT 1:
editted sql.Open to change closing braces to root:mypassword#tcp(127.0.0.1:3306)/test
works fine when querying
but i get panic while inserting in db
func insertUsers(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306)/test")
if err != nil {
panic(err.Error())
}
insert, err := db.Query("Insert into users (personId,firstName,lastName) values(20,tushar,saha)")
if err != nil {
panic(err.Error)
}
defer insert.Close()
}
this is the error 1
2018/06/23 11:54:10 http: panic serving [::1]:54802: 0x126a8b0
goroutine 19 [running]:
net/http.(*conn).serve.func1(0xc4200aebe0)
/usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0x129bb20, 0xc42016a020)
/usr/local/go/src/runtime/panic.go:505 +0x229
main.insertUsers(0x134bc80, 0xc42013c000, 0xc420138200)
/Users/tushar/go/src/github.com/tushar/jump/main.go:50 +0x15d
net/http.HandlerFunc.ServeHTTP(0x1326218, 0x134bc80, 0xc42013c000, 0xc420138200)
/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc42012c000, 0x134bc80, 0xc42013c000, 0xc420138200)
/Users/tushar/go/src/github.com/gorilla/mux/mux.go:162 +0xed
net/http.serverHandler.ServeHTTP(0xc420095040, 0x134bc80, 0xc42013c000, 0xc420138000)
/usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc4200aebe0, 0x134bf40, 0xc42009a200)
/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x27b
The DSN (i.e. second) argument to the sql.Open() method is malformed.
Only the address portion (127.0.0.1:3306) is supposed to be surrounded by parentheses. You've included the database name within the parentheses as well.
Try this instead:
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306)/test")
The documented format is:
[username[:password]#][protocol[(address)]]/dbname[?param1=value1&...&paramN=valueN]
In response to "EDIT 1"
err.Error() is a method.
In your error handling block for the insert query, you're not calling the method. Instead you're printing out it's memory location.
if err != nil {
panic(err.Error)
}
If you actually call the method (similar to how you have in the error handling block for the sql.Open()), you'll print out the error message which can point you towards what's wrong.
if err != nil {
panic(err.Error())
}

Golang MySQL error - packets.go:33: unexpected EOF

I am switching my entire code base from PHP to Go and during several processes that run, I randomly get this error:
[mysql] 2016/10/11 09:17:16 packets.go:33: unexpected EOF
Here is my db package that handles all connections to the database:
package db
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"pkg/db"
)
var connection *sql.DB
var err error
func GetConnection() *sql.DB {
if connection != nil {
fmt.Println("********** CHECKING PING")
err = connection.Ping()
if err == nil {
fmt.Println("************ CONNECTION STILL ACTIVE")
return connection
} else {
fmt.Println("********** PING ERROR: " + err.Error())
}
}
connection, err = sql.Open("mysql", db.DEVUSER + ":" + db.DEVUSER_PASSWORD + "#tcp(localhost:3306)/main?parseTime=true")
if err != nil {
panic(err)
}
return connection
}
Is there anything I'm doing wrong with this db package that causes this error to be thrown? What exactly does this error mean? I make sure to return the current connection if there is one open so for multiple requests it uses the same connection object.
Here's an excerpt from the mysql packets.go:
// Read packet to buffer 'data'
func (mc *mysqlConn) readPacket() ([]byte, error) {
var payload []byte
for {
// Read packet header
data, err := mc.buf.readNext(4)
if err != nil {
errLog.Print(err)
mc.Close()
return nil, driver.ErrBadConn
}
// Packet Length [24 bit]
pktLen := int(uint32(data[0]) | uint32(data[1])<<8 | uint32(data[2])<<16)
if pktLen < 1 {
errLog.Print(ErrMalformPkt)
mc.Close()
return nil, driver.ErrBadConn
}
// Check Packet Sync [8 bit]
if data[3] != mc.sequence {
if data[3] > mc.sequence {
return nil, ErrPktSyncMul
}
return nil, ErrPktSync
}
mc.sequence++
// Read packet body [pktLen bytes]
data, err = mc.buf.readNext(pktLen)
if err != nil {
errLog.Print(err)
mc.Close()
return nil, driver.ErrBadConn
}
isLastPacket := (pktLen < maxPacketSize)
// Zero allocations for non-splitting packets
if isLastPacket && payload == nil {
return data, nil
}
payload = append(payload, data...)
if isLastPacket {
return payload, nil
}
}
}
The first "errLog.Print(err)" is line 33 in the "Read packet header" section.
Any help is greatly appreciated!
I added a few log.Println to the connection package and let the process run, and right where I get this error, this is what the console prints:
********** CHECKING PING
************ CONNECTION STILL ACTIVE
[mysql] 2016/10/11 11:57:27 packets.go:33: unexpected EOF
********** CHECKING PING
************ CONNECTION STILL ACTIVE
Looks like the link to the github issue provided the fix. The fix, at least for my situation was setting the MaxIdleConnections to 0. I have kept a server up for 24 hours, running queries against it every several hours and have yet to reproduce the error.
Thanks to #city for the link.
import (
"database/sql"
"time"
)
//..snip...
db, err = sql.Open("mysql", url)
db.SetConnMaxLifetime(time.Minute * 4) // <-- this
did it for me. Explanation: here
func parent() {
conn, err := db.ClientCat.Conn(ctx)
if err != nil {
return nil, customError.MySqlConnectionError(ctx, errors.New("Connection_not_Established"))
}
//execute some query
defer conn.Close() //*******this won't close until child() finishes
child()
}
func child() {
//under high traffic it won't get connection as they are taken by parent method
//usually we have 10,20,100 etc.. connections available as per configuration, under high traffic all will be taken by parent
conn, err := db.ClientCat.Conn(ctx)
if err != nil {
return nil, customError.MySqlConnectionError(ctx, errors.New("Connection_not_Established"))
}
//execute some query
defer conn.Close()
}
this can also happen in above scenario, please verify.
if system is under load and we have received hundreds of requests then this issue can happen.
basically every time request was going to make connection in child method, it was waiting for connection for a long time then timing out as connections are held by parent method.
also parent method won't be over till child method completes. but child method will wait for connection. So once our parent method has made number of connection = max connection then even in parent method it can start failing to get connection for next requests
DSN adds net_write_timeout option
root:root#tcp(localhost:3306)/prod?net_write_timeout=6000

How to match request to response while proxying packets?

I'm working on MySQL proxy with browser based GUI.
Core of an app looks like:
//Trying to connect to server
rightConn, err := net.Dial("tcp", l.rightAddr)
if err != nil {
return
}
defer rightConn.Close()
wg.Add(2)
//Start passing packets from client to server
go func() {
defer wg.Done()
l.Pipe(leftConn, rightConn)
}()
//Start passing packets from server to client
go func() {
defer wg.Done()
l.Pipe(rightConn, leftConn)
}()
wg.Wait()
And here's definition of Pipe:
func (l *Lottip) Pipe(right, left net.Conn) {
for {
pkt, err := mysql.ReadPacket(right)
if err != nil {
break
}
if _, err = mysql.WritePacket(pkt, left); err != nil {
break
}
}
}
The reason i'm using my custom proxy function instead of
go io.Copy(left, right)
go io.Copy(right, left)
is i have to parse each MySQL's Request/Response packet and prepare it for further processing.
As we all know MySQL client can send a lot of queries and get responses in random order.
The problem is i cannot get how to match request to it's response. While i'm piping using 2 goroutines(1 per direction) i can easily read server response but i don't know to what request it responding.
I've examined request/response between MySQL client and server via Wireshark and found no mention for packet_id or query_id or similar