Golang MySQL error - packets.go:33: unexpected EOF - mysql

Question

I am switching my entire code base from PHP to Go and during several processes that run, I randomly get this error:
[mysql] 2016/10/11 09:17:16 packets.go:33: unexpected EOF
Here is my db package that handles all connections to the database:
package db
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"pkg/db"
)
var connection *sql.DB
var err error
func GetConnection() *sql.DB {
if connection != nil {
fmt.Println("********** CHECKING PING")
err = connection.Ping()
if err == nil {
fmt.Println("************ CONNECTION STILL ACTIVE")
return connection
} else {
fmt.Println("********** PING ERROR: " + err.Error())
}
}
connection, err = sql.Open("mysql", db.DEVUSER + ":" + db.DEVUSER_PASSWORD + "#tcp(localhost:3306)/main?parseTime=true")
if err != nil {
panic(err)
}
return connection
}
Is there anything I'm doing wrong with this db package that causes this error to be thrown? What exactly does this error mean? I make sure to return the current connection if there is one open so for multiple requests it uses the same connection object.
Here's an excerpt from the mysql packets.go:
// Read packet to buffer 'data'
func (mc *mysqlConn) readPacket() ([]byte, error) {
var payload []byte
for {
// Read packet header
data, err := mc.buf.readNext(4)
if err != nil {
errLog.Print(err)
mc.Close()
return nil, driver.ErrBadConn
}
// Packet Length [24 bit]
pktLen := int(uint32(data[0]) | uint32(data[1])<<8 | uint32(data[2])<<16)
if pktLen < 1 {
errLog.Print(ErrMalformPkt)
mc.Close()
return nil, driver.ErrBadConn
}
// Check Packet Sync [8 bit]
if data[3] != mc.sequence {
if data[3] > mc.sequence {
return nil, ErrPktSyncMul
}
return nil, ErrPktSync
}
mc.sequence++
// Read packet body [pktLen bytes]
data, err = mc.buf.readNext(pktLen)
if err != nil {
errLog.Print(err)
mc.Close()
return nil, driver.ErrBadConn
}
isLastPacket := (pktLen < maxPacketSize)
// Zero allocations for non-splitting packets
if isLastPacket && payload == nil {
return data, nil
}
payload = append(payload, data...)
if isLastPacket {
return payload, nil
}
}
}
The first "errLog.Print(err)" is line 33 in the "Read packet header" section.
Any help is greatly appreciated!
I added a few log.Println to the connection package and let the process run, and right where I get this error, this is what the console prints:
********** CHECKING PING
************ CONNECTION STILL ACTIVE
[mysql] 2016/10/11 11:57:27 packets.go:33: unexpected EOF
********** CHECKING PING
************ CONNECTION STILL ACTIVE

Answer 1

Looks like the link to the github issue provided the fix. The fix, at least for my situation was setting the MaxIdleConnections to 0. I have kept a server up for 24 hours, running queries against it every several hours and have yet to reproduce the error.
Thanks to #city for the link.

Answer 2

import (
"database/sql"
"time"
)
//..snip...
db, err = sql.Open("mysql", url)
db.SetConnMaxLifetime(time.Minute * 4) // <-- this
did it for me. Explanation: here

Answer 3

func parent() {
conn, err := db.ClientCat.Conn(ctx)
if err != nil {
return nil, customError.MySqlConnectionError(ctx, errors.New("Connection_not_Established"))
}
//execute some query
defer conn.Close() //*******this won't close until child() finishes
child()
}
func child() {
//under high traffic it won't get connection as they are taken by parent method
//usually we have 10,20,100 etc.. connections available as per configuration, under high traffic all will be taken by parent
conn, err := db.ClientCat.Conn(ctx)
if err != nil {
return nil, customError.MySqlConnectionError(ctx, errors.New("Connection_not_Established"))
}
//execute some query
defer conn.Close()
}
this can also happen in above scenario, please verify.
if system is under load and we have received hundreds of requests then this issue can happen.
basically every time request was going to make connection in child method, it was waiting for connection for a long time then timing out as connections are held by parent method.
also parent method won't be over till child method completes. but child method will wait for connection. So once our parent method has made number of connection = max connection then even in parent method it can start failing to get connection for next requests

Answer 4

DSN adds net_write_timeout option
root:root#tcp(localhost:3306)/prod?net_write_timeout=6000