I have MySql 5.7 and following golang code that may be run concurrently:
tx := s.db.Begin()
if err := tx.Exec(fmt.Sprintf("DELETE FROM related_table WHERE id = %d " item.ID)).Error; err != nil {
tx.Rollback()
}
// Save is ORM method, it make inserts into 'related_table' from the first query
if err := tx.Save(&item).Error; err != nil {
tx.Rollback()
}
I catch error during tx.Save(&item)
Error 1213: Deadlock found when trying to get lock; try restarting transaction
The question is:
How it possible that mysql transaction is not protected against deadlocks? Don't transactions run sequentially?
i can see something "Error; err != nil { tx.Rollback()" ; so i guess this means rollback on failure, right? ;so what happens if its a success; do you have to mention commit explictly; or its autocommit in your setup?
select ... for update is setting exclusive locks on the rows until the end of transaction
Just run following at the beginning of transaction:
tx.Exec("SELECT * FROM %s WHERE coupon_id = ? FOR UPDATE", item.ID))
Related
I'm setting up testing in Go. I use go-sqlmock to test mysql connection. Now I try to test mysql insert logic. But the error occurs.
I want to know how to resolve this error.
server side: golang
db: mysql
web framework: gin
dao.go
func PostDao(db *sql.DB, article util.Article, uu string) {
ins, err := db.Prepare("INSERT INTO articles(uuid, title,content) VALUES(?,?,?)")
if err != nil {
log.Fatal(err)
}
ins.Exec(uu, article.TITLE, article.CONTENT)
}
dao_test.go
func TestPostArticleDao(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
}
defer db.Close()
mock.ExpectExec("^INSERT INTO articles*").
WithArgs("bea1b24d-0627-4ea0-aa2b-8af4c6c2a41c", "test", "test").
WillReturnResult(sqlmock.NewResult(1, 1))
article := util.Article{
ID: 1,
TITLE: "test",
CONTENT: "test",
}
PostDao(db, article, "bea1b24d-0627-4ea0-aa2b-8af4c6c2a41c")
if err := mock.ExpectationsWereMet(); err != nil {
t.Errorf("there were unfulfilled expections: %s", err)
}
}
I expect go test -v runs without error.
But the actual is not.
Here is the error.
=== RUN TestPostArticleDao
2019/08/31 00:08:11 call to Prepare statement with query 'INSERT INTO articles(uuid, title,content) VALUES(?,?,?)', was not expected, next expectation is: ExpectedExec => expecting Exec or ExecContext which:
- matches sql: 'INSERT INTO articles(uuid, title,content) VALUES(?,?,?)'
- is with arguments:
0 - bea1b24d-0627-4ea0-aa2b-8af4c6c2a41c
1 - test
2 - test
- should return Result having:
LastInsertId: 1
RowsAffected: 1
exit status 1
FAIL article/api/dao 0.022s
As #Flimzy suggested, it needs to set ExpectPrepare first.
So I changed dao_test.go in this way:
prep := mock.ExpectPrepare("^INSERT INTO articles*")
prep.ExpectExec().
WithArgs("bea1b24d-0627-4ea0-aa2b-8af4c6c2a41c", "test", "test").
WillReturnResult(sqlmock.NewResult(1, 1))
In my case it worked without asterix:
mock.ExpectExec("INSERT INTO `mytable`").WithArgs(mockdbutils.AnyTime{}, mockdbutils.AnyTime{}, nil, 4455,false).WillReturnResult(sqlmock.NewResult(int64(4455), 1))
mock.ExpectCommit()
I am using GORM with MySQL, I have encountered and handled the error Error 1062: Duplicate entry. The problem is that it's still printed to the console.
Code in gym/models/auth.go:49:
func AddAuth(username, password string) error {
passwordHash, err := auth.HashPassword(password, argon2Conf)
if err != nil {
return err
}
userAuth := Auth{
Username: username,
Password: passwordHash,
}
return db.Create(&userAuth).Error
}
I am handling the error in the handler function:
func SignUpHandler(c *gin.Context) {
var form user
if err := c.ShouldBind(&form); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
if err := models.AddAuth(form.Username, form.Password); err == nil {
c.JSON(http.StatusOK, gin.H{"status": "you are signed in"})
} else {
// I think I have handled the sql error here
c.JSON(http.StatusBadRequest, gin.H{"error": "sign in failed"})
}
}
When I send a POST request, the error is correctly handled and I get the correct response with {"error": "sign in failed"}. But the console still prints this error message:
(/...../gym/models/auth.go:49)
[2019-04-28 23:37:06] Error 1062: Duplicate entry '123123' for key 'username'
[GIN] 2019/04/28 - 23:37:06 | 400 | 136.690908ms | ::1 | POST /signup
I am confused since I handled the error but it still gets printed. How to prevent this error from getting printed to the error log? Or am I handle the error correct?
UPDATE: for gorm v2:
Use the Logger in gorm.Config:
db, err := gorm.Open(sqlite.Open("test.db"), &gorm.Config{
Logger: logger.Default.LogMode(logger.Silent),
})
For gorm v1:
Use db.LogMode to silence the default error logger.
LogMode set log mode, true for detailed logs, false for no log, default, will only print error logs.
db.LogMode(false) should do the job!
I don't have enough reputation to comment but just to add to the answer by ifnotak, you can log sql conditionally by controlling it through an environment variable. This can be handy during debugging.
gormConfig := &gorm.Config{}
if !logSql {
// I use an env variable LOG_SQL to set logSql to either true or false.
gormConfig.Logger = logger.Default.LogMode(logger.Silent)
}
db, err := gorm.Open(dialector, gormConfig)
I am getting the unexpected EOF and busy buffer error in go-sql-driver/mysql despite after setting the SetConnMaxLifetime, SetMaxIdleConns and SetMaxOpenConns as suggested here. Can anyone tell me the proper solution of this issue nothing seems to work for me?
db, err := sql.Open("mysql", "USERNAME:PASSWORD#tcp(IP:PORT)/DB?charset=utf8")
checkErr(err)
db.SetConnMaxLifetime(time.Second * 5)
db.SetMaxIdleConns(0)
db.SetMaxOpenConns(151)
rows, err := db.Query("Select col1, col2, col3 from tbl")
checkErr(err)
for rows.Next() {
var col1 string
var col2 int32
var col3 uint64
err = rows.Scan(&col1, &col2, &col3)
checkErr(err)
Process(col1, col2, col3)
}
I setup a local MySQL database and ran your code:
package main
import (
"database/sql"
"fmt"
"log"
"time"
_ "github.com/go-sql-driver/mysql"
)
func main() {
db, err := sql.Open("mysql", "root#tcp(localhost)/test?charset=utf8")
if err != nil {
log.Fatalln(err)
}
db.SetConnMaxLifetime(time.Second * 5)
db.SetMaxIdleConns(0)
db.SetMaxOpenConns(151)
rows, err := db.Query("SELECT col1, col2, col3 FROM tbl2")
if err != nil {
log.Fatalln(err)
}
for rows.Next() {
var col1 string
var col2 int32
var col3 uint64
err = rows.Scan(&col1, &col2, &col3)
if err != nil {
log.Fatalln(err)
}
fmt.Println(col1, col2, col3)
}
}
..and it worked just fine for me. My CREATE TABLE statement looks like this:
CREATE TABLE `tbl2` (
`col1` varchar(25) DEFAULT NULL,
`col2` int(11) DEFAULT NULL,
`col3` bigint(20) unsigned DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
What does your table structure look like?
I found two ways for work arround the packets.go:36: unexpected EOF error, first way was changing driver to ziutek/mymysql driver, works ok and more or less with the same performance.
Second way, with the default driver, is to set db.SetMaxIdleConns(100) and db.SetMaxOpenConns(100), my mysql has max_connections to 151,
so I thought that limit to 100 will be ok.
In addition, prepared statements has speeded up a lot, before they were slower than Db.Query, now are at least twice as fast. (Tested with 200k queries in 20 goroutines)
I think the problem could be in file buffer.go of driver go-sql-driver/mysql, in line: nn, err := b.nc.Read(b.buf[n:]), some kind of timeout.
None of these worked for me. I have used the Docker container then I tried to log in to it from the local machine with MySQL client.
mysql -h localhost -P 3306 --protocol=tcp
After this go programme worked fine.
According to go-sql-driver/mysql/issues/314, bad_buffer could also happen when attempting to run multiple statements within a transaction while still having an open *sql.Rows.
Ensuring that all *sql.Rows were closed before running subsequent statements resolves the issue.
Before
rows, queryErr = tx.Query(selectQuery, queryArgs...)
// process rows.Next()
// Attempt to perform additional query
rows2, query2Err = tx.Query(selectQuery2, query2Args ...)
// Get bad_buffer error (from logs:)
// [mysql] {timestamp} packets.go:447: busy buffer
// [mysql] {timestamp} connection.go:173: bad connection
After
rows, queryErr = tx.Query(selectQuery, queryArgs...)
defer func(result *sql.Rows) {
_ = result.Close()
}(rows)
// process rows.Next()
// Attempt to perform additional query
rows2, query2Err = tx.Query(selectQuery2, query2Args ...)
defer func(result *sql.Rows) {
_ = result.Close()
}(rows2)
// No error
I'm puzzled.
I try to run test cases using gorm with mysql in golang and I wanna buile MySQL just for testing, but it does not run safely.
I wanna use this package go-test-mysqld
Error message is below.
panic: sql: Register called twice for driver mysql
My code is
func TestMain(m *testing.M) {
mysqld, err := mysqltest.NewMysqld(nil)
if err != nil {
log.Fatal("runTests: failed", err)
}
defer mysqld.Stop()
dbm, err = gorm.Open("mysqld", mysqld.Datasource("test", "", "", 0 ))
if err != nil {
log.Fatal("db connection error:", err)
}
defer dbm.Close()
code := m.Run()
os.Exit(code)
}
What is the problems in my code?
Or is it impossible to build another mysql in using gorm?
Do you have some ideas?
I'm using github.com/go-sql-driver/mysql package to connect to MySQL. It works well except when I select a database (USE), I can't run queries against it.
package main
import (
"database/sql"
"fmt"
"log"
)
import _ "github.com/go-sql-driver/mysql"
func main() {
dsn := "root:#/"
db, err := sql.Open("mysql", dsn)
if err != nil {
fmt.Println("Failed to prepare connection to database. DSN:", dsn)
log.Fatal("Error:", err.Error())
}
err = db.Ping()
if err != nil {
fmt.Println("Failed to establish connection to database. DSN:", dsn)
log.Fatal("Error:", err.Error())
}
_, err = db.Query("USE test")
if err != nil {
fmt.Println("Failed to change database.")
log.Fatal("Error:", err.Error())
}
_, err = db.Query("SHOW TABLES")
if err != nil {
fmt.Println("Failed to execute query.")
log.Fatal("Error:", err.Error())
}
}
The program produces this output:
Error 1046: No database selected
Specify the database directly in the DSN (Data Source Name) part of the sql.Open function:
dsn := "user:password#/dbname"
db, err := sql.Open("mysql", dsn)
That's because db maintains a connection pool that has several connections to mysql database."USE test" just let one connection use schema test.
When you do database query later,the driver will select one idle connection,if the connection that use test schema is selected,it will be normal,but if another connection is chosen, it does not use test,so it will report an error:no database selected.
If you add a clause:
db.SetMaxOpenConns(1)
the db will maintain only one connection,it will not have an error.And of course it's impossible in high concurrency scene.
If you specify the database name in sql.open() function,all the connection will use this data base which can avoid this problem.
In your case you need to use transactions:
tx, _ := db.Begin()
tx.Query("USE test")
tx.Query("SHOW TABLES")
tx.Commit()
For SELECT/UPDATE/INSERT/etc need to specify DB name in the query.
As other answers mentioned, sql.DB is not a single connection but a connection pool. When you execute use database, imagine you executed your query on just one connection in the pool. Next query will get another connection from the pool which has no databases selected.
I would strongly advise against using transactions for this (as several places suggest).
I would suggest to use context:
ctx := context.Background()
conn, err := db.Conn(ctx)
conn.ExecContext(ctx, "use mydb")
defer conn.Close()
var found int
err = conn.QueryRowContext(ctx, "SELECT count(*) as found FROM mytable").Scan(&found)
if err != nil {
panic(err)
}