When I using sql package of golang, if I make a query within transaction, and encounter an error while calling rows.Scan(), which method should I call first after this point? *sql.Tx.Rollback() or *sql.Rows.Close()? Currently I call *sql.Rows.Close() before *sql.Tx.Rollback(), but I want to know, what will happen if I reverse this order?
tx, err := db.Begin()
if err != nil {
... // handle error
}
rows, err := tx.Query("sqlstmt")
if err != nil {
... // handle error
}
defer rows.Close() // can I use defer at this place, though it will be called after tx.Rollback()?
if err := rows.Scan(vars...); err != nil {
if e := tx.Rollback(); e != nil {
log(e)
return e
}
return err
}
https://go-review.googlesource.com/c/go/+/44812/
The code is here
It doesn't matter even if skip the rows.Close() within transaction
When the transaction has commit or rollback, the rows will be closed by transaction context.
Related
Why call rows.Close() takes too long time when I call it after I exit from loop rows.Next() before processing all elements of loop.
Its happen when I make request which returns huge amount of data (around 300 000 rows).
This problem doesn't exists when amount of rows is not so big.
func SelectHugeAmountOfRows() {
query := `SELECT * FROM big_table`
rows, _ := Conn.Query(context.Background(), query)
defer func() {
fmt.Println("Start close rows")
start := time.Now()
rows.Close()
duration := time.Since(start)
fmt.Println("rowsClose duration:", duration)
}()
for rows.Next() {
rowValues, err := rows.Values()
if err != nil {
fmt.Println(err)
return
}
// do something with rowValues and get error in process
err = func() error {
fmt.Println(rowValues)
return errors.New("some error")
}()
if err != nil {
return
}
}
if err := rows.Err(); err != nil {
fmt.Println(err)
return
}
}
rowsClose duration: 1m2.5488669s
Interesting that duration of rows.Close() in this case in same as duration which I will have if will process all elements of rows.Next() loop without break it.
when i use golang to query mysql, sometimes i found "deadlock err" in my code.
my question is not "why deadlock occurred", but why deadlock err found in "err = rows.Err()". in my mind, if deadlock occurred, i should get it at "tx.Query"'s return err.
this is demo code, "point 2" is where deadlock error occurred
func demoFunc(tx *sql.Tx, arg1, arg2 int) ([]outItem, error) {
var ret []outItem
var err error
var rows *sql.Rows
//xxxLockSql may deadlock, so try again for 3-times
for i := 0; i < 3; i++ {
//------ point 1
rows, err = tx.Query(xxxLockSql, arg1, arg2)
if err == nil {
break
}
log.Printf("[ERROR] xxxLockSql failed, err %s, retry %d", err.Error(), i)
time.Sleep(time.Millisecond * 10)
}
//if query xxxLockSql failed up to 3-times, then return
if err != nil {
log.Printf("[ERROR] xxxLockSql failed, err %s", err.Error())
return ret, err
}
defer rows.Close()
for rows.Next() {
err = rows.Scan(&a1, &a2)
if err != nil {
return ret, err
}
ret = append(ret, acl)
}
//------ point 2
if err = rows.Err(); err != nil {
// i found deadlock err in this "if" segment.
// err content is "Error 1213: Deadlock found when trying to get lock; try restarting transaction"
log.Printf("[ERROR] loop rows failed, err %s", err.Error())
return ret, err
}
return ret, nil
}
I cannot be sure about the reason since you did not mention your database driver (and which sql package you are using). But I think this is because sql.Query is lazy, which means querying and loading the rows is postponed until actual use, i.e., rows.Next() - that is why the deadlock error occurs there.
As of why it is out of the loop, it's because that when an error occurs, rows.Next() returns false and break the loop.
I'm trying to create a very simple Bolt database called "ledger.db" that includes one Bucket, called "Users", which contains Usernames as a Key and Balances as the value that allows users to transfer their balance to one another. I am using Bolter to view the database in the command line
There are two problems, both contained in this transfer function issue resides in the transfer function.
The First: Inside the transfer function is an if/else. If the condition is true, it executes as it should. If it's false, nothing happens. There's no syntax errors and the program runs as though nothing is wrong, it just doesn't execute the else statement.
The Second: Even if the condition is true, when it executes, it doesn't update BOTH the respective balance values in the database. It updates the balance of the receiver, but it doesn't do the same for the sender. The mathematical operations are completed and the values are marshaled into a JSON-compatible format.
The problem is that the sender balance is not updated in the database.
Everything from the second "Success!" fmt.Println() function onward is not processed
I've tried changing the "db.Update()" to "db.Batch()". I've tried changing the order of the Put() functions. I've tried messing with goroutines and defer, but I have no clue how to use those, as I am rather new to golang.
func (from *User) transfer(to User, amount int) error{
var fbalance int = 0
var tbalance int = 0
db, err := bolt.Open("ledger.db", 0600, nil)
if err != nil {
log.Fatal(err)
}
defer db.Close()
return db.Update(func(tx *bolt.Tx) error {
uBuck := tx.Bucket([]byte("Users"))
json.Unmarshal(uBuck.Get([]byte(from.username)), &fbalance)
json.Unmarshal(uBuck.Get([]byte(to.username)), &tbalance)
if (amount <= fbalance) {
fbalance = fbalance - amount
encoded, err := json.Marshal(fbalance)
if err != nil {
return err
}
tbalance = tbalance + amount
encoded2, err := json.Marshal(tbalance)
if err != nil {
return err
}
fmt.Println("Success!")
c := uBuck
err = c.Put([]byte(to.username), encoded2)
return err
fmt.Println("Success!")
err = c.Put([]byte(from.username), encoded)
return err
fmt.Println("Success!")
} else {
return fmt.Errorf("Not enough in balance!", amount)
}
return nil
})
return nil
}
func main() {
/*
db, err := bolt.Open("ledger.db", 0600, nil)
if err != nil {
log.Fatal(err)
}
defer db.Close()
*/
var b User = User{"Big", "jig", 50000, 0}
var t User = User{"Trig", "pig", 40000, 0}
// These two functions add each User to the database, they aren't
// the problem
b.createUser()
t.createUser()
/*
db.View(func(tx *bolt.Tx) error {
c := tx.Bucket([]byte("Users"))
get := c.Get([]byte(b.username))
fmt.Printf("The return value %v",get)
return nil
})
*/
t.transfer(b, 40000)
}
I expect the database to show Big:90000 Trig:0 from the beginning values of Big:50000 Trig:40000
Instead, the program outputs Big:90000 Trig:40000
You return unconditionally:
c := uBuck
err = c.Put([]byte(to.username), encoded2)
return err
fmt.Println("Success!")
err = c.Put([]byte(from.username), encoded)
return err
fmt.Println("Success!")
You are not returning and checking errors.
json.Unmarshal(uBuck.Get([]byte(from.username)), &fbalance)
json.Unmarshal(uBuck.Get([]byte(to.username)), &tbalance)
t.transfer(b, 40000)
And so on.
Debug your code statement by statement.
I am new to go and backend development. I am trying to use a MySQL database with go to create a REST API.
func getUsers(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306/test)")
if err != nil {
panic(err.Error())
}
results, err := db.Query("Select * from users")
if err != nil {
panic(err.Error())
}
for results.Next() {
var user User
err = results.Scan(&user.FirstName)
if err != nil {
panic(err.Error)
}
fmt.Println(user.FirstName)
}
//scores is an array that i have already created just to return dummy data
json.NewEncoder(w).Encode(scores)
}
I get this error:
http: panic serving [::1]:54508: invalid DSN: network address not terminated (missing closing brace)
goroutine 5 [running]:
net/http.(*conn).serve.func1(0xc42009abe0)
/usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0x129cea0, 0xc420010ec0)
/usr/local/go/src/runtime/panic.go:505 +0x229
main.getUsers(0x134bca0, 0xc42011e000, 0xc42011c200)
/Users/tushar/go/src/github.com/tushar/jump/main.go:62 +0x33f
net/http.HandlerFunc.ServeHTTP(0x1326230, 0x134bca0, 0xc42011e000, 0xc42011c200)
/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc420110000, 0x134bca0, 0xc42011e000, 0xc42011c200)
/Users/tushar/go/src/github.com/gorilla/mux/mux.go:162 +0xed
net/http.serverHandler.ServeHTTP(0xc42008aea0, 0x134bca0, 0xc42011e000, 0xc42011c000)
/usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc42009abe0, 0x134bf60, 0xc420062240)
/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x27b
I am not able to understand the issue.
It works fine when there is no DB query.
EDIT 1:
editted sql.Open to change closing braces to root:mypassword#tcp(127.0.0.1:3306)/test
works fine when querying
but i get panic while inserting in db
func insertUsers(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306)/test")
if err != nil {
panic(err.Error())
}
insert, err := db.Query("Insert into users (personId,firstName,lastName) values(20,tushar,saha)")
if err != nil {
panic(err.Error)
}
defer insert.Close()
}
this is the error 1
2018/06/23 11:54:10 http: panic serving [::1]:54802: 0x126a8b0
goroutine 19 [running]:
net/http.(*conn).serve.func1(0xc4200aebe0)
/usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0x129bb20, 0xc42016a020)
/usr/local/go/src/runtime/panic.go:505 +0x229
main.insertUsers(0x134bc80, 0xc42013c000, 0xc420138200)
/Users/tushar/go/src/github.com/tushar/jump/main.go:50 +0x15d
net/http.HandlerFunc.ServeHTTP(0x1326218, 0x134bc80, 0xc42013c000, 0xc420138200)
/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc42012c000, 0x134bc80, 0xc42013c000, 0xc420138200)
/Users/tushar/go/src/github.com/gorilla/mux/mux.go:162 +0xed
net/http.serverHandler.ServeHTTP(0xc420095040, 0x134bc80, 0xc42013c000, 0xc420138000)
/usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc4200aebe0, 0x134bf40, 0xc42009a200)
/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x27b
The DSN (i.e. second) argument to the sql.Open() method is malformed.
Only the address portion (127.0.0.1:3306) is supposed to be surrounded by parentheses. You've included the database name within the parentheses as well.
Try this instead:
db, err := sql.Open("mysql", "root:mypassword#tcp(127.0.0.1:3306)/test")
The documented format is:
[username[:password]#][protocol[(address)]]/dbname[?param1=value1&...¶mN=valueN]
In response to "EDIT 1"
err.Error() is a method.
In your error handling block for the insert query, you're not calling the method. Instead you're printing out it's memory location.
if err != nil {
panic(err.Error)
}
If you actually call the method (similar to how you have in the error handling block for the sql.Open()), you'll print out the error message which can point you towards what's wrong.
if err != nil {
panic(err.Error())
}
Issue description
I used DB.Begin() to start a transaction and DB.Commit() or DB.Rollback() to end transaction, but I found some connections in the status of "SLEEP" for quite a long time (> 2 hours). And this is not expected. So do I miss something to make sure the connections are all to be freed?
Example code
mgr.statsConn, err = sql.Open("mysql", statsConnStr)
// ......
tx, err := mgr.statsConn.Begin()
if err != nil {
LogError(FAIL_CRITICAL, DB, "Begin transaction failed:%v", err)
return err
}
defer tx.Rollback()
// ...
// multiple Exec clauses just like:
rst, err = tx.Exec(sqlStr)
if err != nil {
LogInfo("Update xxx error. Sql:%s, Error:%v", sqlStr, err)
return err
}
// ...
if err := tx.Commit(); err != nil {
LogError(FAIL_CRITICAL, DB, "Commit error:%v", err)
return err
}
sql.Open() will be called only once of the entire lifetime of the process. and mgr.statsConn will be used everytime when opening a transaction
I'm wondering the right position for the tx.Rollback(). If error happens, is it necessary to call tx.Rollback()? Will tx be non-nill then?
Configuration
Go version:1.7.5
Server version: MYSQL 5.5.37-enterprise-commercial-advanced-log