mysql safe read - update - write - mysql

I'm try to understand mysql lock record mechanic, this is show case in Golang
or here https://play.golang.org/p/2fGKEyh0Wl
it run 2 concurrent transactions, and they read-update on the same row
- the first transaction will try to lock the row, do something (sleep for 3 secs)
- the second then try to read-update on the same key
the test source code
package main
import (
"github.com/jmoiron/sqlx"
"github.com/satori/go.uuid"
"log"
"sync"
"time"
_ "github.com/go-sql-driver/mysql"
)
type Wallet struct {
ID string
Balance int64
}
func main() {
db, err := sqlx.Connect("mysql", "root:abc123#tcp(mysql:3306)/test?parseTime=true")
if err != nil {
log.Println(err)
return
}
db.Exec(`CREATE TABLE test_wallet (
id varchar(64) NOT NULL,
balance bigint(20) DEFAULT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
`)
var wg sync.WaitGroup
wg.Add(2)
wID := uuid.NewV4().String()
db.Exec("INSERT INTO test_wallet (id,balance) VALUES (?,?)", wID, 10)
go func() {
defer wg.Done()
tx, err := db.Beginx()
w1 := &Wallet{}
err = db.Get(w1, "SELECT * FROM test_wallet WHERE id=? FOR UPDATE", wID) // read and lock the record
if err != nil {
log.Println(err)
}
log.Printf("got %+v on r1\n", w1)
time.Sleep(time.Second * 3)
res, err := tx.Exec("UPDATE test_wallet SET balance=? WHERE id=?", w1.Balance+5, wID)
if err != nil {
log.Println(err)
}
n, err := res.RowsAffected()
if n != 1 {
log.Println("update not affected r1")
}
tx.Commit()
log.Println("done on r1")
}()
time.Sleep(time.Second) // make sure go-routine lock `id` row
go func() {
defer wg.Done()
tx, err := db.Beginx()
w2 := &Wallet{}
err = db.Get(w2, "SELECT * FROM test_wallet WHERE id=? FOR UPDATE", wID)
if err != nil {
log.Println(err)
}
log.Printf("got %+v on r2\n", w2)
res, err := tx.Exec("UPDATE test_wallet SET balance=? WHERE id=?", w2.Balance+7, wID)
if err != nil {
log.Println(err)
}
n, err := res.RowsAffected()
if n != 1 {
log.Println("update not affected r2")
}
tx.Commit()
log.Println("done on r2")
}()
wg.Wait()
w := &Wallet{}
err = db.Get(w, "SELECT * FROM test_wallet WHERE id=?", wID)
if err != nil {
log.Println(err)
}
log.Printf("%+v\n", w)
}
result from my terminal
2016/10/01 09:57:00 got &{ID:aab7165c-4b3b-406d-b1d0-caf3f45f72be Balance:10} on r1
2016/10/01 09:57:01 got &{ID:aab7165c-4b3b-406d-b1d0-caf3f45f72be Balance:10} on r2
2016/10/01 09:57:01 done on r2
2016/10/01 09:57:03 done on r1
2016/10/01 09:57:03 &{ID:aab7165c-4b3b-406d-b1d0-caf3f45f72be Balance:15}
it seem the second routine has not been lock ???

You misused the transaction. The db is not in transaction, only tx is. So, the statements in first and second go routine which are
err = db.Get(w1, "SELECT * FROM test_wallet WHERE id=? FOR UPDATE", wID) // read and lock the record
err = db.Get(w2, "SELECT * FROM test_wallet WHERE id=? FOR UPDATE", wID)
will not lock the row, because db is not wrapped in transaction. You must use tx to execute the query, i.e.
err = tx.Get(w1, "SELECT * FROM test_wallet WHERE id=? FOR UPDATE", wID) // read and lock the record
err = tx.Get(w2, "SELECT * FROM test_wallet WHERE id=? FOR UPDATE", wID)
After modification, I got the following result:
2016/10/01 13:26:10 got &{ID:6ff45acd-701c-458f-a17f-84cc4e982c80 Balance:10} on r1
2016/10/01 13:26:14 done on r1
2016/10/01 13:26:14 got &{ID:6ff45acd-701c-458f-a17f-84cc4e982c80 Balance:15} on r2
2016/10/01 13:26:14 done on r2
2016/10/01 13:26:14 &{ID:6ff45acd-701c-458f-a17f-84cc4e982c80 Balance:22}

Related

sql.Query truncated or incomplete results

I've the following code:
const qInstances = `
SELECT
i.uuid,
i.host,
i.hostname
FROM
db.instances AS i
WHERE
i.deleted_at IS NULL
GROUP BY i.uuid;
`
...
instancesMap := make(map[string]*models.InstanceModel)
instances := []models.Instance{}
instancesCount := 0
instancesRow, err := db.Query(qInstances)
if err != nil {
panic(err.Error())
}
defer instancesRow.Close()
for instancesRow.Next() {
i := models.Instance{}
err = instancesRow.Scan(&i.UUID, &i.Host, &i.Hostname)
if err != nil {
log.Printf("[Error] - While Scanning Instances Rows, error msg: %s\n", err)
panic(err.Error())
} else {
if i.UUID.String != "" {
instancesCount++
}
if _, ok := instancesMap[i.UUID.String]; !ok {
instancesMap[i.UUID.String] = &models.InstanceModel{}
inst := instancesMap[i.UUID.String]
inst.UUID = i.UUID.String
inst.Host = i.Host.String
inst.Hostname = i.Hostname.String
} else {
inst := instancesMap[i.UUID.String]
inst.UUID = i.UUID.String
inst.Host = i.Host.String
inst.Hostname = i.Hostname.String
}
instances = append(instances, i)
}
}
log.Printf("[Instances] - Total Count: %d\n", instancesCount)
The problem that I'm facing is that if run the SQL query directly to the database (mariadb) it returns 7150 records, but the total count inside the program outputs 5196 records. I also check the SetConnMaxLifetime parameter for the db connection and set it to 240 seconds, and it doesn't show any errors or broken connectivity between the db and the program. Also I try to do some pagination (LIMIT to 5000 records each) and issue two different queries and the first one returns the 5000 records, but the second one just 196 records. I'm using the "github.com/go-sql-driver/mysql" package. Any ideas?

Does gorm shares an ID from previous transaction Create

I have a query, that requires ID from the previous insert. Does gorm gives away the ID in transaction mode:
Example:
err := db.Transaction(func(tx *gorm.DB) error {
if err := tx.Model(&model.user).Create(&user).Error; err != nil {
tx.Rollback()
}
// if user inserted get the user ID from previous user insert of DB
for _, v := profiles {
v.UserID = user.ID // Can I get this ID from previous Create operation?
}
if err := tx.Model(&model.profile).Create(&profiles).Error; err != nil {
tx.Rollback()
}
})
Yes, you can use the Find function toget the data that you created.
if err := tx.Model(&model.profile).Create(&profiles).Find(&profiles).Error; err != nil {
tx.Rollback()
}

Function in Go to execute select query on database and return json output

I am writing a function in Go to execute select query on database.
Input: String e.g. "Select id, name, age from sometable"
This query changes everytime.
Output: Output of select query in json format.
Sample Expected output: {"Data":[{"id":1,"name":"abc", "age":40},{"id":2,"name":"xyz", "age":45}]}
Sample Actual output: {"Data":[[1,"abc",40],[2,"xyz",45]]}
Instead of i.e. column_name:value, I get only values.
How do I get the expected output?
func executeSQL(queryStr string) []byte {
connString := createConnectString()
conn, err := sql.Open("mssql", connString)
if err != nil {
log.Fatal("Error while opening database connection:", err.Error())
}
defer conn.Close()
rows, err := conn.Query(queryStr)
if err != nil {
log.Fatal("Query failed:", err.Error())
}
defer rows.Close()
columns, _ := rows.Columns()
count := len(columns)
var v struct {
Data []interface{} // `json:"data"`
}
for rows.Next() {
values := make([]interface{}, count)
valuePtrs := make([]interface{}, count)
for i, _ := range columns {
valuePtrs[i] = &values[i]
}
if err := rows.Scan(valuePtrs...); err != nil {
log.Fatal(err)
}
v.Data = append(v.Data, values)
}
jsonMsg, err := json.Marshal(v)
return jsonMsg
}
Got the solution. Here is what I did.
func executeSQL(queryStr string) []byte {
connString := createConnectString()
conn, err := sql.Open("mssql", connString)
if err != nil {
log.Fatal("Error while opening database connection:", err.Error())
}
defer conn.Close()
rows, err := conn.Query(queryStr)
if err != nil {
log.Fatal("Query failed:", err.Error())
}
defer rows.Close()
columns, _ := rows.Columns()
count := len(columns)
var v struct {
Data []interface{} // `json:"data"`
}
for rows.Next() {
values := make([]interface{}, count)
valuePtrs := make([]interface{}, count)
for i, _ := range columns {
valuePtrs[i] = &values[i]
}
if err := rows.Scan(valuePtrs...); err != nil {
log.Fatal(err)
}
//Created a map to handle the issue
var m map[string]interface{}
m = make(map[string]interface{})
for i := range columns {
m[columns[i]] = values[i]
}
v.Data = append(v.Data, m)
}
jsonMsg, err := json.Marshal(v)
return jsonMsg
}
Let me know if there exists a better solution.
This code is directly from my "sandbox" for MsSQL (using denisenkom/go-mssqldb and jmoiron/sqlx) - I think it helps showing different approaches and probably QueryIntoMap is what you were looking for:
package main
import (
"log"
"fmt"
_ "github.com/denisenkom/go-mssqldb"
"time"
"github.com/jmoiron/sqlx"
"encoding/json"
)
type Customer struct {
CustomerId string `db:"customerID" json:"customer_id"`
Company interface{} `db:"companyName" json:"company_name"`
Contact interface{} `db:"contactName" json:"contact_name"`
}
func main() {
connection := "server=192.168.55.3\\SqlExpress2012;database=Northwind;user id=me;Password=secret"
//QueryIntoMap(connection)
ScanIntoSlice(connection)
}
func QueryIntoMap(connection string) {
fmt.Println("QueryIntoMap sample")
fmt.Println("--------------------")
sel := `select customerId, companyName, contactName
from customers
where customerId = :id`
values := make(map[string]interface{})
db, err := sqlx.Open("mssql", connection)
//db.MapperFunc(strings.ToUpper)
e(err)
defer db.Close()
tx := db.MustBegin()
stmt, err := tx.PrepareNamed(sel)
e(err)
stmt.QueryRowx(map[string]interface{}{"id": "BONAP"}).MapScan(values)
tx.Commit()
for k, v := range values {
fmt.Printf("%s %v\n", k, v)
}
js, err := json.Marshal(values)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(js))
fmt.Println("--------------------")
}
func ScanIntoStruct(connection string) {
fmt.Println("Scan into struct sample")
fmt.Println("--------------------")
db, err := sqlx.Open("mssql", connection)
e(err)
defer db.Close()
customer := Customer{}
rows, err := db.Queryx(`select customerID, companyName, contactName
from Customers`)
for rows.Next() {
err = rows.StructScan(&customer)
if err != nil {
log.Fatalln(err)
}
//fmt.Printf("%#v\n", user)
fmt.Printf("%-10s %-50v %-50v\n",
customer.CustomerId,
customer.Company,
customer.Contact)
js, err := json.Marshal(customer)
e(err)
fmt.Println(string(js))
}
fmt.Println("--------------------")
}
func ScanIntoSlice(connection string) {
fmt.Println("Scan into slice sample")
fmt.Println("--------------------")
start := time.Now()
db, err := sqlx.Open("mssql", connection)
e(err)
defer db.Close()
customers := []Customer{}
err = db.Select(&customers, `select customerID, companyName, contactName from customers`)
e(err)
for i, customer := range customers {
fmt.Printf("%3d. %-10s %-50v %-50v\n",
i,
customer.CustomerId,
customer.Company,
customer.Contact)
}
js, err := json.Marshal(customers)
e(err)
fmt.Println(string(js))
fmt.Printf("%s", time.Since(start))
fmt.Println("--------------------")
}
func e(err error) {
if err != nil {
log.Fatal(err)
}
}

Retrieve a record from mysql in golang using database/sql

I'm trying to retrieve a record from mysql table and marshal it to json.
But i gotted the error that says : ".\main.go:67: no new variables on left sife of :="
I'm new in Golang Plz! help me to solve out this error!
My Code is :
func GetUser(w http.ResponseWriter, r *http.Request) {
urlParams := mux.Vars(r)
id := urlParams["id"]
ReadUser := User{}
con, err := sql.Open("mysql", "root:YES#/social_network?charset=utf8")
err := con.QueryRow("select * from users where user_id=?",id).Scan(&ReadUser.ID, &ReadUser.Name, &ReadUser.First, &ReadUser.Last, &ReadUser.Email )
switch {
case err == sql.ErrNoRows:
fmt.Fprintf(w,"No such user")
case err != nil:
log.Fatal(err)
default:
output, _ := json.Marshal(ReadUser)
fmt.Fprintf(w,string(output))
}
}
and Routes in main:
func main() {
gorillaRoute := mux.NewRouter()
gorillaRoute.HandleFunc("/api/user/create", CreateUser)
gorillaRoute.HandleFunc("/api/user/read/:id", GetUser)
http.Handle("/", gorillaRoute)
http.ListenAndServe(":8080", nil)
}
Here is my new Code , but again not able to get record from database table, plz help me to figure out the error OR what things i do wrong?
package main
import (
"database/sql"
"encoding/json"
"fmt"
"log"
"net/http"
_ "github.com/go-sql-driver/mysql"
"github.com/gorilla/mux"
)
type API struct {
Message string "json:message"
}
type User struct {
ID int "json:id"
Name string "json:username"
Email string "json:email"
First string "json:first"
Last string "json:last"
}
func Hello(w http.ResponseWriter, r *http.Request) {
// urlParams := mux.Vars(r)
// name := urlParams["user"]
HelloMessage := "User Creation page"
message := API{HelloMessage}
output, err := json.Marshal(message)
if err != nil {
fmt.Println("Something went wrong!")
}
fmt.Fprintf(w, string(output))
}
//POST A USER INTO DB
func CreateUser(w http.ResponseWriter, r *http.Request) {
NewUser := User{}
NewUser.Name = r.FormValue("user")
NewUser.Email = r.FormValue("email")
NewUser.First = r.FormValue("first")
NewUser.Last = r.FormValue("last")
output, err := json.Marshal(NewUser)
fmt.Println(string(output))
if err != nil {
fmt.Println("Something went wrong!")
}
con, err := sql.Open("mysql", "root:YES#/social_network?charset=utf8")
sqlQuery := "INSERT INTO users set user_nickname='" + NewUser.Name + "', user_first='" + NewUser.First + "', user_last='" + NewUser.Last + "', user_email='" + NewUser.Email + "'"
q, err := con.Exec(sqlQuery)
if err != nil {
fmt.Println(err)
}
fmt.Println(q)
}
//GET USERS FROM DB
func GetUsers(w http.ResponseWriter, r *http.Request) {
db, err := sql.Open("mysql", "root:YES#/social_network?charset=utf8")
if err != nil {
panic(err)
}
err = db.Ping()
if err != nil {
panic(err)
}
defer db.Close()
rows, err := db.Query("select * from users ")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
var rowBuf, _ = rows.Columns()
var cols = make([]string, len(rowBuf))
copy(cols, rowBuf)
fmt.Println(rowBuf)
var vals = make([]interface{}, len(rowBuf))
for i, _ := range rowBuf {
vals[i] = &rowBuf[i]
}
for rows.Next() {
err := rows.Scan(vals...)
if err != nil {
log.Fatal(err)
}
var m = map[string]interface{}{}
for i, col := range cols {
m[col] = vals[i]
}
obj, _ := json.Marshal(m)
//
fmt.Fprintf(w, string(obj))
}
err = rows.Err()
if err != nil {
log.Fatal(err)
}
}
func GetUser(w http.ResponseWriter, r *http.Request) {
urlParams := mux.Vars(r)
id := urlParams["id"]
ReadUser := User{}
db, err := sql.Open("mysql", "root:YES#/social_network?charset=utf8")
stmt := db.QueryRow("select * from users where id = ?", id)
if err != nil {
log.Fatal(err)
}
err = stmt.Scan(&ReadUser.ID, &ReadUser.Name, &ReadUser.First, &ReadUser.Last, &ReadUser.Email)
if err != nil {
log.Fatal(err)
}
result, err := json.Marshal(ReadUser)
fmt.Fprintf(w, string(result))
}
func main() {
gorillaRoute := mux.NewRouter()
gorillaRoute.HandleFunc("/api/user/create", CreateUser)
gorillaRoute.HandleFunc("/api/user/read", GetUsers)
gorillaRoute.HandleFunc("/api/user/:id", GetUser)
http.Handle("/", gorillaRoute)
http.ListenAndServe(":8080", nil)
}
Remove the colon : from err := as you are assigning a new value to existing variable.

Is there a better way to marshal sql rows?

I have the following struct:
type MyTable struct{
DBColA []byte `db:"cola" json:"-"`
ColA string `json:"cola"`
DBColB []byte `db:"colb" json:"-"`
ColB string `json:"colb"`
}
I map to []byte [to better handle null values in my sql][1]
When I grab the rows I need to output it as json. In order to do that I convert []byte to string:
var rows []*MyTable
if _, err := Session.Select(&rows, sql, args...); err != nil {
log.Println(err)
}
for _, row := range rows{
row.ColA = string(row.DBColA)
row.ColB = string(row.DBColB)
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(rows); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
It seems very inefficient to have DBColA and ColA in my struct and then converting DBColA to a string....I have a lot of columns. Is there a better way?
[1]: https://github.com/go-sql-driver/mysql/wiki/Examples
Have you tried gosqljson in https://github.com/elgs/gosqljson ?
See example:
```golang
package main
import (
"database/sql"
"fmt"
"github.com/elgs/gosqljson"
_ "github.com/go-sql-driver/mysql"
)
func main() {
ds := "username:password#tcp(host:3306)/db"
db, err := sql.Open("mysql", ds)
defer db.Close()
if err != nil {
fmt.Println("sql.Open:", err)
}
theCase := "lower" // "lower" default, "upper", camel
a, _ := gosqljson.QueryDbToArrayJson(db, theCase, "SELECT ID,NAME FROM t LIMIT ?,?", 0, 3)
fmt.Println(a)
// [["id","name"],["0","Alicia"],["1","Brian"],["2","Chloe"]]
m, _ := gosqljson.QueryDbToMapJson(db, theCase, "SELECT ID,NAME FROM t LIMIT ?,?", 0, 3)
fmt.Println(m)
// [{"id":"0","name":"Alicia"},{"id":"1","name":"Brian"},{"id":"2","name":"Chloe"}]
}
````