I can't understand how to select and then update table with several goroutines. In the documentation for db and stmt it says: "is safe for concurrent use by multiple goroutines." Also I use transatcions but without success. I want to start 7 goroutines and take every row.
Data
+--------+-----------+---------------------+
| idTest | someValue | date |
+--------+-----------+---------------------+
| 1 | 1 | 2019-06-11 11:29:42 |
| 2 | 2 | 2019-06-11 11:29:42 |
| 3 | 3 | NULL |
| 4 | 4 | NULL |
| 5 | 5 | NULL |
| 6 | 6 | NULL |
| 7 | 7 | NULL |
+--------+-----------+---------------------+
current code
db, err := sql.Open("mysql", strConn)
if err != nil {
fmt.Printf("Troubles in connaction! %s", err)
}
var idTest int
var someValue string
stmt, err := db.Prepare("select idTest,someValue from test where date is null limit 1")
CheckError(err)
defer stmt.Close()
rows, err := stmt.Query()
CheckError(err)
defer rows.Close()
for rows.Next() {
rows.Scan(&idTest, &someValue)
stmt, err = db.Prepare("update test set date = now() where idTest= ?")
CheckError(err)
_, err = stmt.Exec(idTest)
CheckError(err)
}
Every goroutine have db.conn and sometimes trying to select and update table.
func main() {
for i := 0; i < 7; i++ {
dbConn := "blabla"
go ChildBot(dbConn)
}
var input string
fmt.Scanln(&input)
}
you should read the data and then run a goroutine.
for rows.Next() {
rows.Scan(&idTest, &someValue)
go func(idTest int) {
stmt, err = db.Prepare("update test set date = now() where idTest= ?")
CheckError(err)
_, err = stmt.Exec(idTest)
CheckError(err)
}(idTest)
}
it's done this way because you have to read the data first before doing anything with it. Otherwise, the next read could override your previous value from time to time.
Related
I am trying the print the struct variable but it's returning the zero value where I want an actual database row value.
Code:
package main
import (
"database/sql"
"fmt"
"log"
"time"
_ "github.com/go-sql-driver/mysql"
)
type Users struct {
Id int
Username string
Password string
Email string
First_name string
Last_name string
Created_at time.Time
Super_user bool
}
func main() {
db, err := sql.Open("mysql", "root:Megamind#1#(127.0.0.1:3306)/note?parseTime=true")
if err != nil {
log.Fatalln("Couldn't connect to the database")
}
var user Users
row := db.QueryRow("select password from users where username=$1", "someone")
row.Scan(&user.Id, &user.Username, &user.Password, &user.Email, &user.First_name, &user.Last_name, &user.Created_at, &user.Super_user)
fmt.Println(user)
}
Database:
mysql> select * from users;
+----+---------------+--------------------------------------------------------------+---------------------------+------------+-----------+---------------------+------------+
| id | username | password | email | first_name | last_name | created_at | super_user |
+----+---------------+--------------------------------------------------------------+---------------------------+------------+-----------+---------------------+------------+
| 3 | someone | $2a$10$a0g.eIGEHoVnD/s55YCePeL5BxCPYDF58nP2gb.TmYKwCuV5E7gP. | abc#gmail.com | NULL | NULL | 2020-06-22 01:24:43 | NULL |
| 4 | oneanother | $2a$10$VMo4iuvruCA/yQlkfMI2QOMWo2H2jIiyyoYprKQtQMT4U7UWb78CS | one#one.com | NULL | NULL | 2020-06-22 01:26:48 | NULL |
| 5 | alpha | $2a$10$oD0YKBkTvJQVPF4rilEVYemjRtwNF3ATGlVLUOVGZzR5lNx5fRl3. | alpha#gmail.com | NULL | NULL | 2020-06-22 01:45:34 | NULL |
+----+---------------+--------------------------------------------------------------+---------------------------+------------+-----------+---------------------+------------+
3 rows in set (0.02 sec)
Output:
(base) [dave#192 test]$ go run main.go
{0 0001-01-01 00:00:00 +0000 UTC false}
Server Version & gorm package:
❯ docker exec -it mysql mysqld --version
mysqld Ver 5.7.29 for Linux on x86_64 (MySQL Community Server (GPL))
❯ docker exec -it mysql mysql --version
mysql Ver 14.14 Distrib 5.7.29, for Linux (x86_64) using EditLine wrapper
import "github.com/jinzhu/gorm"
Two tables:
mysql> desc t1;
+------------+-------------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------------+------+-----+-------------------+----------------+
| ... | | | | | |
| created_at | timestamp | YES | | CURRENT_TIMESTAMP | |
+------------+-------------------+------+-----+-------------------+----------------+
mysql> desc t2;
+----------------+------------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+------------------+------+-----+-------------------+----------------+
| ... | ... | ... | ... | ... | ... |
| initiated_at | timestamp | YES | | CURRENT_TIMESTAMP | |
+----------------+------------------+------+-----+-------------------+----------------+
The gorm model while defining stuct{} is as following:
// t1
type T1 struct {
ID uint `gorm:"primary_key"`
// others are here
// ...
CreatedAt time.Time `gorm:"timestamp;default:CURRENT_TIMESTAMP" json:"created_at" form:"created_at" query:"created_at" sql:"DEFAULT:current_timestamp"`
}
// t2 ...
type T2 struct {
ID uint `gorm:"primary_key"`
// others are here
// ...
InitiatedAt time.Time `gorm:"timestamp;default:CURRENT_TIMESTAMP" json:"initiated_at" form:"initiated_at" query:"initiated_at" sql:"DEFAULT:current_timestamp"`
}
insert with uninitialized timestamp
dbSource := fmt.Sprintf(
"%s:%s#tcp(%s:%s)/%s?charset=utf8&parseTime=True&loc=Local",
cnf.Username,
cnf.Password,
cnf.Host,
cnf.Port,
cnf.DBName,
)
db, err := gorm.Open("mysql", dbSource)
if err != nil {
logrus.Warn("Got error when connect database:", err)
return err
}
t1 := T1{} // created_at is not set
t2 := T2{} // initiated_at is not set
tx := db.Begin()
defer func() {
if r := recover(); r != nil {
logrus.Error("Rolling back")
tx.Rollback()
}
}()
// If failed to begin transaction
if err := tx.Error; err != nil {
return err
}
if err := db.Create(&t1).Error; err != nil {
logrus.Warn(err)
// rollback the transaction in case of error
tx.Rollback()
return derror.ErrorBadRequest
}
if err := db.Create(&t2).Error; err != nil {
logrus.Warn(err)
// rollback the transaction in case of error
tx.Rollback()
return derror.ErrorBadRequest
}
// Or commit the transaction
if err := tx.Commit().Error; err != nil {
logrus.Warn(err)
// rollback the transaction in case of error
tx.Rollback()
return derror.ErrorBadRequest
}
What I see, when do select queries
mysql> select * from t1;
+-----+---------------------+
| ... | created_at |
+-----+---------------------+
| ... | 2020-03-24 02:38:26 |
+-----+---------------------+
mysql> select * from t2;
+-----+---------------------+
| ... | initiated_at |
+-----+---------------------+
| ... | 2020-03-23 20:38:26 |
+-----+---------------------+
Expectation:
Note that, I am in asia/dhaka(+06:00) region. And the time of created_at of t1 table is the BST current time of my region. On the other hand, the time of initiated_at of t2 table is the UTC current time.
But I expect that both the times are the same (I mean the either UTC or BST).
Want to know:
The reason why the two times are of different region.
Any solution so that both the times are of same region
In Table t1, CreatedAt set by Gorm in local timezone since you use loc=Local.
Ref: https://github.com/jinzhu/gorm/blob/master/callback_create.go#L32
And in Table t2, initiated_at not set by Gorm it set by Mysql since you use default value as CURRENT_TIMESTAMP in MySql.
Solution:
You can change Gorm timezone to UTC using loc=UTC in connection.
Or
You can set your local timezone as Mysql timezone.
Note that sets the location for time.Time values but does not change MySQL's time_zone setting.For that see the time_zone system variable, which can also be set as a DSN parameter.
Ref: https://github.com/go-sql-driver/mysql#loc
I used Java before, so some columns' type in database table is bit(1). But now I want to use beego to rebuild my project and I don't want to alter my database table (need do much). I use beego's orm in my project. So which Go type should I use?
Table like this and the deleted column has the question:
+--------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| id | varchar(255) | NO | PRI | NULL | |
| created_time | datetime | YES | | NULL | |
| deleted | bit(1) | NO | | NULL | |
| updated_time | datetime | YES | | NULL | |
| icon_class | varchar(255) | YES | | NULL | |
| mark | varchar(255) | YES | | NULL | |
| name | varchar(255) | YES | | NULL | |
| parent | varchar(255) | YES | | NULL | |
+--------------+--------------+------+-----+---------+-------+
go struct like this:
type BaseModel struct {
Id string `orm:"pk";form:"id"`
CreatedTime time.Time `orm:"auto_now_add;type(datetime)";form:"-"`
UpdatedTime time.Time `orm:"auto_now;type(datetime)";form:"-"`
Deleted bool `form:"-"`
}
When I use bool in my code, the error like this:
`[0]` convert to `*orm.BooleanField` failed, field: shareall-go/models.Category.BaseModel.Deleted err: strconv.ParseBool: parsing "\x00": invalid syntax
Sqlx has created also a custom bool datatype for such situations and it works fine. Link to related code
// BitBool is an implementation of a bool for the MySQL type BIT(1).
// This type allows you to avoid wasting an entire byte for MySQL's boolean type TINYINT.
type BitBool bool
// Value implements the driver.Valuer interface,
// and turns the BitBool into a bitfield (BIT(1)) for MySQL storage.
func (b BitBool) Value() (driver.Value, error) {
if b {
return []byte{1}, nil
} else {
return []byte{0}, nil
}
}
// Scan implements the sql.Scanner interface,
// and turns the bitfield incoming from MySQL into a BitBool
func (b *BitBool) Scan(src interface{}) error {
v, ok := src.([]byte)
if !ok {
return errors.New("bad []byte type assertion")
}
*b = v[0] == 1
return nil
}
So which Go type should I use?
Generally, this depends on how you're using the data more than how it's stored. As you eluded to, you tried using it as a Bool (which makes sense) but got an error.
The problem is that MySQL expresses a BIT differently than a BOOL, and the Go MySQL driver expects a MySQL BOOL. You can fix this by using a custom type that implements the sql.Scanner interface. Since you presumably have only two (or maybe three, if you count NULL) inputs, it should be fairly easy. Note this code is incomplete and untested. It is meant to serve as a guide, not a copy-and-paste solution.
type MyBool bool
func (b *MyBool) Scan(src interface{}) error {
str, ok := src.(string)
if !ok {
return fmt.Errorf("Unexpected type for MyBool: %T", src)
}
switch str {
case "\x00":
v := false
*b = v
case "\x01":
v := true
*b = v
}
return nil
}
I have no idea why Tarmo's solution doesn't work for me. But after a little bit modification, it works.
type BitBool bool
func (bb BitBool) Value() (driver.Value, error) {
return bool(bb), nil
}
func (bb *BitBool) Scan(src interface{}) error {
if src == nil {
// MySql NULL value turns into false
*bb = false
return nil
}
bs, ok := src.([]byte)
if !ok {
return fmt.Errorf("Not byte slice!")
}
*bb = bs[0] == 1
return nil
}
In this way, I can do the following
var isVip BitBool
row := db.QueryRow("SELECT is_vip FROM user WHERE user_id = '12345'")
err := row.Scan(&isVip)
var isVip BitBool = true
rows, err := db.Query("SELECT username FROM user WHERE is_vip = ?", isVip)
Suppose there is a csv file is formated as below :-
first file
firstname|lastname|Email| other fields
| | |
| | |
| | |
| | |
Second file:-
email|firstname|lastname|other fields
| | |
| | |
| | |
| | |
third file
lastname|firstname|email|other fields
| | |
| | |
| | |
| | |
So I want to save these three files in mongodb database separately. In the given below format:
Format is first_name, last_name,email, otherfield
Code I'm using:-
package main
import (
"encoding/csv"
"gopkg.in/mgo.v2"
"io"
"log"
"os"
)
type Mongo struct {
// Id int `json:"_id" bson:"_id"`
FirstName string `json:"first_name,omitempty" bson:"first_name,omitempty"`
LastName string `json:"last_name,omitempty" bson:"last_name,omitempty"`
Email string `json:"email,omitempty" bson:"email,omitempty"`
PhoneNumber string `json:"phone_number,omitempty" bson:"phone_number,omitempty"`
Gender string `json:"gender,omitempty" bson:"gender,omitempty"`
Address string `json:"address,omitempty" bson:"address,omitempty"`
Apartment string `json:"apartment,omitempty" bson:"apartment,omitempty"`
Description string `json:"description,omitempty" bson:"description,omitempty"`
}
func main() {
session, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("Import_Users").C("users")
file, err := os.Open("customers.csv")
if err != nil {
panic(err)
}
defer file.Close()
reader := csv.NewReader(file)
for {
record, err := reader.Read()
if err == io.EOF {
break
} else if err != nil {
panic(err)
}
err = c.Insert(&Mongo{record[0], record[1], record[2], record[3], record[4], record[5], record[6], record[7]})
if err != nil {
panic(err)
}
log.Printf("%#v", record)
}
}
The code I'm using is simple to save the data in the mongodb but it will not format the data which I need to save
How can I achieve my goal?
I think you can use reflection to map each row of the csv over a structure. Something like the json encoding package is doing from a string to a structure. Then that structure can be inserted in mongodb.
Something like this you want?
https://play.golang.org/p/hg3K-u8l9mx
You need to more protection on it and you can do a lot of speed improvements on it, but this is something like the Unmarshal from json pkg.
I am currently working on a Golang Google App Engine Project and have run into a small problem. I have a database "party" with table "parties". The problem is that when the following code is executed, a EMPTY json array is printed - it actually is properly long, but it only contains empty Parties. (And I do have entries in my database)
Go code (not all of it):
func getParties(w http.ResponseWriter, r *http.Request) {
rows := getRowsFromSql("select * from parties;")
parties := scanForParties(rows)
json, _ := json.Marshal(parties)
fmt.Fprint(w, string(json))
}
func scanForParties(rows *sql.Rows) []Party {
var parties []Party
for rows.Next() {
var id int
var name, author, datetime, datetime_to, host, location, description, longtitude, latitude, primary_image_id string
rows.Scan(&id, &name, &author, &datetime, &datetime_to, &host, &location, &description, &longtitude, &latitude, &primary_image_id)
party := Party{
Id: id,
Name: name,
Author: author,
Datetime: datetime,
Datetime_to: datetime_to,
Host: host,
Location: location,
Description: description,
Longtitude: longtitude,
Latitude: latitude,
PrimaryImgId: primary_image_id,
}
parties = append(parties, party)
}
return parties
}
func getRowsFromSql(query string) *sql.Rows {
con, err := sql.Open("mysql", dbConnectString)
if err != nil {
panic(err)
}
defer con.Close()
rows, err2 := con.Query(query)
if err != nil {
panic(err2)
}
return rows
}
type Party struct {
Id int
Name string
Author string
Datetime string
Datetime_to string
Host string
Location string
Description string
Longtitude string
Latitude string
PrimaryImgId string
}
And my parties table:
mysql> describe parties;
+------------------+----------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+----------------+------+-----+-------------------+-----------------------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(64) | NO | | | |
| author | varchar(64) | YES | | NULL | |
| datetime | datetime | YES | | NULL | |
| last_edited | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| datetime_to | datetime | YES | | NULL | |
| host | text | YES | | NULL | |
| location | text | YES | | NULL | |
| description | text | YES | | NULL | |
| longitude | decimal(23,20) | YES | | NULL | |
| latitude | decimal(23,20) | YES | | NULL | |
| primary_image_id | varchar(256) | YES | | NULL | |
+------------------+----------------+------+-----+-------------------+-----------------------------+
However, this old version of code works just fine:
func getParties(w http.ResponseWriter, r *http.Request) {
con, dbErr := sql.Open("mysql", dbConnectString)
defer con.Close()
if dbErr == nil {
rows, _ := con.Query("select id, name, author, datetime from parties where datetime >= NOW();")
var parties []Party
var id int
var name string
var author string
var datetime string
for rows.Next() {
rows.Scan(&id, &name, &author, &datetime)
party := Party{}
party.Id = id
party.Name = name
party.Author = author
party.Datetime = datetime
parties = append(parties, party)
}
if len(parties) > 0 {
json, _ := json.Marshal(parties)
fmt.Fprint(w, string(json))
} else {
fmt.Fprint(w, "{}")
}
} else {
fmt.Fprint(w, "{\"Error\"}")
}
}
Any idea why this happens?
Thanks in advance :)
This is a guess, but I'm thinking that it's because you're closing the connection to the database here:
defer con.Close()
This will close the connection to the database when getRowsFromSql returns, so by the time you start calling rows.Next() in scanForParties the DB connection is gone. Once the DB connection is closed, any collection of rows will no longer be available.
Something is probably returning an error because of this, but since you're not checking any errors anywhere you won't know. In Go it is idiomatic to check for errors whenever a function can return one (and other languages too, just more so in Go because of the lack of exceptions).
Okay so all the others were right about the errors: rows.Scan() returns an error. And when I finally checked it, it said that there are insufficient scan variables provided. Simple fix: add the missing one.
Thank you guys :)