I've the following code:
const qInstances = `
SELECT
i.uuid,
i.host,
i.hostname
FROM
db.instances AS i
WHERE
i.deleted_at IS NULL
GROUP BY i.uuid;
`
...
instancesMap := make(map[string]*models.InstanceModel)
instances := []models.Instance{}
instancesCount := 0
instancesRow, err := db.Query(qInstances)
if err != nil {
panic(err.Error())
}
defer instancesRow.Close()
for instancesRow.Next() {
i := models.Instance{}
err = instancesRow.Scan(&i.UUID, &i.Host, &i.Hostname)
if err != nil {
log.Printf("[Error] - While Scanning Instances Rows, error msg: %s\n", err)
panic(err.Error())
} else {
if i.UUID.String != "" {
instancesCount++
}
if _, ok := instancesMap[i.UUID.String]; !ok {
instancesMap[i.UUID.String] = &models.InstanceModel{}
inst := instancesMap[i.UUID.String]
inst.UUID = i.UUID.String
inst.Host = i.Host.String
inst.Hostname = i.Hostname.String
} else {
inst := instancesMap[i.UUID.String]
inst.UUID = i.UUID.String
inst.Host = i.Host.String
inst.Hostname = i.Hostname.String
}
instances = append(instances, i)
}
}
log.Printf("[Instances] - Total Count: %d\n", instancesCount)
The problem that I'm facing is that if run the SQL query directly to the database (mariadb) it returns 7150 records, but the total count inside the program outputs 5196 records. I also check the SetConnMaxLifetime parameter for the db connection and set it to 240 seconds, and it doesn't show any errors or broken connectivity between the db and the program. Also I try to do some pagination (LIMIT to 5000 records each) and issue two different queries and the first one returns the 5000 records, but the second one just 196 records. I'm using the "github.com/go-sql-driver/mysql" package. Any ideas?
Related
I have the following query which returns results when run directly from mysql.
The same query returns 0 values, when run from golang program.
package main
import (
"github.com/rs/zerolog/log"
_ "github.com/go-sql-driver/mysql"
"github.com/jmoiron/sqlx"
)
var DB *sqlx.DB
func main() {
DB, err := sqlx.Connect("mysql", "root:password#(localhost:3306)/jsl2")
if err != nil {
log.Error().Err(err)
}
sqlstring := `SELECT
salesdetails.taxper, sum(salesdetails.gvalue),
sum(salesdetails.taxamt)
FROM salesdetails
Inner Join sales ON sales.saleskey = salesdetails.saleskey
where
sales.bdate >= '2021-12-01'
and sales.bdate <= '2021-12-31'
and sales.achead IN (401975)
group by salesdetails.taxper
order by salesdetails.taxper`
rows, err := DB.Query(sqlstring)
for rows.Next() {
var taxper int
var taxableValue float64
var taxAmount float64
err = rows.Scan(&taxper, &taxableValue, &taxAmount)
log.Print(taxper, taxableValue, taxAmount)
}
err = rows.Err()
if err != nil {
log.Error().Err(err)
}
}
On the console, running the program returns the following values.
In SQL browser, it returns 4 rows which is correct.
The result from the sql browser for the same query is
0 1278.00 0.00
5 89875.65 4493.78
12 3680.00 441.60
18 94868.73 17076.37
But in the program also return 4 rows with 0 value.
{"level":"debug","time":"2022-01-13T17:07:39+05:30","message":"0 0 0"}
{"level":"debug","time":"2022-01-13T17:07:39+05:30","message":"0 0 0"}
{"level":"debug","time":"2022-01-13T17:07:39+05:30","message":"0 0 0"}
{"level":"debug","time":"2022-01-13T17:07:39+05:30","message":"0 0 0"}
How to set the datatype for the aggregate functions.
I changed the datatype of taxper to float and it worked.
I found this after checking the err from rows.Scan( as suggested by #mkopriva
I have a query, that requires ID from the previous insert. Does gorm gives away the ID in transaction mode:
Example:
err := db.Transaction(func(tx *gorm.DB) error {
if err := tx.Model(&model.user).Create(&user).Error; err != nil {
tx.Rollback()
}
// if user inserted get the user ID from previous user insert of DB
for _, v := profiles {
v.UserID = user.ID // Can I get this ID from previous Create operation?
}
if err := tx.Model(&model.profile).Create(&profiles).Error; err != nil {
tx.Rollback()
}
})
Yes, you can use the Find function toget the data that you created.
if err := tx.Model(&model.profile).Create(&profiles).Find(&profiles).Error; err != nil {
tx.Rollback()
}
I'm updating records from another resource into the database.
// Turn into batches with batchSize == 10
batches := service.TurnIntoBatches(cModels, 10)
// Sync to our db
var wg sync.WaitGroup
customerChannel := make(chan []*model.Customer, len(batches))
for _, batch := range batches {
wg.Add(1)
go UpdateCustomers(batch, &wg, uc.customerRp, customerChannel)
}
wg.Wait()
close(customerChannel)
func UpdateCustomers(customers []*model.Customer,
wg *sync.WaitGroup,
rp repository.CustomerRepositoryInterface,
channel chan []*model.Customer) {
defer wg.Done()
for _, c := range customers {
//err := Update logic
if err != nil {
logger.WithFields(logrus.Fields{
"error": err,
}).WithError(err).Error("Updating failed!")
}
}
}
If I got 50 records, that's 5 batches of 10.
There are 50 insert query, as expected. All of them returned [1 rows affected or returned ]
However, if I query from the database, there are sometimes 45 or 46 records.
What is the problem and how to fix it?
Found a strange inconsistency upon selecting rows from mysql 8.0.19 after some transaction on resultset (e.g. at mysqlworkbench by editing some rows).
(for reference this function: https://golang.org/pkg/database/sql/#DB.Query)
In others words db.Query(SQL) returns the old state of my resultset (before editing and committing).
MYSQL rows before editing:
105 admin
106 user1
107 user2
109 user3
MYSQL rows after editing:
105 admin
106 user11
107 user22
109 user33
But Golang db.Query(SQL) still continuously returns:
105 admin
106 user1
107 user2
109 user3
Does db.Query(SQL) require being committed to maintain consistency with current database state? Because after I have added db.Begin() and db.Commit() it started to work consistently. Haven't tried other databases, doesn't look like a driver issue, or variable copy issue. It is bit odd coming from JDBC. Autocommit disabled.
The code:
func FindAll(d *sql.DB) ([]*usermodel.User, error) {
const SQL = `SELECT * FROM users t ORDER BY 1`
//tx, _ := d.Begin()
rows, err := d.Query(SQL)
if err != nil {
return nil, err
}
defer func() {
err := rows.Close()
if err != nil {
log.Fatalln(err)
}
}()
l := make([]*usermodel.User, 0)
for rows.Next() {
t := usermodel.NewUser()
if err = rows.Scan(&t.UserId, &t.Username, &t.FullName, &t.PasswordHash, &t.Email, &t.ExpireDate,
&t.LastAuthDate, &t.StateId, &t.CreatedAt, &t.UpdatedAt); err != nil {
return nil, err
}
l = append(l, t)
}
if err = rows.Err(); err != nil {
return nil, err
}
//_ = tx.Commit()
return l, nil
}
This is purely about MySQL MVCC (see https://dev.mysql.com/doc/refman/8.0/en/innodb-multi-versioning.html and https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html) and not Go/DB driver.
In short, if you start a transaction, read some data, then another transaction changes it and commits, you may or may not see the results, depending on the transaction isolation level set on the MySQL server.
Following is my code to get multiple rows from db and it works.
defer db.Close()
for rows.Next() {
err = rows.Scan(&a)
if err != nil {
log(err)
}
How can I check if the rows contains No row?
Even I tried like below
if err == sql.ErrNoRows {
fmt.Print(No rows)
}
and also checked while scanning
err = rows.Scan(&a)
if err == sql.ErrNoRows {
fmt.Print(No rows)
}
I don't understand which one gives ErrNoRows either *Rows or err or Scan
QueryRow returns a *Row (not a *Rows) and you cannot iterate through the results (because it's only expecting a single row back). This means that rows.Scan in your example code will not compile).
If you expect your SQL query to return a single resullt (e.g. you are running a count() or selecting using an ID) then use QueryRow; for example (modified from here):
id := 43
var username string
err = stmt.QueryRow("SELECT username FROM users WHERE id = ?", id).Scan(&username)
switch {
case err == sql.ErrNoRows:
log.Fatalf("no user with id %d", id)
case err != nil:
log.Fatal(err)
default:
log.Printf("username is %s\n", username)
}
If you are expecting multiple rows then use Query() for example (modified from here):
age := 27
rows, err := db.Query("SELECT name FROM users WHERE age=?", age)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
names := make([]string, 0)
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
log.Fatal(err)
}
names = append(names, name)
}
// Check for errors from iterating over rows.
if err := rows.Err(); err != nil {
log.Fatal(err)
}
// Check for no results
if len(names) == 0 {
log.Fatal("No Results")
}
log.Printf("%s are %d years old", strings.Join(names, ", "), age)
The above shows one way of checking if there are no results. If you are not putting the results into a slice/map then you can keep a counter or set a boolean within the loop. Note that no error will be returned if there are no results (because this is a perfectly valid outcome) and the SQL package provides no way to check the number of results other than iterate through them (if all you are interested in is the number of results then run select count(*)...).