Golang, mysql: Error 1040: Too many connections - mysql

I'm using the github.com/go-sql-driver/mysql driver for go.
I open a database:
db, err := sql.Open("mysql", str)
Then I have two functions that are called 200 times each with following mysql code:
rows, err := db.Query("select name from beehives")
if err != nil {
panic(err)
}
defer rows.Close()
The second:
err = db.QueryRow("select id, secret, shortname from beehives where shortname = ?", beehive).Scan(&id, &secre
switch {
case err == sql.ErrNoRows:
err = errors.New("Beehive '"+beehive+"' not found.")
case err != nil:
panic("loginBeehive: "+ err.Error())
default:
// ... do the work
The first one is panicing.
How can there be more than one connection when I open the database only once and how do I close them?

sql.Open doesn't really open a connection to your database.
A sql.DB maintains a pool of connections to your database. Each time you query your database your program will try to get a connection from this pool or create a new one otherwise. These connections are than put back into the pool once you close them.
This is what rows.Close() does.
Your db.QueryRow("...") does the same thing internally when you call Scan(...).
The basic problem is that you're creating too many queries, of which each one needs a connection, but you are not closing your connections fast enough. This way your program has to create a new connection for each query.
You can limit the maximum number of connections your program uses by calling SetMaxOpenConns on your sql.DB.
See http://go-database-sql.org/surprises.html for more information.

The *DB object that you get back from sql.Open doesn't corresponds to a single connection. It is better thought as a handle for the database: it manages a connection pool for you.
You can control the number of open connections with `(*DB).SetMaxOpenConns and its pair for idling connections.
So basically what happens here is that db.Query and db.QueryRow tries to acquire a connection for themselves and the DB handle doesn't put any restrictions on the number of simultaneous connections so your code panics when it opens more than what mysql can handle.

Try to make prepared statements db.Prepare(query string) (*Stmt, error) and than stmt.Query or stmt.Exec and than stmt.Close to reuse connections.

hi can you try close connection after used
db, err := sql.Open("mysql", str)
defer db.Close() // close after end scope

My program is connecting always to database. (Realtime Face Recognition for Attendance)
Therefore opening and closing database connection is worthless.
Therefore it's keep opens the database connection only initializing the program.
func GetAllFaces() interface{} {
OpenDatabaseConnection() ...
}
But access database later, increased the no of connection and crashed the program. But closing the rows object kept no of active connection at minimum. (for me 1)
func SaveAttendance(faceId int, date time.Time) error {
sqlQuery := fmt.Sprintf("SELECT ... "))
rows, err := DB.Query(sqlQuery) ...
err = rows.Close()
return err
}

Related

simple planetscale - golang app won't select the database properly

After connecting to a Planetscale database using DSN with the following structure:
user:password#tcp(host)/database?tls=true
I decided to test it by running a simple query. Here's my main.go file:
package main
import (
"database/sql"
"fmt"
"os"
_ "github.com/go-sql-driver/mysql"
)
func GetDatabase() (*sql.DB, error) {
db, err := sql.Open("mysql", os.Getenv("DSN"))
return db, err
}
func main() {
db, err := GetDatabase()
if err != nil {
panic(err)
}
if err := db.Ping(); err != nil {
panic(err)
}
// ---
query, err := db.Query("SELECT name FROM status;")
if err != nil {
panic(err.Error())
}
var name string
for query.Next() {
err = query.Scan(&name)
if err != nil {
panic(err.Error())
}
fmt.Println(name)
}
// ---
fmt.Println("Successfully connected to PlanetScale!")
}
If I remove the section between the two comment lines, it will print out the 'successfully connected' message; however, if I try to run it including the query part, the outcome changes to:
panic: Error 1046: No database selected
Well, I decided to try with the query:
"USE database; SELECT name FROM status;"
However, the default driver won't take multiple statements and returns syntax error.
I know it seems like a simple question, but I already read the documentation from planetscale, from the go mysql driver, and I haven't been able to overcome this little issue. Any help will be greatly appreciated.
Looks like the problem is with the .env file. If I save the same information to a string variable within the same GetDatabase() function, and then pass it as argument instead of os.Getenv(), it connects normally. Can't tell exactly why that happens, but if someone can provide further information, I'll be thankful.
Edit:
when saving my credentials to an .env file, and using the built-in function os.Getenv("DSN"), the program will indeed connect to planetscale, but it won't select the given database in the DSN string. As mentioned in the question, selecting the database first and then running statements / queries doesn't work because of the constraints of the driver.
If I hard code the variables to a string variable, eg:
dsn := "root:mlc#tcp(127.0.0.1:3306)/app"
and then use that to open the connection, this time the program will use the desired database with any problem.

sql query runs two times with go

I'm trying to setup a connection with MySQL server. As you can see I'm trying to make a simple insert but every time I run my program, it seems that it has run the INSERT query two times.
I run my program via command line go run test_sql.go.
package main
import (
"fmt"
"database/sql"
_ "github.com/go-sql-driver/mysql"
)
func main() {
fmt.Println("Go MySQL Tutorial")
// Open up our database connection.
// I've set up a database on my local machine using phpmyadmin.
// The database is called testDb
db, err := sql.Open("mysql", "root:PASSWORD#tcp(127.0.0.1:3306)/test1")
// if there is an error opening the connection, handle it
if err != nil {
panic(err.Error())
}
// perform a db.Query insert
insert, err := db.Query("INSERT INTO TEST VALUES ( 1, 'TEST' )")
// if there is an error inserting, handle it
if err != nil {
panic(err.Error())
}
// be careful deferring Queries if you are using transactions
defer insert.Close()
// defer the close till after the main function has finished
// executing
defer db.Close()
}

ory/ladon manager with golang - Tables not created

github.com/ory/ladon is a library to manage role based access, written in golang. It contains a manager that is supposed to persist policies in the database and work with the database. The manager works fine for the in-memory case. When I use the manager to interface with sql, the required tables are not getting created.
db, err := sqlx.Open("mysql", "tx81:#tcp(127.0.0.1:3306)/policies")
......
err=db.Ping()
if err == nil {
fmt.Printf("Database is up")
}
warden := ladon.Ladon{
Manager: manager.NewSQLManager(db, nil),
}
var pol = &ladon.DefaultPolicy{
......
}
err = warden.Manager.Create(pol)
fmt.Printf("%s", err)
The error is printed as:
Table 'policies.ladon_policy' doesn't exist.
Why aren't the tables getting created?
Resources: https://github.com/ory/ladon#persistence
You need to call manager.CreateSchemas, which is never done by the manager itself.
You provide it with the schema name (postgresql only I believe) and table name to keep track of migration info.
eg:
if num, err := m.CreateSchemas("", "my_migration_table"); err != nil {
panic(err)
} else {
log.Infof("ran %d migrations", num)
}
It could probably use more documentation, you may want to file an issue with the author.

Apex Process exited before completing request when with Golang

All the answers for something like this are in Javascript, and I'm not sure if it applies in Go.
I've done this
func main() {
db, err := sql.Open("mysql", "db_details")
err = db.Ping()
if err != nil {
fmt.Println("Failed to prepare connection to database")
// log.Fatal("Error:", err.Error())
}
apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) {
fmt.Println(ctx)
return map[string]string{"hello": "world"}, nil
})
}
So I'm trying to hit my Amazon RDS MySql db using golang's sql driver.
I get this error
Error: function response: Response Id: <some_id> Process exited before completing request
From looking around, there are two causes - 1. I need Go's equivalent of context.done, or 2. I need to raise the timeout.
As I'm using Apex, I raised the timeout to be 300s, which is the maximum. No luck there.
I then tried going through the Apex code to see if there was a Context.Done defined or used anywhere - there isn't.
How do I get around this?

Why does sql.Open() return nil as error when it should not?

I am trying to connect to a mysql database.
I tried to see if I would get an error if I gave it wrong connection information but it still returns nil as error. Even If I shut down mysql completely it still does not return an error. What is the point of check for errors after this function if it does not return errors?
This is on Windows, I am using XAMPP and I don't have a password for the database. Username is "root".
import (
"database/sql"
"log"
_ "github.com/go-sql-driver/mysql"
)
func main() {
db, err := sql.Open("mysql", "root#tcp(127.0.0.1:3306)/dbname?charset=utf8")
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
SQL.Open only creates the DB object, but does not open any connections to the database. If you want to test your connections you have to execute a query to force opening a connection. The common way for this is to call Ping() on your DB object.
See http://golang.org/pkg/database/sql/#Open and http://golang.org/pkg/database/sql/#DB.Ping
Quoting from the doc of sql.Open():
Open may just validate its arguments without creating a connection to the database. To verify that the data source name is valid, call Ping.
As stated, Open() may not open any physical connection to the database server, but it will validate its arguments. That being said if arguments are valid, it may return nil error even if the database server is not reachable, or even if the host denoted by dataSourceName does not exist.
To answer your other question:
What is the point of check for errors after this function if it does not return errors?
You have to check returned errors because it can return errors. For example if the specified driverName is invalid, a non-nil error will be returned (see below).
To test if the database server is reachable, use DB.Ping(). But you can only use this if the returned error is nil, else the returned DB might also be nil (and thus calling the Ping() method on it may result in run-time panic):
if db, err := sql.Open("nonexistingdriver", "somesource"); err != nil {
fmt.Println("Error creating DB:", err)
fmt.Println("To verify, db is:", db)
} else {
err = db.Ping()
if err != nil {
fmt.Println("db.Ping failed:", err)
}
}
Output (try it on the Go Playground):
Error creating DB: sql: unknown driver "nonexistingdriver" (forgotten import?)
To verify, db is: <nil>
It turns out that this does not actually connect to the database right away.
Source: http://go-database-sql.org/accessing.html
To check the validity of connection, we can do this:
db, err := sql.Open("mysql", "root#tcp(127.0.0.1:3306)/dbname?charset=utf8")
err = db.Ping() // Need to do this to check that the connection is valid
if err != nil {
log.Fatal(err)
}