when I run this I get Too many connections error unfortunately I couldn't find the solution, I will generate 50 million data
main.go content
package main
import (
"dopinghafiza-videoplayer-mysql/mysql"
_ "github.com/go-sql-driver/mysql"
)
func main() {
var player mysql.Player
player.Muted = 1
player.UserId = 55
player.PlayBackRate = "2.55"
player.Volume = "4.5888"
player.Volume = "2.55"
connection := mysql.GetConnection()
for i := 0; i < 100000; i++ {
player.Muted = int8(i % 2)
player.UserId = i
mysql.InsertPlayer(player, connection)
}
}
insert.go file content
package mysql
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
type Player struct {
UserId int
Muted int8
PlayBackRate string
Volume string
}
func InsertPlayer(player Player, db *sql.DB) {
res := db.QueryRow("insert into players (muted,user_id,volume,play_back_rate) values (?,?,?,?)", player.Muted, player.UserId, player.Volume, player.PlayBackRate)
db.Close()
fmt.Println(res)
}
You should not close connections; Go comes indeed with some connection pooling.
You should also use Exec() instead of QueryRow, because you are keeping connection open since MySQL still has to send data (although it looks odd).
func InsertPlayer(player Player, db *sql.DB) {
res, err := db.Exec("insert into players (muted,user_id,volume,play_back_rate) values (?,?,?,?)",
player.Muted,
player.UserId,
player.Volume,
player.PlayBackRate)
if err != nil {
// and handle errors!
}
fmt.Println(res.RowsAffected())
}
You shouldn't open and close the connection each time; that's wrong and has a huge cost for your application.
A better method is to use a singleton pattern, open the connection once, use it several times, and close the connection when your application closes, not after each query.
You can find the sample code for the singleton pattern here.
How to create singleton DB class in GoLang
Login to mysql using mysql -u root -p
Use this command to see the value of max_connections show variables like %max_conne%;
Increase the value of max_connections using set global max_connections = 4096;
This will solve your issue.
Related
I'm collecting information and metrics about RDS Instances from CloudWatch and RDS services.
I tried to get metrics and information using the following methods:
DescribeDBInstances
GetMetricData
Unfortunately, there is no information about the Total Memory of DB Instance.
I'm assuming you want to get the memory for the underlying EC2 that you run your RDS.
If you need the memory allocated to the DB, it'll depend on the DB engine type. For example, in PostgreSQL, you need to query wal_buffers.
To get the instance details for your RDS, you need to get the instance type from rds.DescribeDBInstances and then query for the instance type details from ec2.DescribeInstanceTypes
import (
"context"
"log"
"strings"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/ec2"
ec2types "github.com/aws/aws-sdk-go-v2/service/ec2/types"
"github.com/aws/aws-sdk-go-v2/service/rds"
)
func getRdsMemoryMB(cfg aws.Config) int64 {
rdsClient := rds.NewFromConfig(cfg)
// get the instance details
input := rds.DescribeDBInstancesInput{
// if you omit this paramter, DescribeDBInstances will return the info for all instances
DBInstanceIdentifier: aws.String("instance-1-ARN"),
}
output, err := rdsClient.DescribeDBInstances(context.Background(),
&input,
func(opt *rds.Options) { opt.Region = "us-east-1" })
if err != nil {
log.Fatal(err)
}
// get the instance type details
ec2Client := ec2.NewFromConfig(cfg)
instanceName := strings.TrimPrefix(*output.DBInstances[0].DBInstanceClass, "db.")
params := ec2.DescribeInstanceTypesInput{
InstanceTypes: []ec2types.InstanceType{ec2types.InstanceType(instanceName)},
}
ec2Output, err := ec2Client.DescribeInstanceTypes(context.Background(), ¶ms)
if err != nil {
log.Fatal(err)
}
return *ec2Output.InstanceTypes[0].MemoryInfo.SizeInMiB
}
After connecting to a Planetscale database using DSN with the following structure:
user:password#tcp(host)/database?tls=true
I decided to test it by running a simple query. Here's my main.go file:
package main
import (
"database/sql"
"fmt"
"os"
_ "github.com/go-sql-driver/mysql"
)
func GetDatabase() (*sql.DB, error) {
db, err := sql.Open("mysql", os.Getenv("DSN"))
return db, err
}
func main() {
db, err := GetDatabase()
if err != nil {
panic(err)
}
if err := db.Ping(); err != nil {
panic(err)
}
// ---
query, err := db.Query("SELECT name FROM status;")
if err != nil {
panic(err.Error())
}
var name string
for query.Next() {
err = query.Scan(&name)
if err != nil {
panic(err.Error())
}
fmt.Println(name)
}
// ---
fmt.Println("Successfully connected to PlanetScale!")
}
If I remove the section between the two comment lines, it will print out the 'successfully connected' message; however, if I try to run it including the query part, the outcome changes to:
panic: Error 1046: No database selected
Well, I decided to try with the query:
"USE database; SELECT name FROM status;"
However, the default driver won't take multiple statements and returns syntax error.
I know it seems like a simple question, but I already read the documentation from planetscale, from the go mysql driver, and I haven't been able to overcome this little issue. Any help will be greatly appreciated.
Looks like the problem is with the .env file. If I save the same information to a string variable within the same GetDatabase() function, and then pass it as argument instead of os.Getenv(), it connects normally. Can't tell exactly why that happens, but if someone can provide further information, I'll be thankful.
Edit:
when saving my credentials to an .env file, and using the built-in function os.Getenv("DSN"), the program will indeed connect to planetscale, but it won't select the given database in the DSN string. As mentioned in the question, selecting the database first and then running statements / queries doesn't work because of the constraints of the driver.
If I hard code the variables to a string variable, eg:
dsn := "root:mlc#tcp(127.0.0.1:3306)/app"
and then use that to open the connection, this time the program will use the desired database with any problem.
I am trying to setup a REST API using Gin and Gorm. Following is my main.go:
package main
import (
"app/config"
"app/service"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.Default()
// Connect to database
config.ConnectDatabase()
r.GET("/books", service.FindBooks)
// Run the server
r.Run()
}
And in my config/setup.go, I am trying to connect to DB like so
package config
import (
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/mysql"
)
var DB *gorm.DB
// ConnectDatabase : Setup connection to database
func ConnectDatabase() {
database, err := gorm.Open("mysql", "root:password#tcp(127.0.0.1:3306)/test_db")
database.SingularTable(true)
if err != nil {
panic("Failed to connect to database!")
}
DB = database
}
In my service/book.go, I have the following logic:
package service
import (
"errors"
"app/config"
"app/model"
"fmt"
"net/http"
"github.com/gin-gonic/gin"
)
// FindBooks : Get all books
func FindBooks(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"data": FindAllBooks()})
}
// FindAllBooks : Fetch all book from DB
func FindAllBooks() []model.Book {
var book []model.Book
config.DB.Find(&book)
for i := 0; i < len(book); i++ {
fmt.Println(book[i])
}
return book
}
And my model/Book.go is defined as follows:
package model
type Book struct {
id int64 `json:"id" gorm:"primary_key"`
label string `json:"label" gorm:"type:varchar(255)"`
}
When I run the application using go run main.go, following is the log I can see:
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET /books --> feedconsumer/service.FindBooks (3 handlers)
[GIN-debug] Environment variable PORT is undefined. Using port :8080 by default
[GIN-debug] Listening and serving HTTP on :8080
{0 }
[GIN] 2021/03/09 - 12:43:43 | 200 | 2.313864ms | ::1 | GET "/books"
Basically, the {0 } means the object is not fetched actually. What am I missing here?
GORM uses snake_case for defining field of models. Link to docs
So for your case, it is basically:
package model
type Book struct {
ID int64 `json:"id" gorm:"primary_key"`
Label string `json:"label" gorm:"type:varchar(255)"`
}
I guess you could configure logger for GORM like https://gorm.io/docs/logger.html to be able to see actual db query in those logs. At first glance you are doing everything in the right way as described in https://gorm.io/docs/query.html#Retrieving-all-objects
but I personally met some unpredictable GORM behavior which was not so obvious from doc
I'm trying to setup a connection with MySQL server. As you can see I'm trying to make a simple insert but every time I run my program, it seems that it has run the INSERT query two times.
I run my program via command line go run test_sql.go.
package main
import (
"fmt"
"database/sql"
_ "github.com/go-sql-driver/mysql"
)
func main() {
fmt.Println("Go MySQL Tutorial")
// Open up our database connection.
// I've set up a database on my local machine using phpmyadmin.
// The database is called testDb
db, err := sql.Open("mysql", "root:PASSWORD#tcp(127.0.0.1:3306)/test1")
// if there is an error opening the connection, handle it
if err != nil {
panic(err.Error())
}
// perform a db.Query insert
insert, err := db.Query("INSERT INTO TEST VALUES ( 1, 'TEST' )")
// if there is an error inserting, handle it
if err != nil {
panic(err.Error())
}
// be careful deferring Queries if you are using transactions
defer insert.Close()
// defer the close till after the main function has finished
// executing
defer db.Close()
}
I'm using the github.com/go-sql-driver/mysql driver for go.
I open a database:
db, err := sql.Open("mysql", str)
Then I have two functions that are called 200 times each with following mysql code:
rows, err := db.Query("select name from beehives")
if err != nil {
panic(err)
}
defer rows.Close()
The second:
err = db.QueryRow("select id, secret, shortname from beehives where shortname = ?", beehive).Scan(&id, &secre
switch {
case err == sql.ErrNoRows:
err = errors.New("Beehive '"+beehive+"' not found.")
case err != nil:
panic("loginBeehive: "+ err.Error())
default:
// ... do the work
The first one is panicing.
How can there be more than one connection when I open the database only once and how do I close them?
sql.Open doesn't really open a connection to your database.
A sql.DB maintains a pool of connections to your database. Each time you query your database your program will try to get a connection from this pool or create a new one otherwise. These connections are than put back into the pool once you close them.
This is what rows.Close() does.
Your db.QueryRow("...") does the same thing internally when you call Scan(...).
The basic problem is that you're creating too many queries, of which each one needs a connection, but you are not closing your connections fast enough. This way your program has to create a new connection for each query.
You can limit the maximum number of connections your program uses by calling SetMaxOpenConns on your sql.DB.
See http://go-database-sql.org/surprises.html for more information.
The *DB object that you get back from sql.Open doesn't corresponds to a single connection. It is better thought as a handle for the database: it manages a connection pool for you.
You can control the number of open connections with `(*DB).SetMaxOpenConns and its pair for idling connections.
So basically what happens here is that db.Query and db.QueryRow tries to acquire a connection for themselves and the DB handle doesn't put any restrictions on the number of simultaneous connections so your code panics when it opens more than what mysql can handle.
Try to make prepared statements db.Prepare(query string) (*Stmt, error) and than stmt.Query or stmt.Exec and than stmt.Close to reuse connections.
hi can you try close connection after used
db, err := sql.Open("mysql", str)
defer db.Close() // close after end scope
My program is connecting always to database. (Realtime Face Recognition for Attendance)
Therefore opening and closing database connection is worthless.
Therefore it's keep opens the database connection only initializing the program.
func GetAllFaces() interface{} {
OpenDatabaseConnection() ...
}
But access database later, increased the no of connection and crashed the program. But closing the rows object kept no of active connection at minimum. (for me 1)
func SaveAttendance(faceId int, date time.Time) error {
sqlQuery := fmt.Sprintf("SELECT ... "))
rows, err := DB.Query(sqlQuery) ...
err = rows.Close()
return err
}