Gin + Golang + DB Connection Pooling - mysql

I would like to understand how does GIN ensures that each HTTP request gets a unique DB ( say MySQL ) connection. Here is one example code.
If you see, since 'db' is a global object and therefore, the API router.GET("/person/:age"... gets access to DB.
Now with load, I suppose GIN will have concurrency implemented internally. If yes, then how does it ensures that each request gets a different connection. If no, then it is single threaded imnplementation. Could anyone please correct my understanding.
package main
import (
// "bytes"
"database/sql"
"fmt"
"github.com/gin-gonic/gin"
_ "github.com/go-sql-driver/mysql"
"net/http"
)
func checkErr(err error) {
if err != nil {
panic(err)
} else {
fmt.Println("successful...")
}
}
func main() {
db, err := sql.Open("mysql", "abfl:abfl#tcp(127.0.0.1:3306)/abfl?charset=utf8")
checkErr(err)
defer db.Close()
// make sure connection is available
err = db.Ping()
checkErr(err)
type User struct {
age int
name string
}
router := gin.Default()
// Add API handlers here
// GET a user detail
router.GET("/person/:age", func(c *gin.Context) {
var (
user User
result gin.H
)
age := c.Param("age")
fmt.Println("input age : '%d'", age)
row := db.QueryRow("select age, name from user where age = ?", age)
err = row.Scan(&user.age, &user.name)
fmt.Printf("user : %+v\n", user)
if err != nil {
// If no results send null
result = gin.H{
"user": nil,
"count": 0,
}
} else {
result = gin.H{
"age": user.age,
"name": user.name,
"count": 1,
}
}
c.JSON(http.StatusOK, result)
})
router.Run(":3000")
}

Establishing a new SQL connection for each HTTP request is too heavy and has no sense.
In go there is no user-managable connection pool yet, it is handled internally by go implementation.
sql.DB is ready to be used concurrently, so there is no worry about it.
And GIN has nothing to do with SQL connections at all. It is fully your responsibility to handle queries/transactions properly.

Related

Golang - How to create helper mysql to generate uuid / uuid_short without create new connection? Here's the code

I'm trying to generate UUID_SHORT() to be an ID. Instead of using trigger, I get the UUID_SHORT() first and then insert as an ID. But, I'm confusing of how to create GetUUID() function as helper.
As my code bellow, it always setup new connection before generate the UUID_SHORT() which means there will be so many connection just to generate UUID_SHORT()
How to create func GetUUID() without creating new connection?
Here's the code:
package database
import (
"database/sql"
"fmt"
"log"
"os"
"time"
_ "github.com/go-sql-driver/mysql"
)
type conncetion struct {
sqlDB *sql.DB
}
func NewMysqlConnection(databaseConnection *sql.DB) *conncetion {
return &conncetion{databaseConnection}
}
func SetupMysqlDatabaseConnection() (db *sql.DB) {
var (
driver = os.Getenv("DB_DRIVERNAME")
username = os.Getenv("DB_USERNAME")
password = os.Getenv("DB_PASSWORD")
host = os.Getenv("DB_HOST")
port = os.Getenv("DB_PORT")
name = os.Getenv("DB_NAME")
)
connection := fmt.Sprintf("%s:%s#tcp(%s:%s)/%s?parseTime=true", username, password, host, port, name)
db, err := sql.Open(driver, connection)
if err != nil {
log.Fatal(err)
}
db.SetMaxOpenConns(100)
db.SetMaxIdleConns(100)
db.SetConnMaxLifetime(100 * time.Millisecond)
return
}
func (c *conncetion) GenerateUUID() (uuid uint64, err error) {
uuid = 0
queryGetUUID := c.sqlDB.QueryRow(`SELECT UUID_SHORT()`)
err = queryGetUUID.Scan(
&uuid,
)
return
}
func GetUUID() (uuid uint64, err error) {
mysql := SetupMysqlDatabaseConnection()
db := NewMysqlConnection(mysql)
uuid, err = db.GenerateUUID()
return
}
How about global variable ?
package database
import (
"database/sql"
"fmt"
"log"
"os"
"time"
_ "github.com/go-sql-driver/mysql"
)
type conncetion struct {
sqlDB *sql.DB
}
var globalConnection *conncetion
func GetDB() *conncetion {
return globalConnection
}
func NewMysqlConnection(databaseConnection *sql.DB) *conncetion {
return &conncetion{databaseConnection}
}
func SetupMysqlDatabaseConnection() (db *sql.DB) {
var (
driver = os.Getenv("DB_DRIVERNAME")
username = os.Getenv("DB_USERNAME")
password = os.Getenv("DB_PASSWORD")
host = os.Getenv("DB_HOST")
port = os.Getenv("DB_PORT")
name = os.Getenv("DB_NAME")
)
connection := fmt.Sprintf("%s:%s#tcp(%s:%s)/%s?parseTime=true", username, password, host, port, name)
db, err := sql.Open(driver, connection)
if err != nil {
log.Fatal(err)
}
db.SetMaxOpenConns(100)
db.SetMaxIdleConns(100)
db.SetConnMaxLifetime(100 * time.Millisecond)
return
}
func (c *conncetion) GenerateUUID() (uuid uint64, err error) {
uuid = 0
queryGetUUID := c.sqlDB.QueryRow(`SELECT UUID_SHORT()`)
err = queryGetUUID.Scan(
&uuid,
)
return
}
func GetUUID() (uuid uint64, err error) {
// mysql := SetupMysqlDatabaseConnection()
// db := NewMysqlConnection(mysql)
db = GetDB()
uuid, err = db.GenerateUUID()
return
}
// can be written on main.go / server.go
func init() {
globalConnection = NewMysqlConnection(database.SetupMysqlDatabaseConnection())
}
But I don't know it is good or not to open an idle connection and spamming request on 1 connection.
You're leveraging the database to generated the UUID for you, as long as you're letting the DB do it you'll need a connection to send the query.
You generally have two options here:
let the DB generate the UUID on the fly when inserting your data
generate the UUID in your own code before sending it
Generate ID on insert
This means you need to change the way you operate, your command for inserting data will need to look something like this:
INSERT INTO your_table(id, value)
VALUES (
UUID_SHORT(),
-- other values
);
This will automatically generate the ID for you during the insert, without needing to generate it before.
If you need to know the ID after the insert was performed, you have a few options, like using LAST_INSERT_ID() or querying the data you just created.
See this other question for more info.
Generate ID in code
You can use a package like github.com/google/uuid.
import "github.com/google/uuid"
func GenerateUUID() (uint32, error) {
id, err := uuid.NewRandom()
if err != nil {
return 0, err
}
return id.ID(), nil
}
Note that you can also get a string representation for the UUID, or you can cast it to uint64 easily from uint32.

How to open and close database connection

I'm beginner in go and now want to create a big project
my question is where should create connection and where close connection in http requests
now i declare db in main function and use in all requests and don't close connection:
package main
import (
"fmt"
"github.com/jinzhu/gorm"
"github.com/kataras/iris"
"github.com/kataras/iris/context"
"github.com/kataras/iris/middleware/recover"
_ "github.com/go-sql-driver/mysql"
)
var db *gorm.DB
func main() {
port := "8080"
app := iris.New()
app.Configure()
app.Logger().SetLevel("debug")
app.Use(recover.New())
db1, err := gorm.Open("mysql", "root:#/database?charset=utf8&parseTime=True&loc=Local")
//db1, err := gorm.Open("mysql", "payro:AEkCpNhd#/payro_db?charset=utf8&parseTime=True&loc=Local")
if err != nil {
fmt.Println(err.Error())
panic(err)
}
db=db1
app.Get("/", func(i context.Context) {
db.Exec("update tbl1 set col1=''")
i.Next()
})
app.Get("/test", func(i context.Context) {
db.Exec("update tbl2 set col2=''")
i.Next()
})
_ = app.Run(iris.Addr(":"+port), iris.WithConfiguration(iris.Configuration{
//DisableBodyConsumptionOnUnmarshal:true,
}), iris.WithoutServerError(iris.ErrServerClosed))
}
this code is ok?
Your code is perhaps a test/POC code.
In a production project, you could go with a MVC or any other kind of architecture as per your needs.
It would be hard to pinpoint the exact structure of your project.
But at the very least, you would want to create a db package which declares an interface for all DB related interactions.
e.g
type UserDBRepo interface{
AddUser(context.Context, *User)
GetUser(context.Context, uint64)
}
type userDBRepo struct{ //implements UserDBRepo
*sql.DB // or whatever type gorm.Open returns
}
func NewUserDBRepo(db *sql.DB) DBRepo{
return &dbRepo{DB: db}
}
The above basically represents a single RDBMS table for this example.
There could be n such files for n DB tables.
Now call NewUserDBRepo from the main.go and pass this instance to all services that need this DB.

insert query not working in golang

In my golang program, the insert MySQL query was not inserting the values. Even the syntax of the query was correct.It returns my error statement ie internal server error
package main
import (
"database/sql"
"net/http"
"log"
"golang.org/x/crypto/bcrypt"
"encoding/json"
_ "github.com/go-sql-driver/mysql"
"fmt"
)
const hashCost = 8
var db *sql.DB
func initDB(){
var err error
// Connect to the db
//you might have to change the connection string to add your database credentials
db, err = sql.Open("mysql",
"root:nfn#tcp(127.0.0.1:3306)/mydb")
if err != nil {
panic(err)
}
}
// Create a struct that models the structure of a user, both in the request body, and in the DB
type Credentials struct {
Password string `json:"password", db:"password"`
Username string `json:"username", db:"username"`
}
func Signup(w http.ResponseWriter, r *http.Request){
// Parse and decode the request body into a new `Credentials` instance
creds := &Credentials{}
err := json.NewDecoder(r.Body).Decode(creds)
if err != nil {
// If there is something wrong with the request body, return a 400 status
w.WriteHeader(http.StatusBadRequest)
return
}
// Salt and hash the password using the bcrypt algorithm
// The second argument is the cost of hashing, which we arbitrarily set as 8 (this value can be more or less, depending on the computing power you wish to utilize)
hashedPassword, err := bcrypt.GenerateFromPassword([]byte(creds.Password), 8)
// Next, insert the username, along with the hashed password into the database
// Sql query not working Here.How to solve this error
if _, err = db.Exec("INSERT INTO users(username,password) VALUES (?,?) ", creds.Username, string(hashedPassword)); err != nil {
// If there is any issue with inserting into the database, return a 500 error
w.WriteHeader(http.StatusInternalServerError)
return
}
// We reach this point if the credentials we correctly stored in the database, and the default status of 200 is sent back
}

How do I select the database to query when using Cloud SQL through App Engine?

I get the following error:
Could not query db: Error 1046: No database selected
I understand what the error message means. But I haven't been able to find the documentation where it says how to select a database.
Here is my code:
package main
import (
"database/sql"
"fmt"
"log"
"net/http"
"os"
"time"
"google.golang.org/appengine"
_ "github.com/go-sql-driver/mysql"
)
var db *sql.DB
func main() {
var (
connectionName = mustGetenv("CLOUDSQL_CONNECTION_NAME")
user = mustGetenv("CLOUDSQL_USER")
password = os.Getenv("CLOUDSQL_PASSWORD")
)
var err error
db, err = sql.Open("mysql ", fmt.Sprintf("%s:%s#cloudsql(%s)/", user, password, connectionName))
if err != nil {
log.Fatalf("Could not open db: %v", err)
}
http.HandleFunc("/", handler)
appengine.Main()
}
func handler(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
http.NotFound(w, r)
return
}
w.Header().Set("Content-Type", "text/plain")
rows, err := db.Query("INSERT INTO ping ( ping ) VALUES ( '" + time.Now().Format("2006-01-02 03:04:05") + "' );")
if err != nil {
http.Error(w, fmt.Sprintf("Could not query db: %v", err), 500)
return
}
defer rows.Close()
w.Write([]byte("OK"))
}
func mustGetenv(k string) string {
v := os.Getenv(k)
if v == "" {
log.Panicf("%s environment variable not set.", k)
}
return v
}
It looks like you specified the CONNECTION_NAME, but not the DB_NAME. According to the documentation (scroll down to the "GO > Companion process" section), you should open the connection as:
import (
"github.com/go-sql-driver/mysql"
)
dsn := fmt.Sprintf("%s:%s#tcp(%s)/%s",
dbUser,
dbPassword,
"127.0.0.1:3306",
dbName)
db, err := sql.Open("mysql", dsn)
This piece of code resembles yours, but you did not specify the dbName parameter. Bear in mind that the rest of the configuration should remain the same as you shared in your code, but you should just append the name of your database to the second parameter of the sql.Open() function.
In the example of connecting from App Engine Flexible to Cloud SQL using GO, the same procedure is identified:
db, err = sql.Open("mysql", dbName)
So I guess you should try with this change:
// Old connection opening
db, err = sql.Open("mysql ", fmt.Sprintf("%s:%s#cloudsql(%s)/", user, password, connectionName))
// New connection opening, including dbName
db, err = sql.Open("mysql ", fmt.Sprintf("%s:%s#cloudsql(%s)/%s", user, password, connectionName, dbName))
I am not really familiar with GoLang myself, but according to the documentation, that should work.

How to access Gorm in Revel Controller?

let me start by saying these are my first couple days of toying around in Go.
I'm trying to use the Revel framework with Gorm like this:
app/controllers/gorm.go
package controllers
import (
"fmt"
"go-testapp/app/models"
_ "github.com/go-sql-driver/mysql"
"github.com/jinzhu/gorm"
"github.com/revel/revel"
)
var DB gorm.DB
func InitDB() {
var err error
DB, err = gorm.Open("mysql", "root:#/go-testapp?charset=utf8&parseTime=True")
if err != nil {
panic(err)
}
DB.LogMode(true)
DB.AutoMigrate(models.User{})
}
type GormController struct {
*revel.Controller
DB *gorm.DB
}
app/controller/app.go
package controllers
import (
"fmt"
"go-bingo/app/models"
_ "github.com/go-sql-driver/mysql"
"github.com/revel/revel"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhu", Age: 18}
fmt.Println(c.DB)
c.DB.NewRecord(user)
c.DB.Create(&user)
return c.RenderJson(user)
}
After running it results in:
runtime error: invalid memory address or nil pointer dereference on line 19 c.DB.NewRecord(user)
It successfully creates the datatables with automigrate, but I have no idea how I should use Gorm in my controller.
Any hints in the right direction?
Important note
it's just replacement for GORP of original example of the Revel. And it comes with some pitfalls of the origin. This answer can be used as drop-in replacement for the original one. But it doesn't solve the pitfalls.
Please, take a look comments of this unswer and #MaxGabriel's answer that solves the pitfalls.
I'd recommend to use #MaxGabriel's solution to protect you application against some kinds of slow-* DDoS attacks. And to reduce (in some cases) DB pressure.
Original answer
#rauyran rights, you have to invoke InitDB inside init function (into controllers package).
Full example here (too much):
Tree
/app
/controllers
app.go
gorm.go
init.go
/models
user.go
[...]
user.go
// models/user.go
package models
import "time" // if you need/want
type User struct { // example user fields
Id int64
Name string
EncryptedPassword []byte
Password string `sql:"-"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt time.Time // for soft delete
}
gorm.go
//controllers/gorm.go
package controllers
import (
"github.com/jinzhu/gorm"
_ "github.com/lib/pq" // my example for postgres
// short name for revel
r "github.com/revel/revel"
// YOUR APP NAME
"yourappname/app/models"
"database/sql"
)
// type: revel controller with `*gorm.DB`
// c.Txn will keep `Gdb *gorm.DB`
type GormController struct {
*r.Controller
Txn *gorm.DB
}
// it can be used for jobs
var Gdb *gorm.DB
// init db
func InitDB() {
var err error
// open db
Gdb, err = gorm.Open("postgres", "user=uname dbname=udbname sslmode=disable password=supersecret")
if err != nil {
r.ERROR.Println("FATAL", err)
panic( err )
}
Gdb.AutoMigrate(&models.User{})
// unique index if need
//Gdb.Model(&models.User{}).AddUniqueIndex("idx_user_name", "name")
}
// transactions
// This method fills the c.Txn before each transaction
func (c *GormController) Begin() r.Result {
txn := Gdb.Begin()
if txn.Error != nil {
panic(txn.Error)
}
c.Txn = txn
return nil
}
// This method clears the c.Txn after each transaction
func (c *GormController) Commit() r.Result {
if c.Txn == nil {
return nil
}
c.Txn.Commit()
if err := c.Txn.Error; err != nil && err != sql.ErrTxDone {
panic(err)
}
c.Txn = nil
return nil
}
// This method clears the c.Txn after each transaction, too
func (c *GormController) Rollback() r.Result {
if c.Txn == nil {
return nil
}
c.Txn.Rollback()
if err := c.Txn.Error; err != nil && err != sql.ErrTxDone {
panic(err)
}
c.Txn = nil
return nil
}
app.go
package controllers
import(
"github.com/revel/revel"
"yourappname/app/models"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhup"}
c.Txn.NewRecord(user)
c.Txn.Create(&user)
return c.RenderJSON(user)
}
init.go
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB) // invoke InitDB function before
revel.InterceptMethod((*GormController).Begin, revel.BEFORE)
revel.InterceptMethod((*GormController).Commit, revel.AFTER)
revel.InterceptMethod((*GormController).Rollback, revel.FINALLY)
}
As you can see, it's like Revel's Booking modified for GORM.
Works fine for me. Result:
{
"Id": 5,
"Name": "Jinzhup",
"EncryptedPassword": null,
"Password": "",
"CreatedAt": "2014-09-22T17:55:14.828661062+04:00",
"UpdatedAt": "2014-09-22T17:55:14.828661062+04:00",
"DeletedAt": "0001-01-01T00:00:00Z"
}
This answer is derived from #IvanBlack's answer, at his suggestion. His version is a direct translation of the Revel example code, but we identified some problems with the original code that this answer fixes. The main changes it makes are:
The entire HTTP request is no longer wrapped by a database transaction. Wrapping the entire HTTP request keeps open the transaction far longer than necessary, which could cause a number of problems:
Your transactions will hold locks on the database, and many transactions holding locks could lead to deadlocks
More database resources are used
These problems are magnified if your HTTP requests aren't mostly limited by your SQL database access. E.g. if your HTTP request blocks for 30 seconds on making an external HTTP request or accessing another database, the slowdown on those services could affect your SQL database.
Because the transaction is no longer automatically being checked for errors at the end of the HTTP request, errors are checked as soon as the database insert is made. This is more correct as well: Imagine that after inserting the User struct into a database, you then stored the User.Id in another database like Redis. This would be fine if the database insert worked, but if it failed and you didn't immediately check the error, you would insert the default int64 value of 0 into Redis (before later rolling back only the SQL transaction).
Tree
/app
/controllers
app.go
gorm.go
init.go
/models
user.go
[...]
user.go
// models/user.go
package models
import "time" // if you need/want
type User struct { // example user fields
Id int64
Name string
EncryptedPassword []byte
Password string `sql:"-"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt time.Time // for soft delete
}
gorm.go
//controllers/gorm.go
package controllers
import (
"github.com/jinzhu/gorm"
_ "github.com/lib/pq" // my example for postgres
// short name for revel
r "github.com/revel/revel"
)
// type: revel controller with `*gorm.DB`
type GormController struct {
*r.Controller
DB *gorm.DB
}
// it can be used for jobs
var Gdb *gorm.DB
// init db
func InitDB() {
var err error
// open db
Gdb, err = gorm.Open("postgres", "user=USERNAME dbname=DBNAME sslmode=disable")
Gdb.LogMode(true) // Print SQL statements
if err != nil {
r.ERROR.Println("FATAL", err)
panic(err)
}
}
func (c *GormController) SetDB() r.Result {
c.DB = Gdb
return nil
}
init.go
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB) // invoke InitDB function before
revel.InterceptMethod((*GormController).SetDB, revel.BEFORE)
}
app.go
package controllers
import(
"github.com/revel/revel"
"yourappname/app/models"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhup"} // Note: In practice you should initialize all struct fields
if err := c.DB.Create(&user).Error; err != nil {
panic(err)
}
return c.RenderJSON(user)
}
Result:
{
"Id": 5,
"Name": "Jinzhup",
"EncryptedPassword": null,
"Password": "",
"CreatedAt": "2014-09-22T17:55:14.828661062+04:00",
"UpdatedAt": "2014-09-22T17:55:14.828661062+04:00",
"DeletedAt": "0001-01-01T00:00:00Z"
}
Your error is caused because you haven't initialised your c.DB database variable, it's still nil.
In your controllers/init.go file make sure you are calling revel.OnAppStart(InitDB). It should look something like this:
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB)
// maybe some other init things
}
OR you need to pass in a pointer to AutoMigrate?
DB.AutoMigrate(&models.User{})