How to open and close database connection - mysql

I'm beginner in go and now want to create a big project
my question is where should create connection and where close connection in http requests
now i declare db in main function and use in all requests and don't close connection:
package main
import (
"fmt"
"github.com/jinzhu/gorm"
"github.com/kataras/iris"
"github.com/kataras/iris/context"
"github.com/kataras/iris/middleware/recover"
_ "github.com/go-sql-driver/mysql"
)
var db *gorm.DB
func main() {
port := "8080"
app := iris.New()
app.Configure()
app.Logger().SetLevel("debug")
app.Use(recover.New())
db1, err := gorm.Open("mysql", "root:#/database?charset=utf8&parseTime=True&loc=Local")
//db1, err := gorm.Open("mysql", "payro:AEkCpNhd#/payro_db?charset=utf8&parseTime=True&loc=Local")
if err != nil {
fmt.Println(err.Error())
panic(err)
}
db=db1
app.Get("/", func(i context.Context) {
db.Exec("update tbl1 set col1=''")
i.Next()
})
app.Get("/test", func(i context.Context) {
db.Exec("update tbl2 set col2=''")
i.Next()
})
_ = app.Run(iris.Addr(":"+port), iris.WithConfiguration(iris.Configuration{
//DisableBodyConsumptionOnUnmarshal:true,
}), iris.WithoutServerError(iris.ErrServerClosed))
}
this code is ok?

Your code is perhaps a test/POC code.
In a production project, you could go with a MVC or any other kind of architecture as per your needs.
It would be hard to pinpoint the exact structure of your project.
But at the very least, you would want to create a db package which declares an interface for all DB related interactions.
e.g
type UserDBRepo interface{
AddUser(context.Context, *User)
GetUser(context.Context, uint64)
}
type userDBRepo struct{ //implements UserDBRepo
*sql.DB // or whatever type gorm.Open returns
}
func NewUserDBRepo(db *sql.DB) DBRepo{
return &dbRepo{DB: db}
}
The above basically represents a single RDBMS table for this example.
There could be n such files for n DB tables.
Now call NewUserDBRepo from the main.go and pass this instance to all services that need this DB.

Related

Learning Go: How to use http.HandlerFunc?

Trying To Learn Go
My Test Project
I am building a simple API to communicate with a light SQL database.
I created this function, that gets all hosts from the DB table.
As I understand, my function should take a pointer to the DB and return me http.HandlerFunc.
func (h HostController) GetAllHosts(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var host entities.Host
var hosts []entities.Host
var errorMessage entities.Error
rows, err := db.Query("SELECT * FROM hosts WHERE hosts = ?", host)
if err != nil {
errorMessage.Message = "Error: Get all hosts"
utils.SendError(w, http.StatusInternalServerError, errorMessage)
return
}
defer rows.Close()
for rows.Next() {
err := rows.Scan(&host.ID, &host.Uuid, &host.Name, &host.IPAddress)
if err != nil {
errorMessage.Message = "Error: (Get all hosts) Can't scan rows"
utils.SendError(w, http.StatusInternalServerError, errorMessage)
return
}
hosts = append(hosts, host)
}
////If hosts slice is empty -> no data in database
if len(hosts) == 0 {
errorMessage.Message = "No found hosts in database"
utils.SendError(w, http.StatusInternalServerError, errorMessage)
return
}
//Convert containers content to JSON representation
w.Header().Set("Content-Type", "application/json")
utils.SendSuccess(w, hosts)
}
}
Now, Until here everything should be good, but I don't know how to implement this on main.go
build layout:
.
├── controllers
│ └── hostcontroller.go
│
└── main.go
and this is how I am trying to implement it on the main.go
package main
import (
"examProg/config"
"examProg/controllers"
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
_ "github.com/mattn/go-sqlite3"
)
func main() {
db := config.GetDB() //Establishing connection to DBase
{
router := mux.NewRouter()
router.Handle("/host", controllers.HostController.GetAllHosts(db))
fmt.Println("Server is running at port: 2030")
log.Println("Server Running OK | Port: 2030")
log.Fatal(http.ListenAndServe(":2030", router))
}
}
I don't understand how to implement the http.HandlerFunc 🙃
The expression controllers.HostController.GetAllHosts is a method expression. The result of a method expression is a function that, in addition to the method's argument's, takes an instance of the receiver's type as its first argument.
Hence the "not enough arguments in call ..." error.
To illustrate:
f := controllers.HostController.GetAllHosts
fmt.Println(reflect.TypeOf(f))
// output: func(controllers.HostController, *sql.DB)
So, to make the code compile, you need to pass an instance of controllers.HostController to the call, i.e.
controllers.HostController.GetAllHosts(controllers.HostController{}, db)
As you can see, it ain't pretty. Method expressions have their use, I'm sure, but I haven't encountered them much. What I see most often instead, is the use of method values.
h := controllers.HostController{}
router.Handle("/host", h.GetAllHosts(db))

Golang - How to create helper mysql to generate uuid / uuid_short without create new connection? Here's the code

I'm trying to generate UUID_SHORT() to be an ID. Instead of using trigger, I get the UUID_SHORT() first and then insert as an ID. But, I'm confusing of how to create GetUUID() function as helper.
As my code bellow, it always setup new connection before generate the UUID_SHORT() which means there will be so many connection just to generate UUID_SHORT()
How to create func GetUUID() without creating new connection?
Here's the code:
package database
import (
"database/sql"
"fmt"
"log"
"os"
"time"
_ "github.com/go-sql-driver/mysql"
)
type conncetion struct {
sqlDB *sql.DB
}
func NewMysqlConnection(databaseConnection *sql.DB) *conncetion {
return &conncetion{databaseConnection}
}
func SetupMysqlDatabaseConnection() (db *sql.DB) {
var (
driver = os.Getenv("DB_DRIVERNAME")
username = os.Getenv("DB_USERNAME")
password = os.Getenv("DB_PASSWORD")
host = os.Getenv("DB_HOST")
port = os.Getenv("DB_PORT")
name = os.Getenv("DB_NAME")
)
connection := fmt.Sprintf("%s:%s#tcp(%s:%s)/%s?parseTime=true", username, password, host, port, name)
db, err := sql.Open(driver, connection)
if err != nil {
log.Fatal(err)
}
db.SetMaxOpenConns(100)
db.SetMaxIdleConns(100)
db.SetConnMaxLifetime(100 * time.Millisecond)
return
}
func (c *conncetion) GenerateUUID() (uuid uint64, err error) {
uuid = 0
queryGetUUID := c.sqlDB.QueryRow(`SELECT UUID_SHORT()`)
err = queryGetUUID.Scan(
&uuid,
)
return
}
func GetUUID() (uuid uint64, err error) {
mysql := SetupMysqlDatabaseConnection()
db := NewMysqlConnection(mysql)
uuid, err = db.GenerateUUID()
return
}
How about global variable ?
package database
import (
"database/sql"
"fmt"
"log"
"os"
"time"
_ "github.com/go-sql-driver/mysql"
)
type conncetion struct {
sqlDB *sql.DB
}
var globalConnection *conncetion
func GetDB() *conncetion {
return globalConnection
}
func NewMysqlConnection(databaseConnection *sql.DB) *conncetion {
return &conncetion{databaseConnection}
}
func SetupMysqlDatabaseConnection() (db *sql.DB) {
var (
driver = os.Getenv("DB_DRIVERNAME")
username = os.Getenv("DB_USERNAME")
password = os.Getenv("DB_PASSWORD")
host = os.Getenv("DB_HOST")
port = os.Getenv("DB_PORT")
name = os.Getenv("DB_NAME")
)
connection := fmt.Sprintf("%s:%s#tcp(%s:%s)/%s?parseTime=true", username, password, host, port, name)
db, err := sql.Open(driver, connection)
if err != nil {
log.Fatal(err)
}
db.SetMaxOpenConns(100)
db.SetMaxIdleConns(100)
db.SetConnMaxLifetime(100 * time.Millisecond)
return
}
func (c *conncetion) GenerateUUID() (uuid uint64, err error) {
uuid = 0
queryGetUUID := c.sqlDB.QueryRow(`SELECT UUID_SHORT()`)
err = queryGetUUID.Scan(
&uuid,
)
return
}
func GetUUID() (uuid uint64, err error) {
// mysql := SetupMysqlDatabaseConnection()
// db := NewMysqlConnection(mysql)
db = GetDB()
uuid, err = db.GenerateUUID()
return
}
// can be written on main.go / server.go
func init() {
globalConnection = NewMysqlConnection(database.SetupMysqlDatabaseConnection())
}
But I don't know it is good or not to open an idle connection and spamming request on 1 connection.
You're leveraging the database to generated the UUID for you, as long as you're letting the DB do it you'll need a connection to send the query.
You generally have two options here:
let the DB generate the UUID on the fly when inserting your data
generate the UUID in your own code before sending it
Generate ID on insert
This means you need to change the way you operate, your command for inserting data will need to look something like this:
INSERT INTO your_table(id, value)
VALUES (
UUID_SHORT(),
-- other values
);
This will automatically generate the ID for you during the insert, without needing to generate it before.
If you need to know the ID after the insert was performed, you have a few options, like using LAST_INSERT_ID() or querying the data you just created.
See this other question for more info.
Generate ID in code
You can use a package like github.com/google/uuid.
import "github.com/google/uuid"
func GenerateUUID() (uint32, error) {
id, err := uuid.NewRandom()
if err != nil {
return 0, err
}
return id.ID(), nil
}
Note that you can also get a string representation for the UUID, or you can cast it to uint64 easily from uint32.

Do I need to close connection in Gorm? [duplicate]

So far the hardest part of Go has been understanding how to organize code. It seems incredibly simple on it's face but every time I've tried to do anything I run into circular imports or things like "exported func Start returns unexported type models.dbStore, which can be annoying to use".
Using the following code how do I call db.Close() or am I really not understanding how I'm supposed to provide the database to my models. Here's what I've got:
App.go
package app
import (
"database/sql"
// Comment
_ "github.com/mattn/go-sqlite3"
)
var (
// DB The database connection
db *sql.DB
)
// Setup Sets up the many many app settings
func Setup() {
d, err := sql.Open("sqlite3", "./foo.db")
if err != nil {
panic(err)
}
// TODO: How does the DB get closed?
// defer db.Close()
db = d
}
// GetDB Returns a reference to the database
func GetDB() *sql.DB {
return db
}
Users.go
package models
import (
"github.com/proj/org/app"
)
// User struct
type User struct {
ID int
}
// CreateUser Creates a user
func (u *User) CreateUser() (int64, error) {
// For the sake of brevity just make sure you can
// "connect" to the database
if err := app.GetDB().Ping(); err != nil {
panic(err)
}
return 1234, nil
}
main.go
package main
import (
"fmt"
"net/http"
_ "github.com/mattn/go-sqlite3"
"github.com/proj/org/app"
"github.com/proj/org/models"
)
func homeHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "You are home")
}
func subscribeHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Subscribing...")
u := models.User{}
u.CreateUser()
}
func main() {
fmt.Println("Running")
app.Setup()
http.HandleFunc("/", homeHandler)
http.HandleFunc("/subscribe", subscribeHandler)
err := http.ListenAndServe(":9090", nil)
if err != nil {
panic(err)
}
}
I thought about doing a app.Shutdown() but that wouldn't work for my most normal use case which is CTRL-C. It would seem if I don't close the database the DB connections would just grow... Just trying to understand.
It's not necessary to close the DB.
The returned DB is safe for concurrent use by multiple goroutines and
maintains its own pool of idle connections. Thus, the Open function
should be called just once. It is rarely necessary to close a DB.
From: https://golang.org/pkg/database/sql/#Open
When your program exits then any open connection is closed, it does not stay open somewhere in the ether awaiting your program to start again, so do not worry about the connections "growing" when you CTRL-C your app.
If, however, you still want to close it, then you can just export a CloseDB function the same way you do with GetDB.
App.go
// ...
func CloseDB() error {
return db.Close()
}
main.go
// ...
func main() {
// ...
app.Setup()
defer app.CloseDB()
// ...
}

Gin + Golang + DB Connection Pooling

I would like to understand how does GIN ensures that each HTTP request gets a unique DB ( say MySQL ) connection. Here is one example code.
If you see, since 'db' is a global object and therefore, the API router.GET("/person/:age"... gets access to DB.
Now with load, I suppose GIN will have concurrency implemented internally. If yes, then how does it ensures that each request gets a different connection. If no, then it is single threaded imnplementation. Could anyone please correct my understanding.
package main
import (
// "bytes"
"database/sql"
"fmt"
"github.com/gin-gonic/gin"
_ "github.com/go-sql-driver/mysql"
"net/http"
)
func checkErr(err error) {
if err != nil {
panic(err)
} else {
fmt.Println("successful...")
}
}
func main() {
db, err := sql.Open("mysql", "abfl:abfl#tcp(127.0.0.1:3306)/abfl?charset=utf8")
checkErr(err)
defer db.Close()
// make sure connection is available
err = db.Ping()
checkErr(err)
type User struct {
age int
name string
}
router := gin.Default()
// Add API handlers here
// GET a user detail
router.GET("/person/:age", func(c *gin.Context) {
var (
user User
result gin.H
)
age := c.Param("age")
fmt.Println("input age : '%d'", age)
row := db.QueryRow("select age, name from user where age = ?", age)
err = row.Scan(&user.age, &user.name)
fmt.Printf("user : %+v\n", user)
if err != nil {
// If no results send null
result = gin.H{
"user": nil,
"count": 0,
}
} else {
result = gin.H{
"age": user.age,
"name": user.name,
"count": 1,
}
}
c.JSON(http.StatusOK, result)
})
router.Run(":3000")
}
Establishing a new SQL connection for each HTTP request is too heavy and has no sense.
In go there is no user-managable connection pool yet, it is handled internally by go implementation.
sql.DB is ready to be used concurrently, so there is no worry about it.
And GIN has nothing to do with SQL connections at all. It is fully your responsibility to handle queries/transactions properly.

global var out out init.go in revel

(EDITING to fix capitalization and add context)
In revel's init.go, I have a global var: DB.
package app
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
"github.com/revel/revel"
)
var DB *sql.DB
func InitDB() {
connstring := fmt.Sprintf("revel:revel#tcp(localhost:3336)/revel")
var err error
DB, err = sql.Open("mysql", connstring)
if err != nil {
revel.INFO.Println("DB Error", err)
}
}
How do I get that db variable in the rest of the app? Specifically, I was going to use this var in the models.
To start, how would I get it to work in this controller?
controllers/app.go
package controller
func (c App) Index() revel.Result {
rows, err := app.DB.Exec("SELECT * FROM test_table")
//do something with rows
return c.Render()
}
Revel makes you "roll your own db" so it's up to you where connections are made and how to implement them.
Your InitDB() function must setup a public variable, e.g. DB (meaning its name is capitalized) which you will then access via the package that it lives in.
So notice your app/init.go is inside of package app:
package app
var DB *sql.DB
func InitDB() { DB = .... }
In Go, when your application is running each package acts as a sort of "singleton" so that anywhere else in the application where you access:
app.DB
This will look for the variable called DB in the app package. And if you have already run InitDB() to connect to the database, then app.DB will allow all other packages to use the database.