So far the hardest part of Go has been understanding how to organize code. It seems incredibly simple on it's face but every time I've tried to do anything I run into circular imports or things like "exported func Start returns unexported type models.dbStore, which can be annoying to use".
Using the following code how do I call db.Close() or am I really not understanding how I'm supposed to provide the database to my models. Here's what I've got:
App.go
package app
import (
"database/sql"
// Comment
_ "github.com/mattn/go-sqlite3"
)
var (
// DB The database connection
db *sql.DB
)
// Setup Sets up the many many app settings
func Setup() {
d, err := sql.Open("sqlite3", "./foo.db")
if err != nil {
panic(err)
}
// TODO: How does the DB get closed?
// defer db.Close()
db = d
}
// GetDB Returns a reference to the database
func GetDB() *sql.DB {
return db
}
Users.go
package models
import (
"github.com/proj/org/app"
)
// User struct
type User struct {
ID int
}
// CreateUser Creates a user
func (u *User) CreateUser() (int64, error) {
// For the sake of brevity just make sure you can
// "connect" to the database
if err := app.GetDB().Ping(); err != nil {
panic(err)
}
return 1234, nil
}
main.go
package main
import (
"fmt"
"net/http"
_ "github.com/mattn/go-sqlite3"
"github.com/proj/org/app"
"github.com/proj/org/models"
)
func homeHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "You are home")
}
func subscribeHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Subscribing...")
u := models.User{}
u.CreateUser()
}
func main() {
fmt.Println("Running")
app.Setup()
http.HandleFunc("/", homeHandler)
http.HandleFunc("/subscribe", subscribeHandler)
err := http.ListenAndServe(":9090", nil)
if err != nil {
panic(err)
}
}
I thought about doing a app.Shutdown() but that wouldn't work for my most normal use case which is CTRL-C. It would seem if I don't close the database the DB connections would just grow... Just trying to understand.
It's not necessary to close the DB.
The returned DB is safe for concurrent use by multiple goroutines and
maintains its own pool of idle connections. Thus, the Open function
should be called just once. It is rarely necessary to close a DB.
From: https://golang.org/pkg/database/sql/#Open
When your program exits then any open connection is closed, it does not stay open somewhere in the ether awaiting your program to start again, so do not worry about the connections "growing" when you CTRL-C your app.
If, however, you still want to close it, then you can just export a CloseDB function the same way you do with GetDB.
App.go
// ...
func CloseDB() error {
return db.Close()
}
main.go
// ...
func main() {
// ...
app.Setup()
defer app.CloseDB()
// ...
}
Related
Trying To Learn Go
My Test Project
I am building a simple API to communicate with a light SQL database.
I created this function, that gets all hosts from the DB table.
As I understand, my function should take a pointer to the DB and return me http.HandlerFunc.
func (h HostController) GetAllHosts(db *sql.DB) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
var host entities.Host
var hosts []entities.Host
var errorMessage entities.Error
rows, err := db.Query("SELECT * FROM hosts WHERE hosts = ?", host)
if err != nil {
errorMessage.Message = "Error: Get all hosts"
utils.SendError(w, http.StatusInternalServerError, errorMessage)
return
}
defer rows.Close()
for rows.Next() {
err := rows.Scan(&host.ID, &host.Uuid, &host.Name, &host.IPAddress)
if err != nil {
errorMessage.Message = "Error: (Get all hosts) Can't scan rows"
utils.SendError(w, http.StatusInternalServerError, errorMessage)
return
}
hosts = append(hosts, host)
}
////If hosts slice is empty -> no data in database
if len(hosts) == 0 {
errorMessage.Message = "No found hosts in database"
utils.SendError(w, http.StatusInternalServerError, errorMessage)
return
}
//Convert containers content to JSON representation
w.Header().Set("Content-Type", "application/json")
utils.SendSuccess(w, hosts)
}
}
Now, Until here everything should be good, but I don't know how to implement this on main.go
build layout:
.
├── controllers
│ └── hostcontroller.go
│
└── main.go
and this is how I am trying to implement it on the main.go
package main
import (
"examProg/config"
"examProg/controllers"
"fmt"
"log"
"net/http"
"github.com/gorilla/mux"
_ "github.com/mattn/go-sqlite3"
)
func main() {
db := config.GetDB() //Establishing connection to DBase
{
router := mux.NewRouter()
router.Handle("/host", controllers.HostController.GetAllHosts(db))
fmt.Println("Server is running at port: 2030")
log.Println("Server Running OK | Port: 2030")
log.Fatal(http.ListenAndServe(":2030", router))
}
}
I don't understand how to implement the http.HandlerFunc 🙃
The expression controllers.HostController.GetAllHosts is a method expression. The result of a method expression is a function that, in addition to the method's argument's, takes an instance of the receiver's type as its first argument.
Hence the "not enough arguments in call ..." error.
To illustrate:
f := controllers.HostController.GetAllHosts
fmt.Println(reflect.TypeOf(f))
// output: func(controllers.HostController, *sql.DB)
So, to make the code compile, you need to pass an instance of controllers.HostController to the call, i.e.
controllers.HostController.GetAllHosts(controllers.HostController{}, db)
As you can see, it ain't pretty. Method expressions have their use, I'm sure, but I haven't encountered them much. What I see most often instead, is the use of method values.
h := controllers.HostController{}
router.Handle("/host", h.GetAllHosts(db))
I have just started with Go with a background in NodeJS. I wrote the following program that opens the connection with the database, queries the database table and prints the record.
package main
import (
"fmt"
"github.com/gin-gonic/gin"
_ "github.com/go-sql-driver/mysql"
"github.com/jinzhu/gorm"
"io"
"net/http"
)
type User struct {
gorm.Model
Username string
Password string
}
func main() {
db, err := gorm.Open("mysql","root:1234#(localhost)/database_test")
db.LogMode(true)
defer db.Close()
if err != nil {
panic("failed to connect to the database")
}
var users []User
db.Table("user").Find(&users)
fmt.Println("after finding the user")
fmt.Println("user ", users)
}
The code works fine. But I want to understand this, doesn't this program block the main thread? Should I be doing any async operation like the database call made inside main in a separate goroutine?
I have been using Promises with NodeJS so it feels a little different with Go.
EDIT: After adding goroutine in the main method
The following code invokes a separate goroutine to query the database. But does not it still block the main thread in that it is waiting for the response from the channel? If yes, is there a better to do this?
package main
import (
"fmt"
"github.com/gin-gonic/gin"
_ "github.com/go-sql-driver/mysql"
"github.com/jinzhu/gorm"
"io"
"net/http"
)
type User struct {
gorm.Model
Username string
Password string
}
func dbRead(db *gorm.DB) <-chan []User {
c := make(chan []User)
go func() {
fmt.Println("starting the go routine to find the users")
defer close(c)
var users []User
db.Table("user").Find(&users)
c <- users
}()
return c
}
func main() {
db, err := gorm.Open("mysql","root:1234#(localhost)/database_test")
db.LogMode(true)
defer db.Close()
if err != nil {
panic("failed to connect to the database")
}
userList := <-dbRead(db)
fmt.Println("user ", userList)
}
I'm beginner in go and now want to create a big project
my question is where should create connection and where close connection in http requests
now i declare db in main function and use in all requests and don't close connection:
package main
import (
"fmt"
"github.com/jinzhu/gorm"
"github.com/kataras/iris"
"github.com/kataras/iris/context"
"github.com/kataras/iris/middleware/recover"
_ "github.com/go-sql-driver/mysql"
)
var db *gorm.DB
func main() {
port := "8080"
app := iris.New()
app.Configure()
app.Logger().SetLevel("debug")
app.Use(recover.New())
db1, err := gorm.Open("mysql", "root:#/database?charset=utf8&parseTime=True&loc=Local")
//db1, err := gorm.Open("mysql", "payro:AEkCpNhd#/payro_db?charset=utf8&parseTime=True&loc=Local")
if err != nil {
fmt.Println(err.Error())
panic(err)
}
db=db1
app.Get("/", func(i context.Context) {
db.Exec("update tbl1 set col1=''")
i.Next()
})
app.Get("/test", func(i context.Context) {
db.Exec("update tbl2 set col2=''")
i.Next()
})
_ = app.Run(iris.Addr(":"+port), iris.WithConfiguration(iris.Configuration{
//DisableBodyConsumptionOnUnmarshal:true,
}), iris.WithoutServerError(iris.ErrServerClosed))
}
this code is ok?
Your code is perhaps a test/POC code.
In a production project, you could go with a MVC or any other kind of architecture as per your needs.
It would be hard to pinpoint the exact structure of your project.
But at the very least, you would want to create a db package which declares an interface for all DB related interactions.
e.g
type UserDBRepo interface{
AddUser(context.Context, *User)
GetUser(context.Context, uint64)
}
type userDBRepo struct{ //implements UserDBRepo
*sql.DB // or whatever type gorm.Open returns
}
func NewUserDBRepo(db *sql.DB) DBRepo{
return &dbRepo{DB: db}
}
The above basically represents a single RDBMS table for this example.
There could be n such files for n DB tables.
Now call NewUserDBRepo from the main.go and pass this instance to all services that need this DB.
I'm new with golang. I'm trying to share mysql database connection in my package, latter maybe in several packages. To skip defining database connection in every package I've created Database package and now I'm trying to get that package, connect to db and use that object in whole package.
I'm using this mysql plugin: github.com/go-sql-driver/mysql
here is my code:
main.go
package main
import (
"log"
"./packages/db" // this is my custom database package
"database/sql"
_ "github.com/go-sql-driver/mysql"
)
var dbType Database.DatabaseType
var db *sql.DB
func main() {
log.Printf("-- entering main...")
dbType := Database.New()
db = dbType.GetDb()
dbType.DbConnect()
delete_test_data()
dbType.DbClose()
}
func delete_test_data(){
log.Printf("-- entering delete_test_data...")
//db.Exec("DELETE FROM test;")
}
packages/db/db.go
package Database
import (
"log"
"database/sql"
_ "github.com/go-sql-driver/mysql"
)
type DatabaseType struct {
DatabaseObject *sql.DB
}
func New()(d *DatabaseType) {
d = new(DatabaseType)
//db.DatabaseObject = db.DbConnect()
return d
}
func (d *DatabaseType) DbConnect() *DatabaseType{
log.Printf("-- entering DbConnect...")
var err error
if d.DatabaseObject == nil {
log.Printf("--------- > Database IS NIL...")
d.DatabaseObject, err = sql.Open("mysql", "...")
if err != nil {
panic(err.Error())
}
err = d.DatabaseObject.Ping()
if err != nil {
panic(err.Error())
}
}
return d
}
func (d *DatabaseType) DbClose(){
log.Printf("-- entering DbClose...")
defer d.DatabaseObject.Close()
}
func (d *DatabaseType) GetDb() *sql.DB{
return d.DatabaseObject
}
Everything is ok and without error until I uncomment this line:
db.Exec("DELETE FROM test;")
Can someone tell me what is correct way to share db connection?
Your dbType.DbConnect() method returns a DatabaseType with an initialized connection, but you're ignoring the return value entirely.
Further - to simplify your code - look at having New(host string) *DB instead of three different functions (New/DbConnect/GetDb) that do the same thing.
e.g.
package datastore
type DB struct {
// Directly embed this
*sql.DB
}
func NewDB(host string) (*DB, error) {
db, err := sql.Open(...)
if err != nil {
return nil, err
}
return &DB{db}, nil
}
package main
var db *datastore.DB
func main() {
var err error
db, err = datastore.NewDB(host)
if err != nil {
log.Fatal(err)
}
err := someFunc()
}
func someFunc() error {
rows, err := db.Exec("DELETE FROM ...")
// Handle the error, parse the result, etc.
}
This reduces the juggling you have to do, and you can still call close on your DB type because it embeds *sql.DB - there's no need to implement your own Close() method.
let me start by saying these are my first couple days of toying around in Go.
I'm trying to use the Revel framework with Gorm like this:
app/controllers/gorm.go
package controllers
import (
"fmt"
"go-testapp/app/models"
_ "github.com/go-sql-driver/mysql"
"github.com/jinzhu/gorm"
"github.com/revel/revel"
)
var DB gorm.DB
func InitDB() {
var err error
DB, err = gorm.Open("mysql", "root:#/go-testapp?charset=utf8&parseTime=True")
if err != nil {
panic(err)
}
DB.LogMode(true)
DB.AutoMigrate(models.User{})
}
type GormController struct {
*revel.Controller
DB *gorm.DB
}
app/controller/app.go
package controllers
import (
"fmt"
"go-bingo/app/models"
_ "github.com/go-sql-driver/mysql"
"github.com/revel/revel"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhu", Age: 18}
fmt.Println(c.DB)
c.DB.NewRecord(user)
c.DB.Create(&user)
return c.RenderJson(user)
}
After running it results in:
runtime error: invalid memory address or nil pointer dereference on line 19 c.DB.NewRecord(user)
It successfully creates the datatables with automigrate, but I have no idea how I should use Gorm in my controller.
Any hints in the right direction?
Important note
it's just replacement for GORP of original example of the Revel. And it comes with some pitfalls of the origin. This answer can be used as drop-in replacement for the original one. But it doesn't solve the pitfalls.
Please, take a look comments of this unswer and #MaxGabriel's answer that solves the pitfalls.
I'd recommend to use #MaxGabriel's solution to protect you application against some kinds of slow-* DDoS attacks. And to reduce (in some cases) DB pressure.
Original answer
#rauyran rights, you have to invoke InitDB inside init function (into controllers package).
Full example here (too much):
Tree
/app
/controllers
app.go
gorm.go
init.go
/models
user.go
[...]
user.go
// models/user.go
package models
import "time" // if you need/want
type User struct { // example user fields
Id int64
Name string
EncryptedPassword []byte
Password string `sql:"-"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt time.Time // for soft delete
}
gorm.go
//controllers/gorm.go
package controllers
import (
"github.com/jinzhu/gorm"
_ "github.com/lib/pq" // my example for postgres
// short name for revel
r "github.com/revel/revel"
// YOUR APP NAME
"yourappname/app/models"
"database/sql"
)
// type: revel controller with `*gorm.DB`
// c.Txn will keep `Gdb *gorm.DB`
type GormController struct {
*r.Controller
Txn *gorm.DB
}
// it can be used for jobs
var Gdb *gorm.DB
// init db
func InitDB() {
var err error
// open db
Gdb, err = gorm.Open("postgres", "user=uname dbname=udbname sslmode=disable password=supersecret")
if err != nil {
r.ERROR.Println("FATAL", err)
panic( err )
}
Gdb.AutoMigrate(&models.User{})
// unique index if need
//Gdb.Model(&models.User{}).AddUniqueIndex("idx_user_name", "name")
}
// transactions
// This method fills the c.Txn before each transaction
func (c *GormController) Begin() r.Result {
txn := Gdb.Begin()
if txn.Error != nil {
panic(txn.Error)
}
c.Txn = txn
return nil
}
// This method clears the c.Txn after each transaction
func (c *GormController) Commit() r.Result {
if c.Txn == nil {
return nil
}
c.Txn.Commit()
if err := c.Txn.Error; err != nil && err != sql.ErrTxDone {
panic(err)
}
c.Txn = nil
return nil
}
// This method clears the c.Txn after each transaction, too
func (c *GormController) Rollback() r.Result {
if c.Txn == nil {
return nil
}
c.Txn.Rollback()
if err := c.Txn.Error; err != nil && err != sql.ErrTxDone {
panic(err)
}
c.Txn = nil
return nil
}
app.go
package controllers
import(
"github.com/revel/revel"
"yourappname/app/models"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhup"}
c.Txn.NewRecord(user)
c.Txn.Create(&user)
return c.RenderJSON(user)
}
init.go
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB) // invoke InitDB function before
revel.InterceptMethod((*GormController).Begin, revel.BEFORE)
revel.InterceptMethod((*GormController).Commit, revel.AFTER)
revel.InterceptMethod((*GormController).Rollback, revel.FINALLY)
}
As you can see, it's like Revel's Booking modified for GORM.
Works fine for me. Result:
{
"Id": 5,
"Name": "Jinzhup",
"EncryptedPassword": null,
"Password": "",
"CreatedAt": "2014-09-22T17:55:14.828661062+04:00",
"UpdatedAt": "2014-09-22T17:55:14.828661062+04:00",
"DeletedAt": "0001-01-01T00:00:00Z"
}
This answer is derived from #IvanBlack's answer, at his suggestion. His version is a direct translation of the Revel example code, but we identified some problems with the original code that this answer fixes. The main changes it makes are:
The entire HTTP request is no longer wrapped by a database transaction. Wrapping the entire HTTP request keeps open the transaction far longer than necessary, which could cause a number of problems:
Your transactions will hold locks on the database, and many transactions holding locks could lead to deadlocks
More database resources are used
These problems are magnified if your HTTP requests aren't mostly limited by your SQL database access. E.g. if your HTTP request blocks for 30 seconds on making an external HTTP request or accessing another database, the slowdown on those services could affect your SQL database.
Because the transaction is no longer automatically being checked for errors at the end of the HTTP request, errors are checked as soon as the database insert is made. This is more correct as well: Imagine that after inserting the User struct into a database, you then stored the User.Id in another database like Redis. This would be fine if the database insert worked, but if it failed and you didn't immediately check the error, you would insert the default int64 value of 0 into Redis (before later rolling back only the SQL transaction).
Tree
/app
/controllers
app.go
gorm.go
init.go
/models
user.go
[...]
user.go
// models/user.go
package models
import "time" // if you need/want
type User struct { // example user fields
Id int64
Name string
EncryptedPassword []byte
Password string `sql:"-"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt time.Time // for soft delete
}
gorm.go
//controllers/gorm.go
package controllers
import (
"github.com/jinzhu/gorm"
_ "github.com/lib/pq" // my example for postgres
// short name for revel
r "github.com/revel/revel"
)
// type: revel controller with `*gorm.DB`
type GormController struct {
*r.Controller
DB *gorm.DB
}
// it can be used for jobs
var Gdb *gorm.DB
// init db
func InitDB() {
var err error
// open db
Gdb, err = gorm.Open("postgres", "user=USERNAME dbname=DBNAME sslmode=disable")
Gdb.LogMode(true) // Print SQL statements
if err != nil {
r.ERROR.Println("FATAL", err)
panic(err)
}
}
func (c *GormController) SetDB() r.Result {
c.DB = Gdb
return nil
}
init.go
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB) // invoke InitDB function before
revel.InterceptMethod((*GormController).SetDB, revel.BEFORE)
}
app.go
package controllers
import(
"github.com/revel/revel"
"yourappname/app/models"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhup"} // Note: In practice you should initialize all struct fields
if err := c.DB.Create(&user).Error; err != nil {
panic(err)
}
return c.RenderJSON(user)
}
Result:
{
"Id": 5,
"Name": "Jinzhup",
"EncryptedPassword": null,
"Password": "",
"CreatedAt": "2014-09-22T17:55:14.828661062+04:00",
"UpdatedAt": "2014-09-22T17:55:14.828661062+04:00",
"DeletedAt": "0001-01-01T00:00:00Z"
}
Your error is caused because you haven't initialised your c.DB database variable, it's still nil.
In your controllers/init.go file make sure you are calling revel.OnAppStart(InitDB). It should look something like this:
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB)
// maybe some other init things
}
OR you need to pass in a pointer to AutoMigrate?
DB.AutoMigrate(&models.User{})