let me start by saying these are my first couple days of toying around in Go.
I'm trying to use the Revel framework with Gorm like this:
app/controllers/gorm.go
package controllers
import (
"fmt"
"go-testapp/app/models"
_ "github.com/go-sql-driver/mysql"
"github.com/jinzhu/gorm"
"github.com/revel/revel"
)
var DB gorm.DB
func InitDB() {
var err error
DB, err = gorm.Open("mysql", "root:#/go-testapp?charset=utf8&parseTime=True")
if err != nil {
panic(err)
}
DB.LogMode(true)
DB.AutoMigrate(models.User{})
}
type GormController struct {
*revel.Controller
DB *gorm.DB
}
app/controller/app.go
package controllers
import (
"fmt"
"go-bingo/app/models"
_ "github.com/go-sql-driver/mysql"
"github.com/revel/revel"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhu", Age: 18}
fmt.Println(c.DB)
c.DB.NewRecord(user)
c.DB.Create(&user)
return c.RenderJson(user)
}
After running it results in:
runtime error: invalid memory address or nil pointer dereference on line 19 c.DB.NewRecord(user)
It successfully creates the datatables with automigrate, but I have no idea how I should use Gorm in my controller.
Any hints in the right direction?
Important note
it's just replacement for GORP of original example of the Revel. And it comes with some pitfalls of the origin. This answer can be used as drop-in replacement for the original one. But it doesn't solve the pitfalls.
Please, take a look comments of this unswer and #MaxGabriel's answer that solves the pitfalls.
I'd recommend to use #MaxGabriel's solution to protect you application against some kinds of slow-* DDoS attacks. And to reduce (in some cases) DB pressure.
Original answer
#rauyran rights, you have to invoke InitDB inside init function (into controllers package).
Full example here (too much):
Tree
/app
/controllers
app.go
gorm.go
init.go
/models
user.go
[...]
user.go
// models/user.go
package models
import "time" // if you need/want
type User struct { // example user fields
Id int64
Name string
EncryptedPassword []byte
Password string `sql:"-"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt time.Time // for soft delete
}
gorm.go
//controllers/gorm.go
package controllers
import (
"github.com/jinzhu/gorm"
_ "github.com/lib/pq" // my example for postgres
// short name for revel
r "github.com/revel/revel"
// YOUR APP NAME
"yourappname/app/models"
"database/sql"
)
// type: revel controller with `*gorm.DB`
// c.Txn will keep `Gdb *gorm.DB`
type GormController struct {
*r.Controller
Txn *gorm.DB
}
// it can be used for jobs
var Gdb *gorm.DB
// init db
func InitDB() {
var err error
// open db
Gdb, err = gorm.Open("postgres", "user=uname dbname=udbname sslmode=disable password=supersecret")
if err != nil {
r.ERROR.Println("FATAL", err)
panic( err )
}
Gdb.AutoMigrate(&models.User{})
// unique index if need
//Gdb.Model(&models.User{}).AddUniqueIndex("idx_user_name", "name")
}
// transactions
// This method fills the c.Txn before each transaction
func (c *GormController) Begin() r.Result {
txn := Gdb.Begin()
if txn.Error != nil {
panic(txn.Error)
}
c.Txn = txn
return nil
}
// This method clears the c.Txn after each transaction
func (c *GormController) Commit() r.Result {
if c.Txn == nil {
return nil
}
c.Txn.Commit()
if err := c.Txn.Error; err != nil && err != sql.ErrTxDone {
panic(err)
}
c.Txn = nil
return nil
}
// This method clears the c.Txn after each transaction, too
func (c *GormController) Rollback() r.Result {
if c.Txn == nil {
return nil
}
c.Txn.Rollback()
if err := c.Txn.Error; err != nil && err != sql.ErrTxDone {
panic(err)
}
c.Txn = nil
return nil
}
app.go
package controllers
import(
"github.com/revel/revel"
"yourappname/app/models"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhup"}
c.Txn.NewRecord(user)
c.Txn.Create(&user)
return c.RenderJSON(user)
}
init.go
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB) // invoke InitDB function before
revel.InterceptMethod((*GormController).Begin, revel.BEFORE)
revel.InterceptMethod((*GormController).Commit, revel.AFTER)
revel.InterceptMethod((*GormController).Rollback, revel.FINALLY)
}
As you can see, it's like Revel's Booking modified for GORM.
Works fine for me. Result:
{
"Id": 5,
"Name": "Jinzhup",
"EncryptedPassword": null,
"Password": "",
"CreatedAt": "2014-09-22T17:55:14.828661062+04:00",
"UpdatedAt": "2014-09-22T17:55:14.828661062+04:00",
"DeletedAt": "0001-01-01T00:00:00Z"
}
This answer is derived from #IvanBlack's answer, at his suggestion. His version is a direct translation of the Revel example code, but we identified some problems with the original code that this answer fixes. The main changes it makes are:
The entire HTTP request is no longer wrapped by a database transaction. Wrapping the entire HTTP request keeps open the transaction far longer than necessary, which could cause a number of problems:
Your transactions will hold locks on the database, and many transactions holding locks could lead to deadlocks
More database resources are used
These problems are magnified if your HTTP requests aren't mostly limited by your SQL database access. E.g. if your HTTP request blocks for 30 seconds on making an external HTTP request or accessing another database, the slowdown on those services could affect your SQL database.
Because the transaction is no longer automatically being checked for errors at the end of the HTTP request, errors are checked as soon as the database insert is made. This is more correct as well: Imagine that after inserting the User struct into a database, you then stored the User.Id in another database like Redis. This would be fine if the database insert worked, but if it failed and you didn't immediately check the error, you would insert the default int64 value of 0 into Redis (before later rolling back only the SQL transaction).
Tree
/app
/controllers
app.go
gorm.go
init.go
/models
user.go
[...]
user.go
// models/user.go
package models
import "time" // if you need/want
type User struct { // example user fields
Id int64
Name string
EncryptedPassword []byte
Password string `sql:"-"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt time.Time // for soft delete
}
gorm.go
//controllers/gorm.go
package controllers
import (
"github.com/jinzhu/gorm"
_ "github.com/lib/pq" // my example for postgres
// short name for revel
r "github.com/revel/revel"
)
// type: revel controller with `*gorm.DB`
type GormController struct {
*r.Controller
DB *gorm.DB
}
// it can be used for jobs
var Gdb *gorm.DB
// init db
func InitDB() {
var err error
// open db
Gdb, err = gorm.Open("postgres", "user=USERNAME dbname=DBNAME sslmode=disable")
Gdb.LogMode(true) // Print SQL statements
if err != nil {
r.ERROR.Println("FATAL", err)
panic(err)
}
}
func (c *GormController) SetDB() r.Result {
c.DB = Gdb
return nil
}
init.go
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB) // invoke InitDB function before
revel.InterceptMethod((*GormController).SetDB, revel.BEFORE)
}
app.go
package controllers
import(
"github.com/revel/revel"
"yourappname/app/models"
)
type App struct {
GormController
}
func (c App) Index() revel.Result {
user := models.User{Name: "Jinzhup"} // Note: In practice you should initialize all struct fields
if err := c.DB.Create(&user).Error; err != nil {
panic(err)
}
return c.RenderJSON(user)
}
Result:
{
"Id": 5,
"Name": "Jinzhup",
"EncryptedPassword": null,
"Password": "",
"CreatedAt": "2014-09-22T17:55:14.828661062+04:00",
"UpdatedAt": "2014-09-22T17:55:14.828661062+04:00",
"DeletedAt": "0001-01-01T00:00:00Z"
}
Your error is caused because you haven't initialised your c.DB database variable, it's still nil.
In your controllers/init.go file make sure you are calling revel.OnAppStart(InitDB). It should look something like this:
package controllers
import "github.com/revel/revel"
func init() {
revel.OnAppStart(InitDB)
// maybe some other init things
}
OR you need to pass in a pointer to AutoMigrate?
DB.AutoMigrate(&models.User{})
Related
What could be a unit test case for the s3/sqs event when unmarshalling json to struck in Go?
One option that comes to mind is if the "Body" is empty, but not sure what else could it be.
Code example:
type Event struct {
Message string `json:"Message"`
}
func unmarshal(message events.SQSMessage) (s3Event, error) {
var e Event
err := json.Unmarshal([]byte(message.Body), &e)
if err != nil {
return s3Event, err
}
var s3Event events.S3Event
err = json.Unmarshal([]byte(e.Message), &sqsEvent)
if err != nil {
return s3Event, err
}
return s3Event, nil
}
Would appreciate any advice.
Premise: I'm not sure if I've fully understood your question. Furthermore, the code has some issues that cause me hard-time to figure out what's going on. With that being said, I'll try to help you by providing a piece of working software (it's not very useful) together with a piece of software written just to test it. Maybe, this can help you in figuring out your issue.
unmarshal.go file
package unmarshaltest
import (
"encoding/json"
"fmt"
"github.com/aws/aws-lambda-go/events"
)
type Event struct {
Message string `json:"body"`
}
func Unmarshal(message events.SQSMessage) (string, error) {
rawBytes, _ := json.Marshal(message)
var e Event
json.Unmarshal(rawBytes, &e)
if e.Message == "" {
return "", fmt.Errorf("err while unmarshaling")
}
return e.Message, nil
}
One thing I fixed and you should pay attention to is how you name your functions. If you call your function unmarshal with lowercase u it means that this function should not be exported from this package (it's private). So I changed the name with the U as I supposed you wanna test this function.
unmarshal_test.go file
package unmarshaltest
import (
"encoding/json"
"testing"
"github.com/aws/aws-lambda-go/events"
"github.com/stretchr/testify/assert"
)
func TestUnmarshal(t *testing.T) {
t.Run("Unmarshal_NotEmpty_Message", func(t *testing.T) {
var sqsMsg events.SQSMessage
sqsEventRaw := `{ "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78", "receiptHandle": "MessageReceiptHandle", "body": "My own event payload!", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1523232000000", "SenderId": "123456789012", "ApproximateFirstReceiveTimestamp": "1523232000001" }, "messageAttributes": {}, "md5OfBody": "4d1d0024b51659ad8c3725f9ba7e2471", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue", "awsRegion": "us-east-1" }`
json.Unmarshal([]byte(sqsEventRaw), &sqsMsg)
res, err := Unmarshal(sqsMsg)
assert.Equal(t, "My own event payload!", res)
assert.Nil(t, err)
})
t.Run("Unmarshal_Empty_Message", func(t *testing.T) {
res, err := Unmarshal(events.SQSMessage{})
assert.Equal(t, "", res)
assert.NotNil(t, err)
})
}
Here you can find a simple piece of code that is used to test the two possible execution flows that our code might take.
You should also pay attention to how you write the code. It's very hard to swap the invocation to the json.Unmarshal function with a mock as it's hard-coded into the code. If your wish is to be able to mock out this function, you should rely on interfaces in your production code and replace them with mocks during tests.
Let me know if this clarifies something and if you need something else!
So far the hardest part of Go has been understanding how to organize code. It seems incredibly simple on it's face but every time I've tried to do anything I run into circular imports or things like "exported func Start returns unexported type models.dbStore, which can be annoying to use".
Using the following code how do I call db.Close() or am I really not understanding how I'm supposed to provide the database to my models. Here's what I've got:
App.go
package app
import (
"database/sql"
// Comment
_ "github.com/mattn/go-sqlite3"
)
var (
// DB The database connection
db *sql.DB
)
// Setup Sets up the many many app settings
func Setup() {
d, err := sql.Open("sqlite3", "./foo.db")
if err != nil {
panic(err)
}
// TODO: How does the DB get closed?
// defer db.Close()
db = d
}
// GetDB Returns a reference to the database
func GetDB() *sql.DB {
return db
}
Users.go
package models
import (
"github.com/proj/org/app"
)
// User struct
type User struct {
ID int
}
// CreateUser Creates a user
func (u *User) CreateUser() (int64, error) {
// For the sake of brevity just make sure you can
// "connect" to the database
if err := app.GetDB().Ping(); err != nil {
panic(err)
}
return 1234, nil
}
main.go
package main
import (
"fmt"
"net/http"
_ "github.com/mattn/go-sqlite3"
"github.com/proj/org/app"
"github.com/proj/org/models"
)
func homeHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "You are home")
}
func subscribeHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Subscribing...")
u := models.User{}
u.CreateUser()
}
func main() {
fmt.Println("Running")
app.Setup()
http.HandleFunc("/", homeHandler)
http.HandleFunc("/subscribe", subscribeHandler)
err := http.ListenAndServe(":9090", nil)
if err != nil {
panic(err)
}
}
I thought about doing a app.Shutdown() but that wouldn't work for my most normal use case which is CTRL-C. It would seem if I don't close the database the DB connections would just grow... Just trying to understand.
It's not necessary to close the DB.
The returned DB is safe for concurrent use by multiple goroutines and
maintains its own pool of idle connections. Thus, the Open function
should be called just once. It is rarely necessary to close a DB.
From: https://golang.org/pkg/database/sql/#Open
When your program exits then any open connection is closed, it does not stay open somewhere in the ether awaiting your program to start again, so do not worry about the connections "growing" when you CTRL-C your app.
If, however, you still want to close it, then you can just export a CloseDB function the same way you do with GetDB.
App.go
// ...
func CloseDB() error {
return db.Close()
}
main.go
// ...
func main() {
// ...
app.Setup()
defer app.CloseDB()
// ...
}
I would like to understand how does GIN ensures that each HTTP request gets a unique DB ( say MySQL ) connection. Here is one example code.
If you see, since 'db' is a global object and therefore, the API router.GET("/person/:age"... gets access to DB.
Now with load, I suppose GIN will have concurrency implemented internally. If yes, then how does it ensures that each request gets a different connection. If no, then it is single threaded imnplementation. Could anyone please correct my understanding.
package main
import (
// "bytes"
"database/sql"
"fmt"
"github.com/gin-gonic/gin"
_ "github.com/go-sql-driver/mysql"
"net/http"
)
func checkErr(err error) {
if err != nil {
panic(err)
} else {
fmt.Println("successful...")
}
}
func main() {
db, err := sql.Open("mysql", "abfl:abfl#tcp(127.0.0.1:3306)/abfl?charset=utf8")
checkErr(err)
defer db.Close()
// make sure connection is available
err = db.Ping()
checkErr(err)
type User struct {
age int
name string
}
router := gin.Default()
// Add API handlers here
// GET a user detail
router.GET("/person/:age", func(c *gin.Context) {
var (
user User
result gin.H
)
age := c.Param("age")
fmt.Println("input age : '%d'", age)
row := db.QueryRow("select age, name from user where age = ?", age)
err = row.Scan(&user.age, &user.name)
fmt.Printf("user : %+v\n", user)
if err != nil {
// If no results send null
result = gin.H{
"user": nil,
"count": 0,
}
} else {
result = gin.H{
"age": user.age,
"name": user.name,
"count": 1,
}
}
c.JSON(http.StatusOK, result)
})
router.Run(":3000")
}
Establishing a new SQL connection for each HTTP request is too heavy and has no sense.
In go there is no user-managable connection pool yet, it is handled internally by go implementation.
sql.DB is ready to be used concurrently, so there is no worry about it.
And GIN has nothing to do with SQL connections at all. It is fully your responsibility to handle queries/transactions properly.
Let's say I have the following Go struct on the server
type account struct {
Name string
Balance int
}
I want to call json.Decode on the incoming request to parse it into an account.
var ac account
err := json.NewDecoder(r.Body).Decode(&ac)
If the client sends the following request:
{
"name": "test#example.com",
"balance": "3"
}
Decode() will return the following error:
json: cannot unmarshal string into Go value of type int
Now it's possible to parse that back into "you sent a string for Balance, but you really should have sent an integer", but it's tricky, because you don't know the field name. It also gets a lot trickier if you have a lot of fields in the request - you don't know which one failed to parse.
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
You can use a custom type with custom unmarshaling algorythm for your "Balance" field.
Now there are two possibilities:
Handle both types:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Int int
type account struct {
Name string
Balance Int
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var s string
err = json.Unmarshal(b, &s)
if err == nil {
var n int
n, err = strconv.Atoi(s)
if err != nil {
return
}
*i = Int(n)
return
}
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
Handle only a numeric type, and fail anything else with a sensible error:
package main
import (
"encoding/json"
"fmt"
)
type Int int
type account struct {
Name string
Balance Int
}
type FormatError struct {
Want string
Got string
Offset int64
}
func (fe *FormatError) Error() string {
return fmt.Sprintf("Invalid data format at %d: want: %s, got: %s",
fe.Offset, fe.Want, fe.Got)
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
return
}
if ute, ok := err.(*json.UnmarshalTypeError); ok {
err = &FormatError{
Want: "number",
Got: ute.Value,
Offset: ute.Offset,
}
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %#v\n", err)
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
There is a third possibility: write custom unmarshaler for the whole account type, but it requires more involved code because you'd need to actually iterate over the input JSON data using the methods of the
encoding/json.Decoder type.
After reading your
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
more carefully, I admit having a custom parser for the whole type is the only sensible possibility unless you are OK with a 3rd-party package implementing a parser supporting validation via JSON schema (I think I'd look at this first as juju is a quite established product).
A solution for this could be to use a type assertion by using a map to unmarshal the JSON data into:
type account struct {
Name string
Balance int
}
var str = `{
"name": "test#example.com",
"balance": "3"
}`
func main() {
var testing = map[string]interface{}{}
err := json.Unmarshal([]byte(str), &testing)
if err != nil {
fmt.Println(err)
}
val, ok := testing["balance"]
if !ok {
fmt.Println("missing field balance")
return
}
nv, ok := val.(float64)
if !ok {
fmt.Println("balance should be a number")
return
}
fmt.Printf("%+v\n", nv)
}
See http://play.golang.org/p/iV7Qa1RrQZ
The type assertion here is done using float64 because it is the default number type supported by Go's JSON decoder.
It should be noted that this use of interface{} is probably not worth the trouble.
The UnmarshalTypeError (https://golang.org/pkg/encoding/json/#UnmarshalFieldError) contains an Offset field that could allow retrieving the contents of the JSON data that triggered the error.
You could for example return a message of the sort:
cannot unmarshal string into Go value of type int near `"balance": "3"`
It would seem that here provides an implementation to work around this issue in Go only.
type account struct {
Name string
Balance int `json:",string"`
}
In my estimation the more correct and sustainable approach is for you to create a client library in something like JavaScript and publish it into the NPM registry for others to use (private repository would work the same way). By providing this library you can tailor the API for the consumers in a meaningful way and prevent errors creeping into your main program.
and need your help.
Wanted to build simple api and stuck with some problem.
I've choose gin and database/sql with postgres driver
package main
import (
"database/sql"
"fmt"
"github.com/gin-gonic/gin"
_ "github.com/lib/pq"
)
func main() {
router := gin.Default()
router.GET("/search/:text", SearchWord)
router.Run(":8080")
}
I need to make query to DB and make json out of this request.
func checkErr(err error) {
if err != nil {
panic(err)
}
}
type Message struct {
ticket_id int `json:"ticket_id"`
event string `json:"event"`
}
func SearchWord(c *gin.Context) {
word := c.Params.ByName("text")
db, err := sql.Open("postgres", "host=8.8.8.8 user= password= dbname=sample")
defer db.Close()
checkErr(err)
rows, err2 := db.Query("SELECT ticket_id,event FROM ....$1, word)
checkErr(err)
for rows.Next() {
var ticket_id int
var event string
err = rows.Scan(&ticket_id, &event)
checkErr(err)
fmt.Printf("%d | %s \n\n", ticket_id, event)
}
}
This coda working nice, but when i need to make json.
I need to make struct of a row
type Message struct {
ticket_id int `json:"ticket_id"`
event string `json:"event"`
}
an then i need to create slice , and append every rows.Next() loop an than answer to browser with Json...
c.JSON(200, messages)
But how to do that...don't know :(
disclaimer: I am brand new to go
Since you Scanned your column data into your variables, you should be able to initialize a structure with their values:
m := &Message{ticket_id: ticket_id, event: event}
You could initialize a slice with
s := make([]*Message, 0)
And then append each of your message structs after instantiation:
s = append(s, m)
Because I'm not too familiar with go there are a couple things i'm not sure about:
after copying data from query to your vars using rows.Scan does initializing the Message struct copy the current iterations values as expected??
If there is a way to get the total number of rows from your query it might be slighlty more performant to initialize a static length array, instead of a slice?
I think #inf deleted answer about marshalling your Message to json down the line might need to be addressed, and Message field's might need to be capitalized
copied from #inf:
The names of the members of your struct need be capitalized so that
they get exported and can be accessed.
type Message struct {
Ticket_id int `json:"ticket_id"`
Event string `json:"event"` }
I'm going to cheat a little here and fix a few things along the way:
First: open your database connection pool once at program start-up (and not on every request).
Second: we'll use sqlx to make it easier to marshal our database rows into our struct.
package main
var db *sqlx.DB
func main() {
var err error
// sqlx.Connect also checks that the connection works.
// sql.Open only "establishes" a pool, but doesn't ping the DB.
db, err = sqlx.Connect("postgres", "postgres:///...")
if err != nil {
log.Fatal(err)
}
router := gin.Default()
router.GET("/search/:text", SearchWord)
router.Run(":8080")
}
// in_another_file.go
type Message struct {
TicketID int `json:"ticket_id" db:"ticket_id"`
Event string `json:"event" db:"event"`
}
func SearchWord(c *gin.Context) {
word := c.Params.ByName("text")
// We create a slice of structs to marshal our rows into
var messages []*Message{}
// Our DB connection pool is safe to use concurrently from here
err := db.Select(&messages, "SELECT ticket_id,event FROM ....$1, word)
if err != nil {
http.Error(c.Writer, err.Error(), 500)
return
}
// Write it out using gin-gonic's JSON writer.
c.JSON(200, messages)
}
I hope that's clear. sqlx also takes care of calling rows.Close() for you, which will otherwise leave connections hanging.