Safely perform DB migrations with Go - mysql

Let's say I have a web app that shows a list of posts. The post struct is:
type Post struct {
Id int64 `sql:",primary"`
Title string
Body string
}
It retrieves the posts with:
var posts []*Post
rows, err := db.QueryContext(ctx, "select * from posts;")
if err != nil {
return nil, oops.Wrapf(err, "could not get posts")
}
defer rows.Close()
for rows.Next() {
p := &Post{}
err := rows.Scan(
&p.Id,
&p.Title,
&p.Body,
)
if err != nil {
return nil, oops.Wrapf(err, "could not scan row")
}
posts = append(posts, p)
}
return posts, nil
All works fine. Now, I want to alter the table schema by adding a column:
ALTER TABLE posts ADD author varchar(62);
Suddenly, the requests to get posts result in:
sql: expected 4 destination arguments in Scan, not 3
which makes sense since the table now has 4 columns instead of the 3 stipulated by the retrieval logic.
I can then update the struct to be:
type Post struct {
Id int64 `sql:",primary"`
Title string
Body string
Author string
}
and the retrival logic to be:
for rows.Next() {
p := &Post{}
err := rows.Scan(
&p.Id,
&p.Title,
&p.Body,
&p.Author
)
if err != nil {
return nil, oops.Wrapf(err, "could not scan row")
}
posts = append(posts, p)
}
which solves this. However, this implies there is always a period of downtime between migration and logic update + deploy. How to avoid that downtime?
I have tried swapping the order of the above changes but this does not work, with that same request resulting in:
sql: expected 3 destination arguments in Scan, not 4
(which makes sense, since the table only has 3 columns at that point as opposed to 4);
and other requests resulting in:
Error 1054: Unknown column 'author' in 'field list'
(which makes sense, because at that point the posts table does not have an author column just yet)

You should be able to achieve your desired behavior by adapting the SQL Query to return the exact fields you want to populate.
SELECT Id , Title , Body FROM posts;
This way even if you add another column Author the query results only contain 3 values.

Related

Golang, MySQL, Can't append query data into struct list [duplicate]

This question already has answers here:
When to use single quotes, double quotes, and backticks in MySQL
(13 answers)
Closed 10 days ago.
When I'm trying to parse data into struct and then append it into a slice, get nothing. But if I use query in MySQL workbench, I get some values....
query, err := db.Query("SELECT 'description','is_done' FROM tasks WHERE 'user_id' = ?;", userId)
if err != nil {
return nil, err
}
defer query.Close()
var tasks []TodoUserDTO
var currentTask TodoUserDTO
for query.Next() {
err = query.Scan(&currentTask.Description, &currentTask.IsDone)
if err != nil {
panic(err)
}
tasks = append(tasks, currentTask)
}
TodoDTO struct looks like this:
type TodoUserDTO struct {
Description string `json:"desc"`
IsDone bool `json:"done"`
}
Based on the code, it looks like you're using the wrong column names in the SELECT statement of your query. The SELECT statement should include the actual column names of the columns in the tasks table, rather than the literal strings of the column names.
Try changing the SELECT statement to this:
"SELECT description, is_done FROM tasks WHERE user_id = ?"

go mysql LAST_INSERT_ID() returns 0

I have this MySQL database where I need to add records with a go program and need to retrieve the id of the last added record, to add the id to another table.
When i run insert INSERT INTO table1 values("test",1); SELECT LAST_INSERT_ID() in MySQL Workbench, it returns the last id, which is auto incremented, with no issues.
If I run my go code however, it always prints 0. The code:
_, err := db_client.DBClient.Query("insert into table1 values(?,?)", name, 1)
var id string
err = db_client.DBClient.QueryRow("SELECT LAST_INSERT_ID()").Scan(&id)
if err != nil {
panic(err.Error())
}
fmt.Println("id: ", id)
I tried this variation to try to narrow down the problem scope further: err = db_client.DBClient.QueryRow("SELECT id from table1 where name=\"pleasejustwork\";").Scan(&id), which works perfectly fine; go returns the actual id.
Why is it not working with the LAST_INSERT_ID()?
I'm a newbie in go so please do not go hard on me if i'm making stupid go mistakes that lead to this error :D
Thank you in advance.
The MySQL protocol returns LAST_INSERT_ID() values in its response to INSERT statements. And, the golang driver exposes that returned value. So, you don't need the extra round trip to get it. These ID values are usually unsigned 64-bit integers.
Try something like this.
res, err := db_client.DBClient.Exec("insert into table1 values(?,?)", name, 1)
if err != nil {
panic (err.Error())
}
id, err := res.LastInsertId()
if err != nil {
panic (err.Error())
}
fmt.Println("id: ", id)
I confess I'm not sure why your code didn't work. Whenever you successfully issue a single-row INSERT statement, the next statement on the same database connection always has access to a useful LAST_INSERT_ID() value. This is true whether or not you use explicit transactions.
But if your INSERT is not successful, you must treat the last insert ID value as unpredictable. (That's a technical term for "garbage", trash, rubbish, basura, etc.)

Avoid loops in - Recursive m2m relation self referencing

this is less of a question about golang or mysql, its more a general question. Hope i am still in the right place and someone could help me to wrap my head around this.
I have a struct Role which can have multiple child roles.
type Role struct{
Name string
Children []Role
}
So let's say Role A has a child Role B and Role B has a Child Role C.
In my frontend, the m2m relation is displayed as a multi-select HTML field.
To avoid an infinite loop (A-B-C-A...) I want that the user can not enter one of the related Roles.
For example, Role C should not display Role A and B, because if a user would select those, an infinite loop would happen.
The database in the backend is looking like this:
roles table (main table)
id, name, ...
role_roles (junction table)
role_id, child_id
I created this helper method to detect the ids which should not get displayed. It's checking if the Role C is somewhere in the field child_id then it takes the role_id of this entry and is doing the same again. This works, but it is looking really unprofessional and I was wondering how this could be solved in a more elegant way - and with fewer SQL queries...
// whereIDLoop returns the ids which should get excluded
func whereIDLoop(id int) ([]int, error) {
ids := []int{}
b := builder.GlobalBuilder
rows, err := b.Select("role_roles").Columns("role_id").Where("child_id = ?", id).All()
if err != nil {
return nil, err
}
for rows.Next() {
var id int
if err := rows.Scan(&id); err != nil {
return nil,err
}
ids = append(ids, id)
id2,err := whereIDLoop(id)
if err != nil {
return nil, err
}
if id2 != nil{
ids = append(ids, id2...)
}
}
err = rows.Close()
if err != nil {
return nil, err
}
return ids, nil
}
Thanks for any help.
Cheers Pat
can't say best practice, my suggestion is put the validation logic at application layer.
a helper function in JS to filter the option in multi-select like this
a validation logic in API
a validation logic at repository layer.
track the roles when apply in case there is a circle.

Dynamic SQL select query in Golang

I am trying to build API, with database/sql and mysql driver, that will read data based on URL parameters.
Something like this
myapi.com/users?columns=id,first_name,last_name,country&sort=desc&sortColumn=last_name&limit=10&offset=20
I know how to get all columns or just specific columns when it is defined in struct. But I want to know is it possible to get columns from url and instead of predefined struct save it to map and than just scan those columns.
I have working code that will get data from above endpoint only if number of columns is same as in struct. If I remove country for example I get error that Scan expects 4 params but 3 are given.
I don't need specific code, just some directions since I am learning Go and my background is PHP where this is easier to do.
Update
Thanks to answers I have partly working solution.
Here is code:
cols := []string{"id", "first_name", "last_name"}
vals := make([]interface{}, len(cols))
w := map[string]interface{}{"id": 105}
var whereVal []interface{}
var whereCol []string
for k, v := range w {
whereVal = append(whereVal, v)
whereCol = append(whereCol, fmt.Sprintf("%s = ?", k))
}
for i := range cols {
vals[i] = new(interface{})
}
err := db.QueryRow("SELECT "+strings.Join(cols, ",")+" FROM users WHERE "+strings.Join(whereCol, " AND "), whereVal...).Scan(vals...)
if err != nil {
fmt.Println(err)
}
b, _ := json.Marshal(vals)
fmt.Println(string(b))
This should query SELECT id, first_name, last_name FROM users WHERE id = 105;
But how do I get data out to proper json object? Now it prints out strings encoded in base64 like this.
[105,"Sm9obm55","QnJhdm8="]
From what I know (also not much experienced in Go) if you don't assign a real type to value then Scan will return []byte and when it is marshalled it returns base64 encoded string.
So you have to assign a type to your columns and if you want proper json then assign keys to values.
In your example it can be done something like this:
cols := []string{"id", "first_name", "last_name"}
vals := make([]interface{}, len(cols))
result := make(map[string]interface{}, len(cols))
for i, key := range cols {
switch key {
case "id", "status":
vals[i] = new(int)
default:
vals[i] = new(string)
}
result[key] = vals[i]
}
b, _ := json.Marshal(result)
fmt.Println(string(b))
So, instead of looping over cols and creating new interface for each column, now we are creating key/value pairs and assigning type based on column name.
Also, if you have nullable columns in table, and you probably have, then you'll get error because nil can't go into string. So I suggest this package gopkg.in/guregu/null.v3 and then assign type like null.String. That way you'll get back null as a value.
For example:
for i, key := range cols {
switch key {
case "id", "status":
vals[i] = new(int)
case "updated_at", "created_at":
vals[i] = new(null.Time)
default:
vals[i] = new(null.String)
}
result[key] = vals[i]
}
Here is an option that a found to return dynamic resultset, you will need a interface{} array but you have to assign to a new(interface{}) to get a pointer that can be write by Scan method
//...
types, _ := rows.ColumnTypes()
for rows.Next() {
row := make([]interface{}, len(types))
for i := range types {
row[i] = new(interface{})
}
rows.Scan(row...)
}
You must first fetch the result columns count and then don't exceed the size.
If you meant the query fields, you need dynamic create the query string, the params size must be the same.
I would create a query statement with the dynamic fields(use placeholder for avoiding sql injection):
rows := db.QueryRow("SELECT {{YOUR_FIELDS}} from table_tbl")
Create variable carrier with the same size of columns
vals := make([]interface{}, len(rows.Columns()))
Use sql.RawBytes for field's type if you don't need type checking or can't know their types, otherwise use the same type of field.
for i, _ := range cols {
vals[i] = new(sql.RawBytes)
//check column name, if it is id, and you know it is integer
//vals[i] = new(int)
}
Iterate rows and scan
for rows.Next() {
err = rows.Scan(vals...)
}

Retrieve relation one to many into JSON sql pure, Golang, Performance

Suppose that I've the following structures that it's the mapped tables.
type Publisher struct{
ID int `db:"id"`
Name string `db:"name"`
Books []*Book
}
type Book struct {
ID int `db:"id"`
Name string `db:"name"`
PublisherID `db:"publisher_id"`
}
So, What if I wanna retrieve all the Publisher with all related Books I would like to get a JSON like this:
[ //Publisher 1
{
"id" : "10001",
"name":"Publisher1",
"books" : [
{ "id":321,"name": "Book1"},
{ "id":333,"name": "Book2"}
]
},
//Publisher 2
{
"id" : "10002",
"name":"Slytherin Publisher",
"books" : [
{ "id":4021,"name": "Harry Potter and the Chamber of Secrets"},
{ "id":433,"name": "Harry Potter and the Order of the Phoenix"}
]
},
]
So I've the following structure that I use to retrieve all kind of query related with Publisher
type PublisherRepository struct{
Connection *sql.DB
}
// GetEbooks return all the books related with a publisher
func (r *PublisherRepository) GetBooks(idPublisher int) []*Book {
bs := make([]Book,0)
sql := "SELECT * FROM books b WHERE b.publisher_id =$1 "
row, err := r.Connection.Query(sql,idPublisher)
if err != nil {
//log
}
for rows.Next() {
b := &Book{}
rows.Scan(&b.ID, &b.Name, &b.PublisherID)
bs := append(bs,b)
}
return bs
}
func (r *PublisherRepository) GetAllPublishers() []*Publisher {
sql := "SELECT * FROM publishers"
ps := make([]Publisher,0)
rows, err := r.Connection.Query(sql)
if err != nil {
// log
}
for rows.Next() {
p := &Publisher{}
rows.Scan(&p.ID,&p.Name)
// Is this the best way?
books := r.GetBooks(p.ID)
p.Books = books
}
return ps
}
So , here my questions
What is the best approach to retrieve all the publisher with the best performance, because a for inside a for is not the best solution, what if I've 200 publisher and in the average of each publisher has 100 books.
Is in GoLang idiomatic PublisherRepository or is there another way to create something to manage the transactions of an entity with pure sql?
1) Bad about this would be the sql request per iteration. So here a solution that does not make an extra request per Publisher:
func (r *PublisherRepository) GetAllPublishers() []*Publisher {
sql := "SELECT * FROM publishers"
ps := make(map[int]*Publisher)
rows, err := connection.Query(sql)
if err != nil {
// log
}
for rows.Next() {
p := &Publisher{}
rows.Scan(&p.ID,&p.Name)
ps[p.ID] = p
}
sql = "SELECT * FROM books"
rows, err := connection.Query(sql)
if err != nil {
//log
}
for rows.Next() {
b := &Book{}
rows.Scan(&b.ID, &b.Name, &b.PublisherID)
ps[b.PublisherID].Books = append(ps[b.PublisherID].Books, b)
}
// you might choose to keep the map as a return value, but otherwise:
// preallocate memory for the slice
publishers := make([]*Publisher, 0, len(ps))
for _, p := range ps {
publishers = append(publishers, p)
}
return publishers
}
2) Unless you create the PublisherRepository only once, this might be a bad idea creating and closing loads of connections. Depending also on your sql client implementation I would suggest (and also have seen it for many other go database clients) to have one connection for the entire server. Pooling is done internally by many of the sql clients, that is why you should check your sql client.
If your sql client library does pooling internally use a global variable for the "connection" (it's not really one connection if pooling is done internally):
connection *sql.DB
func New () *PublisherRepository {
repo := &PublisherRepository{}
return repo.connect()
}
type PublisherRepository struct{
}
func (r *PublisherRepository) connect() *PublisherRepository {
// open new connection if connection is nil
// or not open (if there is such a state)
// you can also check "once.Do" if that suits your needs better
if connection == nil {
// ...
}
return r
}
So each time you create a new PublisherRepository, it will only check if connection already exists. If you use once.Do, go will only create the "connection" once and you are done with it.
If you have other structs that will use the connection as well, you need a global place for your connection variable or (even better) you write a little wrapper package for your sql client, that is in turn used in all your structs.
In your case, the simplest way would be to use json_agg in the query. Like here http://sqlfiddle.com/#!15/97c41/4 (sqlfiddle is slow so here is screenshot http://i.imgur.com/hxMPkUa.png) Not very Go friendly (you need to unmarshal query result data if you want to do something with the books) but all books in one query as you wanted without for loops.
As #TehSphinX said it is better to have single global db connection.
But before implementing strange queries I really suggest you to think: why do you need to return the full list of publishers and their books in one API query? I can't imagine the situation in web or mobile app where your idea might be a good decision. Usually, you just show users list of publishers then users chooses one and you show him the list of books by this publisher. This is "win-win" situation for you and your users - you can make simple queries and your users just get small sets of data that they actually need without paying for unnecessary traffic/wasting browser memory. As you said there can be 200 publishers with 100 books and I'm sure your users don't need 20000 books loaded in one request. Of course, if you are not trying to make your API more data theft friendly.
Even if you have something like a short preview-like list of books for each publisher you should think about pagination for publishers and/or denormalisation of books data for this case (add a column to publishers table with the short list of books in JSON format).