I'm using the github.com/go-sql-driver/mysql and mysql 5.7.10. I have a function:
bulkSetStatus := func(docVers []*_documentVersion) error {
if len(docVers) > 0 {
query := strings.Repeat("CALL documentVersionSetStatus(?, ?); ", len(docVers))
args := make([]interface{}, 0, len(docVers)*2)
for _, docVer := range docVers {
args = append(args, docVer.Id, docVer.Status)
}
_, err := db.Exec(query, args...)
return err
}
return nil
}
which works if len(docVers) == 1 but when there are more, resulting in multiple CALLs to the stored procedure, it errors:
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'CALL documentVersionSetStatus(?, ?)' at line 1
I have also tried a newline character between each call but I get the same error. If I run this in mysql workbench with multiple CALLs to this procedure it works fine, I'm not sure what is wrong with the syntax here.
I have logged out the exact full text with the arguments and it is as expected:
CALL documentVersionSetStatus("9c71cac14a134e7abbc4725997d90d2b", "inprogress"); CALL documentVersionSetStatus("beb65318da96406fa92990426a279efa", "inprogress");
go-sql-driver, by default, does not allow you to have multiple statements in one query (as you are doing by chaining together multiple CALL statements like that) due to the security implications if an attacker manages to perform SQL injection (for example, by injecting 0 OR 0; DROP TABLE foo).
To allow this, you must explicitly enable it by passing multiStatements parameter when connecting to the database, e.g.
db, err := sql.Open("mysql", "user:password#/dbname?multiStatements=True")
Source: https://github.com/go-sql-driver/mysql#multistatements
I have fixed the proc call by doing some manual string interpolation for the parameters instead of using the correct ? way of doing it:
bulkSetStatus := func(docVers []*_documentVersion) error {
if len(docVers) > 0 {
query := strings.Repeat("CALL documentVersionSetStatus(%q, %q); ", len(docVers))
args := make([]interface{}, 0, len(docVers)*2)
for _, docVer := range docVers {
args = append(args, docVer.Id, docVer.Status)
}
_, err := db.Exec(fmt.Sprintf(query, args...))
return err
}
return nil
}
so I swap out the ? for %q and us fmt.Sprintf to inject the parameters, I should note that slugonamission's answer is partially correct, I did need to add the connection string parameter multiStatements=true in order to get this to work with my other changes. I will log an issue on the github repo it looks like there may be some param interpolation issue when there is more than one statement, I think the error was happening because the mysql db was trying to run the script with ? literals in it.
Related
I have a stored procedure which inserts an entity and returns its new UUID, and while I can generate the code to create the right query, it errors out when using go-sql-driver/mysql. So the following code...
func test_insert() *sql.Rows {
db := openDbConnection()
defer db.Close()
results, err := db.Query("call saveargument(null, 'Test Argument', 'Test Argument', '1', null, null, null, 1, 'test_user', #newargumentid);\nselect #newargumentid;")
toolkit.HandleError(err)
return results
}
func openDbConnection() *sql.DB {
var db, err = sql.Open("mysql", getConnectionString(entities.GetConfig()))
toolkit.HandleError(err)
return db
}
... produces the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 'select #newargumentid' at line 2
I'm not sure why such a basic piece of SQL could be so problematic. Any insights anyone?
You can't run more than one statement in a single db.Query() call. This is true of most query interfaces in all programming languages.
Call your stored procedure in one call to db.Query(), then query select #newargumentid in a second call to db.Query().
I just tested #Bill's answer and I can confirm it does not work as at Go1.14. My workaround was to rewrite my stored procedure to "return" values by doing a select at the end:
CREATE PROCEDURE foo()
BEGIN
# do stuff...
SELECT 'bar';
END;
And then in Go just read it like any other query:
res, _ := db.Query("CALL foo()")
res.Next()
var bar string
res.Scan(&bar)
println(bar)
Note this method also works for multiple columns and multiple rows.
OS Mojave,
MySQL v8.0.15,
go 1.12.3 darwin/amd64
import (
"database/sql"
// import mysql driver anonymously (just run the init)
_ "github.com/go-sql-driver/mysql"
)
...
_, err = db.db.Exec("USE ?", "test")
if err != nil {
return errors.Wrapf(err, "error selecting database %s", opt.Database)
}
_, err = db.db.Prepare("SELECT value FROM ? WHERE key = ?")
if err != nil {
return errors.Wrap(err, "error generating SELECT statement")
}
The error I get is error selecting database test: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '?' at line 1
I get the same error (about syntax around '?') for any parameter replacement I attempt to do. I've checked the manual as well as tutorials and examples I've found online and can't quite tell what I'm doing wrong.
If I replace the '?'s with values (strings) then everything works fine.
Question marks ? in prepared statements are for values, not table, database or column names.
Use fmt.Sprintf to fill in the database structure related values.
Example:
_, err := db.db.Prepare(fmt.Sprintf("SELECT value FROM %s WHERE key = ?", table))
The problem
I wrote an application which synchronizes data from BigQuery into a MySQL database. I try to insert roughly 10-20k rows in batches (up to 10 items each batch) every 3 hours. For some reason I receive the following error when it tries to upsert these rows into MySQL:
Can't create more than max_prepared_stmt_count statements:
Error 1461: Can't create more than max_prepared_stmt_count statements
(current value: 2000)
My "relevant code"
// ProcessProjectSkuCost receives the given sku cost entries and sends them in batches to upsertProjectSkuCosts()
func ProcessProjectSkuCost(done <-chan bigquery.SkuCost) {
var skuCosts []bigquery.SkuCost
var rowsAffected int64
for skuCostRow := range done {
skuCosts = append(skuCosts, skuCostRow)
if len(skuCosts) == 10 {
rowsAffected += upsertProjectSkuCosts(skuCosts)
skuCosts = []bigquery.SkuCost{}
}
}
if len(skuCosts) > 0 {
rowsAffected += upsertProjectSkuCosts(skuCosts)
}
log.Infof("Completed upserting project sku costs. Affected rows: '%d'", rowsAffected)
}
// upsertProjectSkuCosts inserts or updates ProjectSkuCosts into SQL in batches
func upsertProjectSkuCosts(skuCosts []bigquery.SkuCost) int64 {
// properties are table fields
tableFields := []string{"project_name", "sku_id", "sku_description", "usage_start_time", "usage_end_time",
"cost", "currency", "usage_amount", "usage_unit", "usage_amount_in_pricing_units", "usage_pricing_unit",
"invoice_month"}
tableFieldString := fmt.Sprintf("(%s)", strings.Join(tableFields, ","))
// placeholderstring for all to be inserted values
placeholderString := createPlaceholderString(tableFields)
valuePlaceholderString := ""
values := []interface{}{}
for _, row := range skuCosts {
valuePlaceholderString += fmt.Sprintf("(%s),", placeholderString)
values = append(values, row.ProjectName, row.SkuID, row.SkuDescription, row.UsageStartTime,
row.UsageEndTime, row.Cost, row.Currency, row.UsageAmount, row.UsageUnit,
row.UsageAmountInPricingUnits, row.UsagePricingUnit, row.InvoiceMonth)
}
valuePlaceholderString = strings.TrimSuffix(valuePlaceholderString, ",")
// put together SQL string
sqlString := fmt.Sprintf(`INSERT INTO
project_sku_cost %s VALUES %s ON DUPLICATE KEY UPDATE invoice_month=invoice_month`, tableFieldString, valuePlaceholderString)
sqlString = strings.TrimSpace(sqlString)
stmt, err := db.Prepare(sqlString)
if err != nil {
log.Warn("Error while preparing SQL statement to upsert project sku costs. ", err)
return 0
}
// execute query
res, err := stmt.Exec(values...)
if err != nil {
log.Warn("Error while executing statement to upsert project sku costs. ", err)
return 0
}
rowsAffected, err := res.RowsAffected()
if err != nil {
log.Warn("Error while trying to access affected rows ", err)
return 0
}
return rowsAffected
}
// createPlaceholderString creates a string which will be used for prepare statement (output looks like "(?,?,?)")
func createPlaceholderString(tableFields []string) string {
placeHolderString := ""
for range tableFields {
placeHolderString += "?,"
}
placeHolderString = strings.TrimSuffix(placeHolderString, ",")
return placeHolderString
}
My question:
Why do I hit the max_prepared_stmt_count when I immediately execute the prepared statement (see function upsertProjectSkuCosts)?
I could only imagine it's some sort of concurrency which creates tons of prepared statements in the meantime between preparing and executing all these statements. On the other hand I don't understand why there would be so much concurrency as the channel in the ProcessProjectSkuCost is a buffered channel with a size of 20.
You need to close the statement inside upsertProjectSkuCosts() (or re-use it - see the end of this post).
When you call db.Prepare(), a connection is taken from the internal connection pool (or a new connection is created, if there aren't any free connections). The statement is then prepared on that connection (if that connection isn't free when stmt.Exec() is called, the statement is then also prepared on another connection).
So this creates a statement inside your database for that connection. This statement will not magically disappear - having multiple prepared statements in a connection is perfectly valid. Golang could see that stmt goes out of scope, see it requires some sort of cleanup and then do that cleanup, but Golang doesn't (just like it doesn't close files for you and things like that). So you'll need to do that yourself using stmt.Close(). When you call stmt.Close(), the driver will send a command to the database server, telling it the statement is no longer needed.
The easiest way to do this is by adding defer stmt.Close() after the err check following db.Prepare().
What you can also do, is prepare the statement once and make that available for upsertProjectSkuCosts (either by passing the stmt into upsertProjectSkuCosts or by making upsertProjectSkuCosts a func of a struct, so the struct can have a property for the stmt). If you do this, you should not call stmt.Close() - because you aren't creating new statements anymore, you are re-using an existing statement.
Also see Should we also close DB's .Prepare() in Golang? and https://groups.google.com/forum/#!topic/golang-nuts/ISh22XXze-s
I've already setup and pinged my mysql database connection. It is working and I can return rows using both db.Query and by preparing a query first. I can use the placeholder ? to then specify an id. Is it possible to use the ? as a placeholder for a column name? In the example here I am trying to return all rows from column firstName in table persons.
qry, err := db.Prepare("SELECT ? FROM persons")
if err != nil { log.Fatal(err) }
defer qry.Close()
rows, err :=qry.Query("firstName")
if err != nil { log.Fatal(err) }
defer rows.Close()
I get the following error:
Error 1064: You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right
syntax to use near '?' at line 1
You can't use placeholders for identifiers (such as table and column names), placeholders are for values. You can think of identifiers as being similar to variable or function names in Go so being able to use placeholders for identifiers would be akin to having an eval as in various scripting languages.
This reduces you to using fmt.Sprintf and similar string operations for building the SQL when you don't know the identifiers until runtime:
col := "firstName"
sql := fmt.Sprintf("select %s from persons", col)
but this opens you up to SQL injection and quoting problems so you'd want some sort of whitelist:
quotedColumns := map[string]string{
"firstName": "`firstName`",
"lastName": "`lastName`",
...
}
quoted, ok := quotedColumns[columnName]
if !ok {
// Do something with the error here and run away...
}
sql := fmt.Sprintf("select %s from persons", quoted)
Note that I've included the MySQL backtick quoting in the map's values. There's nothing in the standard interface for quoting/escaping an identifier so you have to do it yourself. If you're already writing the whitelist map by hand then you may as well include the quoting by hand too; otherwise you could write your own quoting function for identifiers by reading the MySQL documentation on quoting and doing a couple (hopefully) simple string operations.
I'm using the mymysql package and I'm trying to create a function which gets an SQL query and some parameters (as variadic empty interface):
func FindByQuery(statement string, params ...interface{}) (diver *DiverT, err error) {
values := make([]interface{}, len(params))
for i := range params {
values[i] = params[i]
}
// Both statements result in the same error...
row, _, execError := Db.QueryFirst(statement,values...)
row, _, execError := Db.QueryFirst(statement,params...)
// Additional code...
}
When I call this method using some kind of SQL, I always get an SQL error. I do something like:
FindByQuery("SELECT * FROM Diver WHERE Name=?", "Markus")
Which results in the following error:
Received #1064 error from MySQL server: "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '?%!(EXTRA string=Markus)' at line 1"
What should I do that the parameter is converted correctly to a string (or whatever it is, if I have different parameter(s))?
Try using printf format syntax:
FindByQuery("SELECT * FROM Diver WHERE Name=%s", "Markus")