I can connect to the RDS instance using mysql -h ... command so I know it's not a security group problem.
I've tried to use:
sql.Open("mysql", "id:password#tcp(your-amazonaws-uri.com:3306)/dbname")
in the readme file of go-sql-driver(https://github.com/go-sql-driver/mysql), but it doesn't seem to work.
I'm using my username under the RDS instance instead of id here though.
Edit:
The error returned is: panic runtime error: invalid memory address or nil pointer deference [signal 0xb code=0x1 addr=0x20 pc=0x5b551e]
goroutine 16 [running]
runtime.panic(0x7d4fc0, 0xa6ca73)...database/sql.(*Rows).Next...
It works fine with my local DB.
The connection string for sql.Open() is in DSN format.
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
db, err := sql.Open("mysql", "<username>:<password>#tcp(<AWSConnectionEndpoint >:<port>)/<dbname>")
if err != nil {
fmt.Print(err.Error())
}
defer db.Close()
Make sure the actual error isn't related to an import issue (as in issues 266)
Check (to be sure you are using the latest versions, as in this issue):
your Go-MySQL-Driver version (or git SHA)
your Go version (run go version in your console)
If the error isn't directly in the Open step, but when accessing the Rows, check this comment out:
Use either a for loop (for rows.Next() { ... }) or something like this:
if rows.Next() {
// whatever
} else {
// catch error with rows.Err()
}
rows.Close() // <- don't forget this if you are not iterating over ALL results
Related
As the title says, I have an error when importing go-mysql-driver package. I have installed the go-my-sql driver in my machine but the error still persists. I use XAMPP for local hosting and here’s the block of program.
package model
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
type Table interface {
Name() string
Field() ([]string, []interface{})
}
func Connect(username string, password string, host string, database string) (*sql.DB, error) {
conn := fmt.Sprintf("%s:%s#tcp(%s:3306)/%s", username, password, host, database)
db, err := sql.Open("mysql", conn)
return db, err
}
func CreateDB(db *sql.DB, name string) error {
query := fmt.Sprintf("CREATE DATABASE %v", name)
_, err := db.Exec(query)
return err
}
func CreateTable(db *sql.DB, query string) error {
_, err := db.Exec(query)
return err
}
func DropDB(db *sql.DB, name string) error {
query := fmt.Sprintf("DROP DATABASE %v", name)
_, err := db.Exec(query)
return err
}
could not import github.com/go-sql-driver/mysql (no required modules provides package "github.com/go-sql-driver/mysql")
screenshot of what's happening
It seems that you read the tutorial for an older go version.
Go 1.17 requires dependencies must be explicitly in go.mod.
Maybe you could try go module first (https://go.dev/blog/using-go-modules)
Your IDE is not showing you the whole picture. By running go run main.go (or whatever main file you have) on the command line, you can see the same error as you're seeing on your IDE with some extra:
$ go run main.go
main.go:7:5: no required module provides package github.com/go-sql-driver/mysql; to add it:
go get github.com/go-sql-driver/mysql
By issuing the suggested command go get github.com/go-sql-driver/mysql, you'll get the dependency added to your go.mod file and the contents of the package will be downloaded to your machine.
Your next execution will work:
$ go run main.go
Hello world
I've made some small modifications to your code to work, I'll add them here for completeness:
I've used the same source, but changed the package name to main.
I've added a main function to the bottom of the file:
func main() {
fmt.Println("Hello world")
_, err := Connect("username", "password", "localhost", "db")
if err != nil {
panic(err)
}
}
I've saved to a file named main.go
I've initialized the go.mod file by running go mod init test and go mod tidy, then I took the steps described on the beginning of the answer.
I have a Go project using goose for Mysql migrations. I would like to bind the migrations to the package executable so that the executable can be deployed and used independently from any system, similar to JAR files in JAVA projects.
Is there an equivalent in Go to accomplish that?
How to get a single file which can migrate database and work
Install
go get -u github.com/pressly/goose/cmd/goose
Make app. I base it on examplemain.go and add run option. Suppose your project is located at github.com/user/project:
package main
import (
"database/sql"
"flag"
"log"
"os"
"github.com/pressly/goose"
// Init DB drivers. -- here I recommend remove unnecessary - but it's up to you
_ "github.com/go-sql-driver/mysql"
_ "github.com/lib/pq"
_ "github.com/mattn/go-sqlite3"
_ "github.com/ziutek/mymysql/godrv"
// here our migrations will live -- use your path
_ "github.com/user/project/migrations"
)
var (
flags = flag.NewFlagSet("goose", flag.ExitOnError)
dir = flags.String("dir", ".", "directory with migration files")
)
func main() {
flags.Usage = usage
flags.Parse(os.Args[1:])
args := flags.Args()
//////
if len(args) > 1 && args[0] == "run" {
log.Printf("PROGRAM RUN\n") //
.....
os.Exit(0)
}
if len(args) > 1 && args[0] == "create" {
if err := goose.Run("create", nil, *dir, args[1:]...); err != nil {
log.Fatalf("goose run: %v", err)
}
return
}
if len(args) < 3 {
flags.Usage()
return
}
if args[0] == "-h" || args[0] == "--help" {
flags.Usage()
return
}
driver, dbstring, command := args[0], args[1], args[2]
switch driver {
case "postgres", "mysql", "sqlite3", "redshift":
if err := goose.SetDialect(driver); err != nil {
log.Fatal(err)
}
default:
log.Fatalf("%q driver not supported\n", driver)
}
switch dbstring {
case "":
log.Fatalf("-dbstring=%q not supported\n", dbstring)
default:
}
if driver == "redshift" {
driver = "postgres"
}
db, err := sql.Open(driver, dbstring)
if err != nil {
log.Fatalf("-dbstring=%q: %v\n", dbstring, err)
}
arguments := []string{}
if len(args) > 3 {
arguments = append(arguments, args[3:]...)
}
if err := goose.Run(command, db, *dir, arguments...); err != nil {
log.Fatalf("goose run: %v", err)
}
}
func usage() {
log.Print(usagePrefix)
flags.PrintDefaults()
log.Print(usageCommands)
}
var (
usagePrefix = `Usage: goose [OPTIONS] DRIVER DBSTRING COMMAND
Drivers:
postgres
mysql
sqlite3
redshift
Examples:
goose sqlite3 ./foo.db status
goose sqlite3 ./foo.db create init sql
goose sqlite3 ./foo.db create add_some_column sql
goose sqlite3 ./foo.db create fetch_user_data go
goose sqlite3 ./foo.db up
goose postgres "user=postgres dbname=postgres sslmode=disable" status
goose mysql "user:password#/dbname?parseTime=true" status
goose redshift "postgres://user:password#qwerty.us-east-1.redshift.amazonaws.com:5439/db"
status
Options:
`
usageCommands = `
Commands:
up Migrate the DB to the most recent version available
up-to VERSION Migrate the DB to a specific VERSION
down Roll back the version by 1
down-to VERSION Roll back to a specific VERSION
redo Re-run the latest migration
status Dump the migration status for the current DB
version Print the current version of the database
create NAME [sql|go] Creates new migration file with next version
`
)
Create folder for migrations:
mkdir migrations && cd migrations
Create first migrations. We will use go-style migrations:
goose mysql "user:password#/dbname?parseTime=true" create init go
You'll get a file 00001_init.go with Go code. Migrations are baked in it as SQL-commands. Just edit them as you need.
Then go to the main folder and build the application:
cd ..
go build -v -o myapp *.go
You'll get a file myapp with all the migrations baked in it. To check move it to some other place, for example to /tmp folder, and run from there:
./myapp mysql "user:password#/dbname?parseTime=true" status
Run your app:
./myapp run
Result
You have single file which can be used as a migration tool so as a working application itself. All the migration are buil-it. In source code they are stored in a subpackage migrations - so it's easy to edit.
If you use Docker you may put folder to the image.
If not projects like https://github.com/rakyll/statik or https://github.com/jteeuwen/go-bindata can help.
If you're already using Goose, one option would be to write the migrations in Go instead of SQL. Based on the Go migrations example in the Goose repo, when you build the goose binary here, it will bundle all the *.go ones into the binary.
This was the output after I built the example and removed all files except the binary itself. The Go-based migration was embedded:
2017/10/31 11:22:31 Applied At Migration
2017/10/31 11:22:31 =======================================
2017/10/31 11:22:31 Mon Jun 19 21:56:00 2017 -- 00002_rename_root.go
If you're looking to use SQL-based migrations but don't want to take on additional dependencies, you could embed your SQL-based migrations into *.go files as string constants, then, starting from the go-migrations example, add an init phase to main.go that writes them out to the current directory before proceeding.
Hi i am trying to execute a script to fill data into a database using Golang
func executeTestScript(){
cmd := exec.Command("/usr/local/mysql/bin/mysql", "-h127.0.0.1", "-P3333", "-uusr", "-pPassxxx", "-Ddtb_test", "< /Users/XXX/Documents/test/scripts/olds/SCRIPT_XXX.sql")
var out, stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
fmt.Println(fmt.Sprintf("Error executing query. Command Output: %+v\n: %+v, %v", out.String(), stderr.String(), err))
log.Fatalf("Error executing query. Command Output: %+v\n: %+v, %v", out.String(), stderr.String(), err)
}
}
The problem is that i am getting the error:
ERROR 1049 (42000): Unknown database '< /Users/XXX/Documents/test/scripts/olds/SCRIPT_XXX.sql'
i think the problem is the last param (the sql script path) that the exec thinks is the dbname
The following command in the terminal is working:
/usr/local/mysql/bin/mysql --host=127.0.0.1 --port=3333 --user=usr --password=Passxxx --database=dtb_test < /Users/XXX/Documents/roseula/scripts/olds/SCRIPT_XXX.sql
but i try to replicate in Go to automatize the execution of the script.
The script have drop tables, create tables, inserts, and PK with FK relationships its a very complete one so i cant execute line by line, because of that i decided to execute de mysql program to insert the data in the database.
Any suggestions?
+1 to answer from #MatteoRagni for showing how to do stdin redirection in Golang.
But here's a simple alternative that I use:
cmd := exec.Command("/usr/local/mysql/bin/mysql", "-h127.0.0.1", "-P3333",
"-uusr", "-pPassxxx", "-Ddtb_test",
"-e", "source /Users/XXX/Documents/test/scripts/olds/SCRIPT_XXX.sql")
You don't have to make the mysql client read the script using stdin redirection. Instead, you can make the mysql client execute a specific command, which is source <scriptname>.
P.S.: I also would not put the host, port, user, and password in your code. That means you have to recompile your program any time you change those connection parameters. Also it's not secure to use passwords in plaintext on the command-line. Instead, I'd put all the connection parameters into a defaults file and use mysql --defaults-file=FILENAME.
This is a little example that runs something like:
cat < test.txt
that is what I think you are missing in your code:
package main
import (
"fmt"
"os/exec"
"os"
)
func main() {
cmd := exec.Command("cat")
file, _ := os.Open("test.txt")
cmd.Stdin = file
out, _ := cmd.Output()
fmt.Printf("%s\n", out)
}
That prints in the console the content of test.txt, as read by cat. You will need to adapt it to your problem.
Something like:
func executeTestScript(){
cmd := exec.Command("/usr/local/mysql/bin/mysql", "-h127.0.0.1", "-P3333", "-uusr", "-pPassxxx", "-Ddtb_test")
dump, dump_err = os.Open("/Users/XXX/Documents/test/scripts/olds/SCRIPT_XXX.sql")
if dump_err != nil {
/* Handle the error if file not opened */
}
cmd.Stdin = dump
var out, stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
fmt.Println(fmt.Sprintf("Error executing query. Command Output: %+v\n: %+v, %v", out.String(), stderr.String(), err))
log.Fatalf("Error executing query. Command Output: %+v\n: %+v, %v", out.String(), stderr.String(), err)
}
}
if I'm not wrong...
I'm puzzled.
I try to run test cases using gorm with mysql in golang and I wanna buile MySQL just for testing, but it does not run safely.
I wanna use this package go-test-mysqld
Error message is below.
panic: sql: Register called twice for driver mysql
My code is
func TestMain(m *testing.M) {
mysqld, err := mysqltest.NewMysqld(nil)
if err != nil {
log.Fatal("runTests: failed", err)
}
defer mysqld.Stop()
dbm, err = gorm.Open("mysqld", mysqld.Datasource("test", "", "", 0 ))
if err != nil {
log.Fatal("db connection error:", err)
}
defer dbm.Close()
code := m.Run()
os.Exit(code)
}
What is the problems in my code?
Or is it impossible to build another mysql in using gorm?
Do you have some ideas?
I'm using github.com/go-sql-driver/mysql package to connect to MySQL. It works well except when I select a database (USE), I can't run queries against it.
package main
import (
"database/sql"
"fmt"
"log"
)
import _ "github.com/go-sql-driver/mysql"
func main() {
dsn := "root:#/"
db, err := sql.Open("mysql", dsn)
if err != nil {
fmt.Println("Failed to prepare connection to database. DSN:", dsn)
log.Fatal("Error:", err.Error())
}
err = db.Ping()
if err != nil {
fmt.Println("Failed to establish connection to database. DSN:", dsn)
log.Fatal("Error:", err.Error())
}
_, err = db.Query("USE test")
if err != nil {
fmt.Println("Failed to change database.")
log.Fatal("Error:", err.Error())
}
_, err = db.Query("SHOW TABLES")
if err != nil {
fmt.Println("Failed to execute query.")
log.Fatal("Error:", err.Error())
}
}
The program produces this output:
Error 1046: No database selected
Specify the database directly in the DSN (Data Source Name) part of the sql.Open function:
dsn := "user:password#/dbname"
db, err := sql.Open("mysql", dsn)
That's because db maintains a connection pool that has several connections to mysql database."USE test" just let one connection use schema test.
When you do database query later,the driver will select one idle connection,if the connection that use test schema is selected,it will be normal,but if another connection is chosen, it does not use test,so it will report an error:no database selected.
If you add a clause:
db.SetMaxOpenConns(1)
the db will maintain only one connection,it will not have an error.And of course it's impossible in high concurrency scene.
If you specify the database name in sql.open() function,all the connection will use this data base which can avoid this problem.
In your case you need to use transactions:
tx, _ := db.Begin()
tx.Query("USE test")
tx.Query("SHOW TABLES")
tx.Commit()
For SELECT/UPDATE/INSERT/etc need to specify DB name in the query.
As other answers mentioned, sql.DB is not a single connection but a connection pool. When you execute use database, imagine you executed your query on just one connection in the pool. Next query will get another connection from the pool which has no databases selected.
I would strongly advise against using transactions for this (as several places suggest).
I would suggest to use context:
ctx := context.Background()
conn, err := db.Conn(ctx)
conn.ExecContext(ctx, "use mydb")
defer conn.Close()
var found int
err = conn.QueryRowContext(ctx, "SELECT count(*) as found FROM mytable").Scan(&found)
if err != nil {
panic(err)
}