I'm working on Golang script that automatically clone a database.
I'm using go-sql-driver but i can't find in the documentation a way to create a new database.
Connection to MySQL require an URL scheme like:
user:password#tcp(localhost:3306)/database_name
But the database not exists yet, I just want to connect to the server and then create a new one.
How can I do that? I have to use another driver?
You can perfectly use the go-sql-driver. However, you need to use a mysql user which has the proper access rights to create new databases.
Here is an example:
func create(name string) {
db, err := sql.Open("mysql", "admin:admin#tcp(127.0.0.1:3306)/")
if err != nil {
panic(err)
}
defer db.Close()
_,err = db.Exec("CREATE DATABASE "+name)
if err != nil {
panic(err)
}
_,err = db.Exec("USE "+name)
if err != nil {
panic(err)
}
_,err = db.Exec("CREATE TABLE example ( id integer, data varchar(32) )")
if err != nil {
panic(err)
}
}
Note that the database name is not provided in the connection string. We just create the database after the connection (CREATE DATABASE command), and switch the connection to use it (USE command).
Note: the VividCortex guys maintain a nice database/sql tutorial and documentation at http://go-database-sql.org/index.html
If you want to create a new database if it does not exist, and use it directly in your program, be aware that database/sql maintains a connection pool.
Therefore the opened database connection, should preferably contain the database name. I've seen "Error 1046: No database selected" when database/sql opens a new connection after using db.Exec("USE "+name) manually.
func createAndOpen(name string) *sql.DB {
db, err := sql.Open("mysql", "admin:admin#tcp(127.0.0.1:3306)/")
if err != nil {
panic(err)
}
defer db.Close()
_,err = db.Exec("CREATE DATABASE IF NOT EXISTS "+name)
if err != nil {
panic(err)
}
db.Close()
db, err = sql.Open("mysql", "admin:admin#tcp(127.0.0.1:3306)/" + name)
if err != nil {
panic(err)
}
defer db.Close()
return db
}
My db_config.go file looks like for GO ORM 2.0:
package infrastructure
import (
"fmt"
"gorm.io/driver/mysql"
"gorm.io/gorm"
"gorm2.0/utils"
)
//Setup Models: initializaing the mysql database
func GetDatabaseInstance() *gorm.DB {
get := utils.GetEnvWithKey
USER := get("DB_USER")
PASS := get("DB_PASS")
HOST := get("DB_HOST")
PORT := get("DB_PORT")
DBNAME := get("DB_NAME")
dsn := fmt.Sprintf("%s:%s#tcp(%s:%s)/%s?charset=utf8&parseTime=True&loc=Local", USER, PASS, HOST, PORT, DBNAME)
db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
_ = db.Exec("CREATE DATABASE IF NOT EXISTS " + DBNAME + ";")
if err != nil {
panic(err.Error())
}
return db
}
Related
I'm facing some issue while fetching the data from the MySQL database using golang below is my code and the error that I'm facing
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
func ConnectMsqlDb() (db *sql.DB, err error) {
db, err = sql.Open("mysql", fmt.Sprintf("%s:%s#tcp(%s:"+sqlDbPort+")/"+sqlDB, sqlUserName, sqlPassword, dbServerIP))
if err != nil {
return nil, err
}
//defer db.Close()
err = db.Ping()
if err != nil {
return nil, err
}
return db, nil
}
func GetSqlData() (err error, data interface{}) {
db, err := ConnectMsqlDb()
if err != nil {
// here it will returning me the error
return err, nil
}
rows, err := db.Query("SELECT * FROM users")
if err != nil {
return err, nil
}
for rows.Next() {
}
defer db.Close()
fmt.Println(rows)
return err, rows
}
func main() {
err, data := GetSqlData()
fmt.Println("data", data, "err", err)
}
error
data commands out of sync. Did you run multiple statements at once?
Can anyone tell me why I'm facing this issue
If the error is comming while opening a connection to mysqld , it could be possible that MySQL server (mysqld) is blocking the host from connecting to it. It means that mysqld has received many connection requests from the given host that were interrupted in the middle.
Read why? To confirm you could see the DB's logs as well. So, a way to unblock it is to flush the cached hosts using the MySQL CLI:
mysql> FLUSH HOSTS;
And then try again to connect.
Plus, give this answer a read as well. Might help.
You can check the state of the socket used by mysqld using (For Linux):
netstat -nt
Check if there's any previously hanging connection from the host to mysqld. If yes, then kill it and try again.
This question already has answers here:
How to use global var across files in a package?
(3 answers)
Closed 2 years ago.
I am trying to insert something into a MySQL database with this code. When I run a debugger I see that it stops at this line:
stmt, err := users_db.Client.Prepare(queryInsertUser)
and my debugger takes me to this line and the value of db is nil. Note that these lines are not mine and is supposed to be the internal Go modules etc.
Also my database connection has been made as I can see in my console.
// conn returns a newly-opened or cached *driverConn.
func (db *DB) conn(ctx context.Context, strategy connReuseStrategy) (*driverConn, error) {
db.mu.Lock()
if db.closed {
db.mu.Unlock()
return nil, errDBClosed
}
and
func (user *User) Save() *errors.RestError {
stmt, err := users_db.Client.Prepare(queryInsertUser)
if err != nil {
return errors.NewInternalServerError(err.Error())
}
defer stmt.Close()
insertResult, err := stmt.Exec(user.FirstName, user.LastName, user.Email, user.DateCreated)
// not so important part removed
return nil
}
and code inside my users_db package
package users_db
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
"log"
)
var (
Client *sql.DB
username = "bla bla"
password = "bla bla"
host = "bla bla"
schema = "bla bla"
)
func init() {
dataSourceName := fmt.Sprintf("%s:%s#tcp(%s)/%s?charset=utf8",
username, password, host, schema,
)
var err error
Client, err := sql.Open("mysql", dataSourceName)
if err != nil {
panic(err)
}
if err = Client.Ping(); err != nil {
panic(err)
}
log.Println("database successfully configured")
}
What can be the reason for this error and how can this be resolved?
As #mkopriva commented you are declaring Client again in init. You can resolve it by doing this:
c, err := sql.Open("mysql", dataSourceName)
Client = c
I have data on my local computer in 2 MySQL databases (dbConnOuter and dbConnInner) that I want to process and collate into a 3rd database (dbConnTarget).
The code runs for about 17000 cycles, then stops with these error messages:
[mysql] 2018/08/06 18:20:57 packets.go:72: unexpected EOF
[mysql] 2018/08/06 18:20:57 packets.go:405: busy buffer
As far as I can tell I'm properly closing the connections where I'm reading from and I'm using Exec for writing, that I believe handles its own resources. I've also tried prepared statements, but it didn't help, the result was the same.
Below is the relevant part of my code, and it does similar database operations prior to this without any issues.
As this is one of my first experiments with Go, I can't yet see where I might be wasting my resources.
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
)
var dbConnOuter *sql.DB
var dbConnInner *sql.DB
var dbConnTarget *sql.DB
func main() {
dbConnOuter = connectToDb(dbUserDataOne)
dbConnInner = connectToDb(dbUserDataTwo)
dbConnTarget = connectToDb(dbUserDataThree)
// execute various db processing functions
doStuff()
}
func connectToDb(dbUser dbUser) *sql.DB {
dbConn, err := sql.Open("mysql", fmt.Sprintf("%v:%v#tcp(127.0.0.1:3306)/%v", dbUser.username, dbUser.password, dbUser.dbname))
if err != nil {
panic(err)
}
dbConn.SetMaxOpenConns(500)
return dbConn
}
// omitted similar db processing functions that work just fine
func doStuff() {
outerRes, err := dbConnOuter.Query("SELECT some outer data")
if err != nil {
panic(err)
}
defer outerRes.Close()
for outerRes.Next() {
outerRes.Scan(&data1)
innerRes, err := dbConnInner.Query("SELECT some inner data using", data1)
if err != nil {
panic(err)
}
innerRes.Scan(&data2, &data3)
innerRes.Close()
dbConnTarget.Exec("REPLACE INTO whatever", data1, data2, data3)
}
}
I feel like I must be totally missing the point.
I try to run something along the lines of the example below, but the ? is not expanded into the argument passed in.
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
)
db, err := sql.Open(...)
if err != nil { ... }
_, err = db.Query("SELECT * FROM foo WHERE bar=?", bar)
Also, who's concern is expanding it? it show's up in the doc of database/sql but other conversations hinted it may be the concern of the driver.
What am I missing? Any pointer in the right direction is greatly appreciated.
You're (probably) not telling it to use the mysql driver;
db, err := sql.Open("mysql", connString)
Here is some of my sample code:
var query = "INSERT IGNORE INTO blah (`col1`, `col2`, `col3`) VALUES (?, ?, ?)"
r, err := db.Query(query, some_data_1, some_data_2, some_data_3)
// Failure when trying to store data
if err != nil {
msg := fmt.Sprintf("fail : %s", err.Error())
fmt.Println(msg)
return err
}
r.Close() // Always do this or you will leak connections
I create the MySQL pool (yes, it's a pool NOT a connection) with:
import (
// mysql driver
_ "github.com/go-sql-driver/mysql"
"database/sql"
"fmt"
)
connString := fmt.Sprintf("%s:%s#(%s:%d)/%s?timeout=30s",
user,
password,
host,
port,
database,
)
db, err := sql.Open("mysql", connString)
if err != nil {
return nil, err
}
err = db.Ping() // test the pool connection(s)
if err != nil {
return nil, err
}
return db, nil // No error, return the pool connections
I also highly suggest you print out any errors in the database connection process and query process. It could be that you are using the driver, but you lack permissions, the database is down, etc.
So I'm trying to use the OpenID package for Golang, located here: https://github.com/yohcop/openid-go
In the _example it says that it uses in memory storage for storing the nonce/discoverycache information and that it will not free the memory and that I should implement my own version of them using some sort of database.
My database of choice is MySQL, I have tried to implement what I thought was correct (but is not, does not give me any compile errors, but crashes on runtime)
My DiscoveryCache.go is as such:
package openid
import (
"database/sql"
"log"
//"time"
_ "github.com/go-sql-driver/mysql"
"github.com/yohcop/openid-go"
)
type SimpleDiscoveredInfo struct {
opEndpoint, opLocalID, claimedID string
}
func (s *SimpleDiscoveredInfo) OpEndpoint() string { return s.opEndpoint }
func (s *SimpleDiscoveredInfo) OpLocalID() string { return s.opLocalID }
func (s *SimpleDiscoveredInfo) ClaimedID() string { return s.claimedID }
type SimpleDiscoveryCache struct{}
func (s SimpleDiscoveryCache) Put(id string, info openid.DiscoveredInfo) {
/*
db, err := sql.Query("mysql", "db:connectinfo")
errCheck(err)
rows, err := db.Query("SELECT opendpoint, oplocalid, claimedid FROM discovery_cache")
errCheck(err)
was unsure what to do here because I'm not sure how to
return the info properly
*/
log.Println(info)
}
func (s SimpleDiscoveryCache) Get(id string) openid.DiscoveredInfo {
db, err := sql.Query("mysql", "db:connectinfo")
errCheck(err)
var sdi = new(SimpleDiscoveredInfo)
err = db.QueryRow("SELECT opendpoint, oplocalid, claimedid FROM discovery_cache WHERE id=?", id).Scan(&sdi)
errCheck(err)
return sdi
}
And my Noncestore.go
package openid
import (
"database/sql"
"errors"
"flag"
"fmt"
"time"
_ "github.com/go-sql-driver/mysql"
)
var maxNonceAge = flag.Duration("openid-max-nonce-age",
60*time.Second,
"Maximum accepted age for openid nonces. The bigger, the more"+
"memory is needed to store used nonces.")
type SimpleNonceStore struct{}
func (s *SimpleNonceStore) Accept(endpoint, nonce string) error {
db, err := sql.Open("mysql", "dbconnectinfo")
errCheck(err)
if len(nonce) < 20 || len(nonce) > 256 {
return errors.New("Invalid nonce")
}
ts, err := time.Parse(time.RFC3339, nonce[0:20])
errCheck(err)
rows, err := db.Query("SELECT * FROM noncestore")
defer rows.Close()
now := time.Now()
diff := now.Sub(ts)
if diff > *maxNonceAge {
return fmt.Errorf("Nonce too old: %ds", diff.Seconds())
}
d := nonce[20:]
for rows.Next() {
var timeDB, nonce string
err := rows.Scan(&nonce, &timeDB)
errCheck(err)
dbTime, err := time.Parse(time.RFC3339, timeDB)
errCheck(err)
if dbTime == ts && nonce == d {
return errors.New("Nonce is already used")
}
if now.Sub(dbTime) < *maxNonceAge {
_, err := db.Query("INSERT INTO noncestore SET nonce=?, time=?", &nonce, dbTime)
errCheck(err)
}
}
return nil
}
func errCheck(err error) {
if err != nil {
panic("We had an error!" + err.Error())
}
}
Then I try to use them in my main file as:
import _"github.com/mysqlOpenID"
var nonceStore = &openid.SimpleNonceStore{}
var discoveryCache = &openid.SimpleDiscoveryCache{}
I get no compile errors but it crashes
I'm sure you'll look at my code and go what the hell (I'm fairly new and only have a week or so experience with Golang so please feel free to correct anything)
Obviously I have done something wrong, I basically looked at the NonceStore.go and DiscoveryCache.go on the github for OpenId, replicated it, but replaced the map with database insert and select functions
IF anybody can point me in the right direction on how to implement this properly that would be much appreciated, thanks! If you need anymore information please ask.
Ok. First off, I don't believe you that the code compiles.
Let's look at some mistakes, shall we?
db, err := sql.Open("mysql", "dbconnectinfo")
This line opens a database connection. It should only be used once, preferably inside an init() function. For example,
var db *sql.DB
func init() {
var err error
// Now the db variable above is automagically set to the left value (db)
// of sql.Open and the "var err error" above is the right value (err)
db, err = sql.Open("mysql", "root#tcp(127.0.0.1:3306)")
if err != nil {
panic(err)
}
}
Bang. Now you're connected to your MySQL database.
Now what?
Well this (from Get) is gross:
db, err := sql.Query("mysql", "db:connectinfo")
errCheck(err)
var sdi = new(SimpleDiscoveredInfo)
err = db.QueryRow("SELECT opendpoint, oplocalid, claimedid FROM discovery_cache WHERE id=?", id).Scan(&sdi)
errCheck(err)
Instead, it should be this:
// No need for a pointer...
var sdi SimpleDiscoveredInfo
// Because we take the address of 'sdi' right here (inside Scan)
// And that's a useless (and potentially problematic) layer of indirection.
// Notice how I dropped the other "db, err := sql.Query" part? We don't
// need it because we've already declared "db" as you saw in the first
// part of my answer.
err := db.QueryRow("SELECT ...").Scan(&sdi)
if err != nil {
panic(err)
}
// Return the address of sdi, which means we're returning a pointer
// do wherever sdi is inside the heap.
return &sdi
Up next is this:
/*
db, err := sql.Query("mysql", "db:connectinfo")
errCheck(err)
rows, err := db.Query("SELECT opendpoint, oplocalid, claimedid FROM discovery_cache")
errCheck(err)
was unsure what to do here because I'm not sure how to
return the info properly
*/
If you've been paying attention, we can drop the first sql.Query line.
Great, now we just have:
rows, err := db.Query("SELECT ...")
So, why don't you do what you did inside the Accept method and parse the rows using for rows.Next()... ?