How to set up loop in database until it connects to mysqlDB - mysql

I have init funcion in my golang program. I need to make it so the connection loops until it connects to database if it cant connect it idles then try's again and again for 60 seconds if it cant connect at end then it exist out. Also, have environment variable that overrides the default 60 seconds and user can put their own time for until it connects. I have my code below im looking and theres no solid solution in web .
var override string = os.Getenv("OVERRIDE")
func dsn() string {
return fmt.Sprintf("%s:%s#tcp(%s:%s)/%s", username, password, ipdb, portdb, dbName)
func init() {
var db, err = sql.Open("mysql", dsn())
if err != nil {
fmt.Println(err.Error())
} else {
fmt.Printf("Connection established to MYSQL server\n%s\n", dsn())
}
// Checking if table exists in database
rows, table_check := db.Query("select * from " + "Url" + ";")
if table_check == nil {
fmt.Println("\nTable exists in the Database")
count := 0
for rows.Next() {
count++
}
fmt.Printf("Number of records are %d \n", count)
} else {
fmt.Printf("\nTable does not exist in Database %s", table_check)
_, status := db.Exec("CREATE TABLE Url (LongUrl varchar(255),ShortUrl varchar(32));")
if status == nil {
fmt.Println("\n New table created in the Database")
} else {
fmt.Printf("\n The table was not created %s", status)
}
}
defer db.Close()
}
func main(){
......
}

You could use a switch with a timer channel.
For every 500 ms the connection is attempted, and if in 60 seconds the switch has not been resolved then the timeout channel exits the loop.
For example:
func main() {
doneCh := make(chan bool)
db, err := sql.Open("mysql", dsn())
if err != nil {
fmt.Errorf("Could not open connection: %w", err)
os.Exit(1)
}
// Start polling for connection
go func() {
ticker := time.NewTicker(500 * time.Millisecond)
timeoutTimer := time.After(time.Second * 60)
for {
select {
case <-timeoutTimer:
err = errors.New("connection to db timed out")
doneCh <- true
return
case <-ticker.C:
// Simulate some strange wait ... :)
if rand.Intn(10) == 2 {
err = db.Ping()
if err == nil {
doneCh <- true
return
}
}
fmt.Println("Retrying connection ... ")
}
}
}()
// Start waiting for timeout or success
fmt.Println("Waiting for DB connection ...")
<-doneCh
if err != nil {
fmt.Printf("Could not connect to DB: %v \n", err)
os.Exit(1)
}
defer db.Close()
// Do some work
fmt.Printf("Done! Do work now. %+v", db.Stats())
}
Note
Handle your errors as necessary. The sql.Ping() may return you some specific error which will indicate to you that you should retry.
Some errors may not be worth retrying with.
For timers and channels see:
Timers
Channels
Edit: Supposedly simpler way without channels and using a while loop
func main() {
db, err := sql.Open("mysql", dsn())
if err != nil {
fmt.Errorf("Could not open connection: %w", err)
os.Exit(1)
}
defer db.Close()
timeOut := time.Second * 60
idleDuration := time.Millisecond * 500
start := time.Now()
for time.Since(start) < timeOut {
if err = db.Ping(); err == nil {
break
}
time.Sleep(idleDuration)
fmt.Println("Retrying to connect ...")
}
if time.Since(start) > timeOut {
fmt.Printf("Could not connect to DB, timeout after %s", timeOut)
os.Exit(1)
}
// Do some work
fmt.Printf("Done! \nDo work now. %+v", db.Stats())
}

Related

Inserting data to mysql database from function . api

Highlevel:my program gets a long URL and makes it shorter(like tinyurl). But i have problem passing Long URL variable and shortURL variable from function to a mySQL database. I have tried looking it up on internet but noluck and information im getting is not close to what i have. im only posting some parts of my code the full code will be in playground
var db *sql.DB //global variable
func main(){
var db, err = sql.Open("mysql", dsn())
if err != nil {
fmt.Println(err.Error())
} else {
fmt.Printf("Connection established to MYSQL server\n%s\n", dsn())
}
defer db.Close()
linkList = map[string]string{}
http.HandleFunc("/link", addLink)
http.HandleFunc("/hpe/", getLink)
http.HandleFunc("/", Home)
// Flags to pass string ip or port to WEB app
ip := flag.String("i", "0.0.0.0", "")
port := flag.String("p", "8080", "")
flag.Parse()
fmt.Printf("Web application listening on %s \n", net.JoinHostPort(*ip, *port))
log.Fatal(http.ListenAndServe(net.JoinHostPort(*ip, *port), nil))
}
function that creates short url. Every time this func is called it produces link
func addLink(w http.ResponseWriter, r *http.Request) {
log.Println("Add Link")
key, ok := r.URL.Query()["ik"]
if ok {
if !validLink(key[0]) {
w.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(w, "Could not create shortlink need absolute path link.")
return
}
log.Println(key)
if _, ok := linkList[key[0]]; !ok {
genString := randomString(5)
linkList[genString] = key[0]
w.Header().Set("Content-Type", "text/html")
w.WriteHeader(http.StatusAccepted)
linkString := fmt.Sprintf("hpe/%s", genString, genString)
fmt.Fprintf(w, "Added shortlink\n")
fmt.Fprintf(w, linkString)
return
// database function
defer db.Close()
s:
result, err := db.Exec("insert into Url (LongUrl, ShortUrl) value(?,?);",genString,linkString)
if err != nil {
fmt.Print(err.Error())
} else {
_,err:=result.LastInsertId()
}
}
w.WriteHeader(http.StatusConflict)
fmt.Fprintf(w, "Already have this link")
return
}
w.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(w, "Failed to add link")
return
}
func getLink(w http.ResponseWriter, r *http.Request) {
path := r.URL.Path
log.Println("Get Link:", path)
pathArgs := strings.Split(path, "/")
if len(pathArgs[2]) < 1 {
w.WriteHeader(http.StatusNotFound)
http.Redirect(w, r, "0.0.0.0:8080", http.StatusTemporaryRedirect)
return
}
log.Printf("Redirected to: %s", linkList[pathArgs[2]])
http.Redirect(w, r, linkList[pathArgs[2]], http.StatusTemporaryRedirect)
//fmt.Printf("all %s", linkList)
return
}
My expectation is when func addlink gets called the info that generated from long and short url gets put into database like in the code.

I could not connect Mysql with ODBC in Golang

I want to connect mssql, mysql, postgres database servers with ODBC in Go. I have used this package: https://github.com/alexbrainman/odbc
I could connect my local mssql but I could not connect mysql. It gives this error: "Query Preparation Error SQLDriverConnect: {IM002} [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified"
My OS is Windows 10. After that I will also try on Ubuntu.
This is my code:
import (
"database/sql"
"fmt"
"log"
_ "database/sql/driver"
_ "github.com/alexbrainman/odbc"
)
func main() {
// conn, err := sql.Open("odbc",
// "uid=sa;pwd=Str0ngPa$$w0rd;port=1433;driver=sql server;server=host.docker.internal;database=master;TDS_Version=7.2;")
dbConn := fmt.Sprintf("driver=mysql;server=%s;port=3306;=database=%s;user=%s;password=%s;TDS_Version=7.2;",
"host.docker.internal", "sys", "root", "secret")
conn, err := sql.Open("odbc", dbConn)
if err != nil {
fmt.Println("Connecting Error")
return
}
defer conn.Close()
fmt.Println("Sql Server Connected")
stmt, err := conn.Prepare("select thirdtest, onetest from firsttable WHERE secondtest >= 5 limit 5")
if err != nil {
fmt.Println("Query Preparation Error", err)
return
}
defer stmt.Close()
fmt.Println("Query Prepared")
// Use db.Query() to send the query to the database. Check errors as usual.
row, err := stmt.Query()
if err != nil {
fmt.Println("Query Error", err)
return
}
defer row.Close()
fmt.Printf("\nResult set 1:\n")
for row.Next() {
var (
thirdtest float64
onetest string
)
fmt.Println("result 1")
err := row.Scan(&onetest, &thirdtest)
if err == nil {
fmt.Println(onetest, thirdtest)
}
}
err = row.Err()
if err != nil {
fmt.Printf("\nFatal: %s\n", err)
}
stmt, err = conn.Prepare("select top 5 database_id, name from sys.databases WHERE database_id >= ?")
if err != nil {
log.Fatal(err)
}
row, err = stmt.Query(1)
if err != nil {
log.Fatal(err)
}
defer row.Close()
fmt.Printf("\nResult set 2:\n")
for row.Next() {
var (
id int
name string
)
if err := row.Scan(&id, &name); err == nil {
fmt.Println(id, name)
} else {
log.Fatal(err)
}
}
err = row.Err()
if err != nil {
log.Fatal(err)
}
fmt.Printf("\nFinished correctly\n")
return
}

MySQL query sometimes deadlocks

I'm working on a program that makes a query to MySQL, then for each row, changes something with that row and then update the row.
The problem is that sometimes when performing an update I get a deadlock, I'm not sure if it's because the query isn't releasing the lock by the time I update or if it's something else.
Example of what I'm doing:
const (
selectQuery = `select user_id, original_transaction_id, max(payment_id) as max_payment_id from Payment_Receipt
where auto_renew_status = 1 group by user_id, original_transaction_id having count(*) > 1`
updateQuery = `update Payment_Receipt set auto_renew_status = 0, changed_by = "payment_receipt_condenser",
changed_time = ? where user_id = ? and original_transaction_id = ? and payment_id != ? and auto_renew_status = 1`
)
mysql.go:
func New(db *sql.DB, driver string) (database.Database, error) {
sqlDB := sqlx.NewDb(db, driver)
if err := db.Ping(); err != nil {
return nil, errors.Wrap(err, "connecting to database")
}
selectStmt, err := sqlDB.Preparex(selectQuery)
if err != nil {
return nil, errors.Wrap(err, "preparing select query")
}
updateStmt, err := sqlDB.Preparex(updateQuery)
if err != nil {
return nil, errors.Wrap(err, "preparing update query")
}
return &mysql{
db: sqlDB,
selectStmt: selectStmt,
updateStmt: updateStmt,
}, nil
}
func (m *mysql) Query() (<- chan *database.Row, error) {
rowsChan := make(chan *database.Row)
rows, err := m.selectStmt.Queryx()
if err != nil {
return nil, errors.Wrap(err, "making query")
}
go func() {
defer rows.Close()
defer close(rowsChan)
for rows.Next() {
row := &database.Row{}
if err := rows.StructScan(row); err != nil {
log.WithError(err).WithField("user_id", row.UserID.Int32).Error("scanning row")
}
// change some of the data here
// and put into channel for worker to consume
rowsChan <- row
}
}()
return rowsChan, nil
}
func (m *mysql) Update(row *database.Row) error {
tx, err := m.db.Beginx()
if err != nil {
return errors.Wrap(err, "beginning transaction")
}
if _, err := tx.Stmtx(m.updateStmt).Exec(row.ChangedTime); err != nil {
return errors.Wrap(err, "executing update")
}
if err := tx.Commit(); err != nil {
return errors.Wrap(err, "committing transaction")
}
return nil
}
worker.go
func (w *worker) Run(wg *sync.WaitGroup) {
rowsChan, err := w.db.Query()
if err != nil {
log.WithError(err).Fatal("failed making query")
}
for i := 0; i < w.config.Count(); i++ {
wg.Add(1)
go func() {
defer wg.Done()
for row := range rowsChan {
if err := w.db.Update(row); err != nil {
log.WithError(err).WithField("user_id", row.UserID.Int32).Error("updating row")
}
}
}()
}
}
You could make the results (row) channel from a Query() buffered:
func (m *mysql) Query() (<- chan *database.Row, error) {
rowsChan := make(chan *database.Row, 1000) // <- band-aid fix
// ...
}
This will ensure that the row collector function can write multiple results without waiting for your worker go-routine to read the results. The query operation will complete (provided there are 1000 rows or less), and the update go-routine operations can begin their parallel work.
If this fixes things, then consider putting say an SQL limit on your queries (e.g. LIMIT 1000) to ensure you don't hit deadlock again (if 1000+ records is a real possibility).
Crafting "pagination" style queries to grab the next say 1000 rows, using RowID markers etc. to ensure full coverage of results - all while avoiding locking out any of your update operations.

Else condition seems not working in go

I have a MySQL database with one value in it, a string: "192.168.0.1"
Here is my code:
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
func checkErr(err error) {
if err != nil {
panic(err)
}
}
func main() {
db, err := sql.Open("mysql", "be:me#tcp(127.0.0.1:3306)/ipdb?charset=utf8")
checkErr(err)
ip := "'192.168.0.1'"
rows, err := db.Query("SELECT * FROM Ip_ipdata WHERE ipHost=" + ip)
fmt.Println("insert")
if rows != nil {
for rows.Next() {
var id int
var ip string
err = rows.Scan(&id, &ip)
checkErr(err)
fmt.Println(id)
fmt.Println(ip)
}
} else {
fmt.Println("insert2")
stmt, err2 := db.Prepare("INSERT Ip_ipdata SET ipHost=2")
checkErr(err2)
_, err3 := stmt.Exec(ip)
checkErr(err3)
}
fmt.Println("end")
}
When I put "'192.168.0.1'" in ip it works and shows as expected.
But when I put "'192.168.0.2'" in ip the else statement isn't run and it just exits.
It didn't print "insert2"
screenshot 1
screenshot 2
You should get used to using '?' placeholders in your sql to allow for proper escaping and prevent any potential SQL injection attacks.
You should always check the error in Go before using the returned value.
ip := "192.168.0.1"
rows, err := db.Query("SELECT * FROM Ip_ipdata WHERE ipHost=?", ip)
if err != nil {
// handle error
}
// this will ensure that the DB connection gets put back into the pool
defer rows.Close()
for rows.Next() {
// scan here
}
The Rows returned by Query will not be nil in the case of no results, it will be empty. Try something like this:
func main() {
...
fmt.Println("insert")
checkErr(err)
defer rows.Close()
var found bool
for rows.Next() {
found = true
...
}
if !found {
fmt.Println("insert2")
...
}
fmt.Println("end")
}
Note that like #jmaloney said, more robust error handling is a must as is closing your Rows pointer.

smtp won't work in production, Unauthenticated even with credentials

I am having a hell of a time here. My code works on the local instance (OSX) with this config:
mail.host = smtp.gmail.com
mail.port = 25
mail.from = mysite.com
mail.username = email#gmail.com
But says it is unauthenticated in production (Ubuntu 12.0.1):
Here is my code (and you can find all of it at github.com/tanema/revel_mailer). The authentication error is thrown at the c.Mail(username) line.
func (m *Mailer) Send(mail_args map[string]interface{}) error {
m.renderargs = mail_args
pc, _, _, _ := runtime.Caller(1)
names := strings.Split(runtime.FuncForPC(pc).Name(), ".")
m.template = names[len(names)-2] + "/" + names[len(names)-1]
host, host_ok := revel.Config.String("mail.host")
if !host_ok {
revel.ERROR.Println("mail host not set")
}
port, port_ok := revel.Config.Int("mail.port")
if !port_ok {
revel.ERROR.Println("mail port not set")
}
c, err := smtp.Dial(fmt.Sprintf("%s:%d", host, port))
if err != nil {
return err
}
if ok, _ := c.Extension("STARTTLS"); ok {
if err = c.StartTLS(nil); err != nil {
return err
}
}
from, from_ok := revel.Config.String("mail.from")
if !from_ok {
revel.ERROR.Println("mail.from not set")
}
username, username_ok := revel.Config.String("mail.username")
if !username_ok {
revel.ERROR.Println("mail.username not set")
}
if err = c.Auth(smtp.PlainAuth(from, username, getPassword(), host)); err != nil {
return err
}
if err = c.Mail(username); err != nil {
return err
}
if mail_args["to"] != nil {
m.to = makeSAFI(mail_args["to"]) //makeSAFI == make string array from interface
}
if mail_args["cc"] != nil {
m.cc = makeSAFI(mail_args["cc"])
}
if mail_args["bcc"] != nil {
m.bcc = makeSAFI(mail_args["bcc"])
}
if len(m.to) + len(m.cc) + len(m.bcc) == 0 {
return fmt.Errorf("Cannot send email without recipients")
}
recipients := append(m.to, append(m.cc, m.bcc...)...)
for _, addr := range recipients {
if err = c.Rcpt(addr); err != nil {
return err
}
}
w, err := c.Data()
if err != nil {
return err
}
mail, err := m.renderMail(w)
if err != nil {
return err
}
if revel.RunMode == "dev" {
fmt.Println(string(mail))
}
_, err = w.Write(mail)
if err != nil {
return err
}
err = w.Close()
if err != nil {
return err
}
return c.Quit()
}
I cannot figure out what the problem is; please help.
EDIT:
So I bit the bullet and signed up for google apps and set it all up with the domain and email. That lead me to believe that it was actually the MX records that I didnt change. So I tested out sending with my personal email again now that the MX records are set. No Luck. Any idea if I can setup the MX records for a personal gmail email so I dont have to pay for google apps?
I developed a Gmail front-end and encountered a similar problem. Try the following:
Try using ssl://smtp.gmail.com and port 465.
log into your GMail account to see if it says something like 'An unauthorised third-party tried to accesss this account'. If you do, then grant your app permission to log you in.