Go mysql driver hangs on connect - mysql

I met some weird situation with such simple code:
func InitDatabase(dataSourceName string) {
var err error
DBCon, err = sql.Open("mysql", dataSourceName)
if err != nil {
log.Panic(err)
}
log.Println("Ping...")
if err = DBCon.Ping(); err != nil { // <- it hangs here
log.Println(":(")
}
log.Println(":)")
}
I debugged and found, that it locks goroutine in this line (fd_poll_runtime.go, line #85), looks like it even doesn't connect:
res := runtime_pollWait(pd.runtimeCtx, mode)
It can wait here for ages, my mac becomes very hot.
I made a small change:
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if err = DBCon.PingContext(ctx); err != nil {
log.Println(":(")
}
log.Println(":)")
And it still hangs and timeout doesn't work.
Problem is, that it worked yesterday (and, funny, it works on my friend's machine, same params) and now it doesn't even after reboot and after reinstalling mysql (which works perfectly in terminal/goland/mysqlworkbench).
Go version 1.9, mysql driver: "github.com/go-sql-driver/mysql", mysql version: 5.7.20 Homebrew, macos version: High Sierra 10.13.1, I host mysql on localhost, port 3306.
UPDATE
It finally worked. I copied init code to the clean project and it worked there. I started to comment out code, to find where is the problem. I found that I have this code in the imported package. Even if code from this package executes after database ping, but init code executes before and somehow it blocks db.Ping():
func init() {
for i := 1; i <= maxWorkers; i++ {
go func(i int) {
for {
select {
case j, ok := <-jobs:
if ok {
j.invoke()
} else {
// loop
}
default:
// loop
}
}
}(i)
}
}
I rewrote method name and executed it after database ping and everything worked.
I don't know why it happened, but if you have the answer, feel free to post it here.

Related

Go - TCP Client Often Misses Secondary JSON Reply [duplicate]

This question already has an answer here:
Does ReadString() discard bytes following newline?
(1 answer)
Closed 8 months ago.
I'm trying to write a TCP client that sends out a JSON RPC to a locally hosted TCP server. The server should immediately return two replies.
The TCP server is hosted on port 60000.
This is what the client code looks like:
package main
import (
"fmt"
"log"
"bufio"
"net"
)
func main() {
d := net.Dialer{ }
c, err := d.Dial("tcp", "127.0.0.1:60000")
if err != nil {
log.Println(err)
return
}
data := `{"id": 1,"method":"mining.configure","params": [["version-rolling"],{"version-rolling.mask": "1fffe000","version-rolling.min-bit-count": 16}]}`
fmt.Fprintf(c, data+"\n")
for {
message, _ := bufio.NewReader(c).ReadString('\n')
fmt.Println(message)
}
}
This is what I'm sending out ( the "data" variable )
{
"id": 1,
"method":"mining.configure",
"params": [["version-rolling"],
{"version-rolling.mask": "1fffe000",
"version-rolling.min-bit-count": 16}]
}
This is the expected reply:
{"error":null,"id":1,"result":{"version-rolling":true,"version-rolling.mask":"1fffe000"}}
{"id":null,"method":"mining.set_version_mask","params":["1fffe000"]}
Most of the time I only get the first reponse (has the result "field") without the second JSON that has the "method" field. Sometimes I get both.
But this is the usual reply:
{"error":null,"id":1,"result":{"version-rolling":true,"version-rolling.mask":"1fffe000"}}
I know that when code becomes non-deterministic like this it's because of an asynchronous issue going on.
But every beginner tutorial I encountered teaches this exact structure for writing a TCP client. What's missing from it?
I've tested the same RPC call to the server extensively with Python with the Twisted framework and I get back the both replies 100% of the time. This makes me certain the issue is within my client code and not the server.
Depending on the timing that data is received, the call to ReadString can buffer data past the \n. The code in the question discards the reader and data buffered in the reader on each iteration of the loop.
To avoid discarding data, create the reader once and reuse inside the loop:
r := bufio.NewReader(c)
for {
message, _ := r.ReadString('\n')
fmt.Println(message)
}

How to avoid affecting other code when panic error or something else in golang

I am writing a monitor program that will merge into company's system. When the program running, seldom panicked some error since a variety of problem such as a few other's server doesn't obtain the rule of http, network error or anything else.I fix most of them but I am still afraid of influence on the master program by possibly potential errors.
From company's aspect, the stability of master program is most important, secondly, monitoring's result, the stability of monitoring is low-priority.
Is there have a method can isolate error? For example,"try...except Exception..." in python to ignore error(Admittedly, Not recommended)How to avoid effect other code when panic error or something else in golang
With panics, you can use recover like this
func recoverExample() {
defer func() {
if panicMessage := recover(); panicMessage != nil {
fmt.Printf("Panic '%v' captured", panicMessage)
}
}()
panic("Oh no!")
}
With output:
Panic 'Oh no!' captured
User recover function at the top of your function in which you have the chances of getting error. Don't use panic as it will stop the execution of your code. Use below code snippet. Also It will print the log which will let you know in which function the exception occurred.
defer func() {
if r := recover(); r != nil {
log.Println("Recovered in custom function", r)
}
}()

google app engine golang, driver: bad connection

I have some code that is working on the local GAE server but once I publish it to GAE it throws the error "driver: bad connection".
Below code generates a new *sql.DB:
func NewDb() (*sql.DB, error) {
cloud := os.Getenv("dbcloud")
local := os.Getenv("dblocal")
if appengine.IsDevAppServer() {
return sql.Open("mysql", "root#tcp("+local+":3306)/dbo")
}
return sql.Open("mysql", "root#cloudsql("+cloud+")/dbo")
}
In my app.yaml I have the following:
env_variables:
dbcloud: 'projectid:instancename'
dblocal: 'xxx.xxx.xxx.xxx'
It seems to return a new *sql.DB correctly but once I start using prepared statements is when things start to break.
db, err := NewDb() // err is nil
stmt, err := db.Prepare("INSERT INTO dbo.Users (Id) VALUES (?)") // err is driver: bad connection
I've been fighting with this for an hour now and i'm probably doing something very stupid any help would be appreciated!
I ended up needing to change my dbcloud variable to include the region of the SQL server changing it from:
'projectid:instancename'
To:
'projectid:regionname:instancename'
No idea why I need to do this as it's not in the docs of https://github.com/go-sql-driver/mysql but is all working now!

How to connect to MySQL via Standard TCP/IP over SSH using go-sql-driver?

I'm currently using MySQL Workbench on Windows 8.1 to access a remote MySQL database on a Linux server using Standard TCP/IP over SSH. Basically I have the following information:
SSH Hostname: dbserver.myorg.com:ssh-port
SSH Username: myRemoteLoginUsername
SSH Password: (stored in vault)
SSH Key File: path to a local .ppk file
MySQL Hostname: 127.0.0.1
MySQL Server Port: 3306
Username: myRemoteDbUsername
Password: (stored in vault)
Default schema: myRemoteDatabaseName
How can I connect to the database from a Go command application using github.com/go-sql-driver/mysql?
how should my DataSourceName string in the sql.Open statement look like?
db, err := sql.Open("mysql", <DataSourceName> ) {}
Is there any extra work needed to prepare a working DataSourceName string?
On my Windows PC I have putty installed. I read about tunneling and added a Dynamic tunnel for port 3306 (D3306). I expected this will let me connect using a connection to localhost:3306 and automatically forward the request to the remote db whenever I'm connected to the remote host with putty, but this didn't work as expected either.
I promised to provide my example, here it comes. Basically my solution establishes an ssh tunnel to the remote server and queries the remote database through this tunnel. The ssh tunnel is part of the solution.
The first thing I had to do is to convert my PuTTY .ppk private key file into a valid OpenSSH .pem key file. This can easily be done using the Export feature in PuTTYgen. As I wanted to support password encrypted private keys I also needed a function to decrypt the key and reformat it from its decrypted raw format into a valid format accepted by golang.org/x/crypto/ssh/ParsePrivateKey, which is needed to get the list of signers for authentication.
The solution itself consists of a package contained in two files. The main part of the application is done in main.go which contains all relevant data assignments as well as the code related to the database query. Everything related to ssh tunneling and key handling is contained in sshTunnel.go.
The solution does not provide a mechanismn for a secure password store, nor does it ask for a password. The password is provided in the code. However, it would not be too complicated to implement a callback method for password requests.
Please note: from a performance perspective this is not an ideal solution. It also lacks of proper error handling. I have provided this as an example.
The example is a tested and working example. I developed and used this from a Windows 8.1 PC. The database server is on a remote Linux system. All you need to change is the data and the query part in main.go.
Here is the first part contained in main.go:
// mysqlSSHtunnel project main.go
// Establish an ssh tunnel and connect to a remote mysql server using
// go-sql-driver for database queries. Encrypted private key pem files
// are supported.
//
// This is an example to give an idea. It's far from a performant solution. It
// lacks of proper error handling and I'm sure it could really be much better
// implemented. Please forgive me, as I just started with Go about 2 weeks ago.
//
// The database used in this example is from a real Opensimulator installation.
// It queries the migrations table in the opensim database.
//
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
"os"
)
// Declare your connection data and user credentials here
const (
// ssh connection related data
sshServerHost = "test.example.com"
sshServerPort = 22
sshUserName = "tester"
sshPrivateKeyFile = "testkey.pem" // exported as OpenSSH key from .ppk
sshKeyPassphrase = "testoster0n" // key file encrytion password
// ssh tunneling related data
sshLocalHost = "localhost" // local localhost ip (client side)
sshLocalPort = 9000 // local port used to forward the connection
sshRemoteHost = "127.0.0.1" // remote local ip (server side)
sshRemotePort = 3306 // remote MySQL port
// MySQL access data
mySqlUsername = "opensim"
mySqlPassword = "h0tgrits"
mySqlDatabase = "opensimdb"
)
// The main entry point of the application
func main() {
fmt.Println("-> mysqlSSHtunnel")
tunnel := sshTunnel() // Initialize sshTunnel
go tunnel.Start() // Start the sshTunnel
// Declare the dsn (aka database connection string)
// dsn := "opensim:h0tgrits#tcp(localhost:9000)/opensimdb"
dsn := fmt.Sprintf("%s:%s#tcp(%s:%d)/%s",
mySqlUsername, mySqlPassword, sshLocalHost, sshLocalPort, mySqlDatabase)
// Open the database
db, err := sql.Open("mysql", dsn)
if err != nil {
dbErrorHandler(err)
}
defer db.Close() // keep it open until we are finished
// Simple select query to check migrations (provided here as an example)
rows, err := db.Query("SELECT * FROM migrations")
if err != nil {
dbErrorHandler(err)
}
defer rows.Close()
// Iterate though the rows returned and print them
for rows.Next() {
var version int
var name string
if err := rows.Scan(&name, &version); err != nil {
dbErrorHandler(err)
}
fmt.Printf("%s, %d\n", name, version)
}
if err := rows.Err(); err != nil {
dbErrorHandler(err)
}
// Done for now
fmt.Println("<- mysqlSSHtunnel")
}
// Simple mySql error handling (yet to implement)
func dbErrorHandler(err error) {
switch err := err.(type) {
default:
fmt.Printf("Error %s\n", err)
os.Exit(-1)
}
}
Now the second part in sshTunnel.go:
// mysqlSSHtunnel project sshTunnel.go
//
// Everything regarding the ssh tunnel goes here. Credits go to Svett Ralchev.
// Look at http://blog.ralch.com/tutorial/golang-ssh-tunneling for an excellent
// explanation and most ssh-tunneling related details used in this code.
//
// PEM key decryption is valid for password proected SSH-2 RSA Keys generated as
// .ppk files for putty and exported as OpenSSH .pem keyfile using PuTTYgen.
//
package main
import (
"bytes"
"crypto/x509"
"encoding/base64"
"encoding/pem"
"fmt"
"golang.org/x/crypto/ssh"
"io"
"io/ioutil"
"net"
)
// Define an endpoint with ip and port
type Endpoint struct {
Host string
Port int
}
// Returns an endpoint as ip:port formatted string
func (endpoint *Endpoint) String() string {
return fmt.Sprintf("%s:%d", endpoint.Host, endpoint.Port)
}
// Define the endpoints along the tunnel
type SSHtunnel struct {
Local *Endpoint
Server *Endpoint
Remote *Endpoint
Config *ssh.ClientConfig
}
// Start the tunnel
func (tunnel *SSHtunnel) Start() error {
listener, err := net.Listen("tcp", tunnel.Local.String())
if err != nil {
return err
}
defer listener.Close()
for {
conn, err := listener.Accept()
if err != nil {
return err
}
go tunnel.forward(conn)
}
}
// Port forwarding
func (tunnel *SSHtunnel) forward(localConn net.Conn) {
// Establish connection to the intermediate server
serverConn, err := ssh.Dial("tcp", tunnel.Server.String(), tunnel.Config)
if err != nil {
fmt.Printf("Server dial error: %s\n", err)
return
}
// access the target server
remoteConn, err := serverConn.Dial("tcp", tunnel.Remote.String())
if err != nil {
fmt.Printf("Remote dial error: %s\n", err)
return
}
// Transfer the data between and the remote server
copyConn := func(writer, reader net.Conn) {
_, err := io.Copy(writer, reader)
if err != nil {
fmt.Printf("io.Copy error: %s", err)
}
}
go copyConn(localConn, remoteConn)
go copyConn(remoteConn, localConn)
}
// Decrypt encrypted PEM key data with a passphrase and embed it to key prefix
// and postfix header data to make it valid for further private key parsing.
func DecryptPEMkey(buffer []byte, passphrase string) []byte {
block, _ := pem.Decode(buffer)
der, err := x509.DecryptPEMBlock(block, []byte(passphrase))
if err != nil {
fmt.Println("decrypt failed: ", err)
}
encoded := base64.StdEncoding.EncodeToString(der)
encoded = "-----BEGIN RSA PRIVATE KEY-----\n" + encoded +
"\n-----END RSA PRIVATE KEY-----\n"
return []byte(encoded)
}
// Get the signers from the OpenSSH key file (.pem) and return them for use in
// the Authentication method. Decrypt encrypted key data with the passphrase.
func PublicKeyFile(file string, passphrase string) ssh.AuthMethod {
buffer, err := ioutil.ReadFile(file)
if err != nil {
return nil
}
if bytes.Contains(buffer, []byte("ENCRYPTED")) {
// Decrypt the key with the passphrase if it has been encrypted
buffer = DecryptPEMkey(buffer, passphrase)
}
// Get the signers from the key
signers, err := ssh.ParsePrivateKey(buffer)
if err != nil {
return nil
}
return ssh.PublicKeys(signers)
}
// Define the ssh tunnel using its endpoint and config data
func sshTunnel() *SSHtunnel {
localEndpoint := &Endpoint{
Host: sshLocalHost,
Port: sshLocalPort,
}
serverEndpoint := &Endpoint{
Host: sshServerHost,
Port: sshServerPort,
}
remoteEndpoint := &Endpoint{
Host: sshRemoteHost,
Port: sshRemotePort,
}
sshConfig := &ssh.ClientConfig{
User: sshUserName,
Auth: []ssh.AuthMethod{
PublicKeyFile(sshPrivateKeyFile, sshKeyPassphrase)},
}
return &SSHtunnel{
Config: sshConfig,
Local: localEndpoint,
Server: serverEndpoint,
Remote: remoteEndpoint,
}
}
Well, you could do that "full Go", I think.
The SSH part and port-forwarding
I'd start with something like this (I failed to google a better example).
Note two problems with this code:
It's not actually correct: it connects to a remote socket
before accepting a client connection
while it should do the reverse: accept a client connection to
a port-forwarded local socket then use the active SSH session to connect
to the remote socket and if this succeeds, spawn two goroutines to shovel
the data between those two sockets.
When configuring the SSH client, it explicitly allows password-based
authentication for unknown reason. You don't need this as you're using
pubkey-based auth.
An obstacle which might trip you there is managing an access to your SSH key.
The problem with it is that a good key should be protected by a passphrase.
You say the key's password is "stored in valut" and I honestly have no idea what "the valut" is.
On the systems I use, an SSH client either asks for the password to decrypt the key or work with a so-called "SSH agent":
On Linux-based systems it's typically an OpenSSH's ssh-agent binary working in the background which is accessed via a Unix-domain socket, and located by inspecting an environment variable named SSH_AUTH_SOCK.
On Windows I use PuTTY, and it has its own agent, pageant.exe.
I'm don't know which means PuTTY SSH client uses to locate it.
To access the OpenSSH agent, golang.org/x/crypto/ssh provides the agent subpackage which can be used to locate the agent and communicate with it.
If you need to obtain keys from pageant, I'm afraid you'll need to figure out what protocol that uses and implement it.
The MySQL part
The next step would be to integrate this with go-sql-driver.
I'd start the simplest way possible:
When you have your SSH port forwarding working,
make it listen for incoming connections on a random port on localhost.
When the connection is opened, get the port from the returned connection
object.
Use that port number to constuct the connection string to pass to an instance of sql.DB you will create to use go-sql-driver.
The driver will then connect to your port-forwarded port, and your SSH layer will do the rest.
After you have this working, I'd explore whether the driver of your choice
allows some more fine-grained tweaking like allowing you to directly pass it an instance of io.ReadWriter (an opened socket) so that you could skip the port-forwarding setup entirely and just produce new TCP connections forwarded through SSH, that is, skip the "listening locally" step.

Best way to integrate database into Go Web application

I'm new to developing web applications in Go.
I'm looking for the best way to integrate a MySQL database into my web application.
I was thinking of doing something like this:
type Context struct {
Database *sql.DB
}
// Some database methods like Close() and Query() for Context struct here
In the main function for my web application I would have something like this:
db := sql.Open(...)
ctx := Context{db}
I would then pass then pass my Context structure into various handlers that require a database connection. Would this be a good design decision or is there a better way to integrate a SQL database into my web application?
I typically do something like this:
package main
func main(){
db,err := sql.Open(...)
if err != nil {
log.Fatal(err)
}
defer db.Close()
http.HandleFunc("/feed", server.FeedHandler(db))
http.HandleFunc("/gui", server.GuiHandler(db))
...
log.Fatal(http.ListenAndServe(":8000", nil))
}
Where server is a separate package where I define, implement and test all my http handlers.
This is basically what you were thinking of but skipping the step of wrapping the db in a struct which isn't necessary. I would not recommend making the db a global variable. Having global dependencies will completely destroy any possibility of testing your http handlers in a reliable fashion.
Dependency injecting the db, as in the example above, costs you two extra characters to type every time you call the function but it allows you to test your http handlers easily using the go-sqlmock package which you definitely want to do as your web app grows.
package server
func TestFeedHandler(t *testing.T){
mockdb, err := sqlmock.New()
if err != nil {
t.Errorf("An error '%s' was not expected when opening a stub database connection", err)
}
columns := []string{"id", "name"}
sqlmock.ExpectQuery("SELECT id,name from persons;").
WillReturnRows(sqlmock.NewRows(columns).
AddRow(1234, "bob")
handler := FeedHandler(mockdb)
// test that handler returns expected result
}