I have problem with character encoding when pulling data from MySQL database that seems to be specific to Windows computer.
The easy solution that's working on the Linux computer:
# establish connection
db = src_mysql(user = "user", password = "pass", dbname = "training_db", host = "127.0.0.1", port = 3306)
# change/force the character encoding
dbGetQuery(db$con, 'SET NAMES utf8')
If I pull the table with this settings on Linux, everything looks fine, if I do it on Windows computer, I get messy variables from Czech characters in the database.
When I inspect the databases in MySQL Workbench on Win pc, everything seems to be allright.
Does anyone stumbled on the same problem and found a solution?
Found a solution in using dbConnect. Thus in this particular example:
db = dbConnect(drv = RMariaDB::MariaDB(),
username = "user",
password = "pass",
dbname = "training_db",
host = "127.0.0.1",
port = 3306
)
Interestingly, I remember trying to use dbConnect on Linux, but after some issues gravitating to aforementioned src_mysql. Anyway, dbConnect is recommended even in the oficial guidelines nowadays dplyr_guidelines .
Related
I'm encountering problem with MySQL. I'm getting error Lost connection to MySQL server during query. On StackOverflow they say that very likely max_allowed_packet in MySQL config is too low.
I've tried to change it in /etc/mysql/my.cnf but the file isn't editable. sudo can't be used on PA.
I'm using Sqlalchemy to handle interaction with MySQL server.
What can I do ?
Edit:
I've passed it into config variable:
class ProductionConfig:
SQLALCHEMY_DATABASE_URI = 'mysql://myconnection#server$db?max_allowed_packet=32M'
and then passed it into app and db initialization:
app.config.from_object(ProductionConfig)
db.init_app(app) # db = Sqlalchemy()
output:
TypeError: 'max_allowed_packet' is an invalid keyword argument for connect()
Problem went away after I increased pool recycling:
with Sqlalchemy:
class Config:
SQLALCHEMY_DATABASE_URI = 'mysql://<your-connection-string>'
SQLALCHEMY_POOL_RECYCLE = 280
this wasn't my idea. A member of staff at PythonAnywhere suggested it.
You can change global variables if you have SUPER privilege. But you can't make those changes persist, so if the MySQL Server restarts, the change will revert to the original value.
mysql> set global max_allowed_packet=1024*1024*32;
Query OK, 0 rows affected (0.00 sec)
I'm not familiar with Python Anywhere, but I would guess if you don't have permissions to edit the config file, you might not have SUPER privilege either.
The only other option is that you limit yourself to queries and results that fit in the max_allowed_packet that has been configured for you by the people who run Python Anywhere.
If you have need for more power, then it's time to learn to run your own server.
You can specify this flag on client side and on server side, but the server side are the limit. You can follow Bill Karwin answer or try to change on client side and whatch the effects.
On client, shell:
mysql --max_allowed_packet=32M mydb -e 'SELECT * FROM table1'
On client, PyMySQL example:
g.db = pymysql.connect(
host = 'localhost',
port = 3306,
user = 'user',
password = 'password',
database = 'mydb',
max_allowed_packet = '32M'
)
On client, SQLAlchemy, but I'm not sure if this will work as expected:
engine = create_engine("mysql+pymsql://user:pass#some_mariadb/dbname?charset=utf8mb4&max_allowed_packet=32M")
I'm developing a script in RStudio which connects to local MySQL Server using the R package RMariaDB (not RMySQL - for other reasons though the outcome is the same).
I can both connect via storing the password in the script like:
localuserpassword <- "password"
all_projectsDb <- dbConnect(RMariaDB::MariaDB(), user='user', password=localuserpassword, dbname='projects', host='localhost')
or by way of a .my.cnf using credentials:
[client]
[mygroup]
host=127.0.0.1
user=user
password=password
port=3306
database=projects
and R code as
settingsfile = '/Users/oscar_w/.my.cnf'
all_projectsDb <- dbConnect(RMariaDB::MariaDB(), default.file = settingsfile, group="mygroup", dbname = 'projects')
The above work just fine but if I want to connect with .mylogin.cnf created in mysql_config_editor and looks like
[client]
[mygroup]
user = user
password = *****
host = 127.0.0.1
port = 3306
with the R script code like
# define location of config file
settingsfile = '/Users/oscar_w/.mylogin.cnf'
all_projectsDb <- dbConnect(RMariaDB::MariaDB(), default.file = settingsfile, group="mygroup", dbname = 'projects', password = NULL, user = NULL)
I get the error
Error: Failed to connect: Access denied for user 'root'#'localhost' (using password: NO)
I have tried various combinations of arguments expressing null or otherwise. And have entered my password with mysql_config_editor with double quotes around it. In https://cran.r-project.org/web/packages/RMariaDB/RMariaDB.pdf it specifies the use of .mylogin.cnf but I cannot find a way to make it work. Does anyone know a solution to this or has the same issue? Thanks
It looks like you're trying to log in both with and without a password, which isn't allowed. The RMariaDB documentation says that if the password argument is NULL or omitted, only users without a password can log in.
I've set up a wordpress install on appfog (using rackspace), and cloned the install to my local machine for development. I know the install works (using MAMP) because I created a local mysql database and changed wp-config.php to point to it. However, I want to develop without having to change wp-config.php every time I commit. After doing some research, it seems like the Appfog service Caldecott lets me tunnel into the mysql database on the server, using af tunnel. Unfortunately, I'm having issues with getting it working. Even if I change my MAMP mysql port to something like 8889, and tunnel mysql through port 3306, it looks like it's connected but I still get "Error establishing a database connection" when loading my localhost Wordpress. When I quit the mysql monitor (using ctrl+x, ctrl+c), I get a message stating "Error: 'mysql' execution failed; is it in your $PATH?'. Originally, no, it wasn't, but I've fixed my PATH variable on my local machine so that when I go to Terminal and just type mysql, it loads up.
So I guess my question is 2 parts:
1.)Am I going with the right approach for Wordpress development on my local machine
and
2.)If so, why is the tunnel not working?
One way to deal with this is to mimic the VCAP_SERVICES environment variable on your local system with your local database settings. This would allow you to use the same custom AppFog wp-config.php file which looks at VCAP_SERVICES to get its db creds.
export VCAP_SERVICES='{"mysql-5.1": [{"credentials": {
"hostname": "localhost",
"port": 3306,
"password": "root",
"name": "LOCAL_DATABASE_NAME",
"user": "root"}}]}'
EDIT: You will need to restart the Apache server in MAMP after setting this env var. (Thanks Dex)
This will eliminate the need to point your development code at your production database.
simply test for whether vcap_services are available. if they are, use one config. if they arent, use a different config.
here is an example of my local + appfog development website configuration file.
if(getenv("VCAP_SERVICES")){
//if in webserver
$services_json = json_decode(getenv("VCAP_SERVICES"),true);
$mysql_config = $services_json["mysql-5.1"][0]["credentials"];
$username = $mysql_config["username"];
$password = $mysql_config["password"];
$hostname = $mysql_config["hostname"];
$port = $mysql_config["port"];
$db = $mysql_config["name"];
define("DB_SERVER", "$hostname");
define("DB_USER", "$username");//enter your database username
define("DB_PASS", "$password");//databse password
define("DB_NAME", "$db");//database name
} else {
//if in local development
define("DB_SERVER", "localhost");
define("DB_USER", "website");//enter your database username
define("DB_PASS", "dfgdfgdf");//databse password
define("DB_NAME", "fgdf_web");//database name
}
also, you can use .afignore same way you'd use .gitignore to ignore some files from the af update feature. u can update once with appropriate config, then add afignore, then it will never get updated again.
Here is a quick and very dirty script to automate the process based on Tim Santeford's answer. Be sure to change the LOCAL_DATABASE_NAME
#!/bin/bash
export VCAP_SERVICES='{"mysql-5.1": [{"credentials": {"hostname": "localhost", "port": 8889, "password": "root", "name": "LOCAL_DATABASE_NAME", "user": "root"}}]}'
/Applications/MAMP/Library/bin/httpd -k stop
sleep 3
/Applications/MAMP/Library/bin/httpd -k start
I'm trying to make a connection via ODBC from a Python program running on Ubuntu to a MySQL box on the same machine. (I'm using ODBC instead of DB-API because I'm going to be using different database engines and I want a consistent way of getting to the system catalog, like SQLTables.)
But when I connect to my MySQL database and run SQLTables, I get this for my first row:
(u'\U0067007a\U005f0061\U00690062\U006c006c\U006e0069\U005f0067\U00750061\U006f0074\U0061006d\U00690074\U006e006f\U0063005f\U00630063',
u'', u'\U00750061\U00680074\U0067005f\U006f0072\U00700075',
u'\U00410054\U004c0042', u'')
Clearly these are unicode values, but they look like they are for really high codepoints, and sure enough, I can't encode them into ASCII. All my table names should be ASCII.
Is there some setting that I'm missing or have wrong that is causing this?
my odbc.ini is:
[ODBC Data Sources]
mu = MySQL
[mu]
Description = MySQL Database Test
Driver = MySQL
Server = localhost
Database = ccc2
Port = 3306
My odbcinst.ini is
[MySQL]
Description = ODBC for MySQL
Driver = /usr/lib/odbc/libmyodbc.so
FileUsage = 1
Found it. I needed to specify Charset=UTF8 in my connection paramters, like so:
[ODBC Data Sources]
mu = MySQL
[mu]
Description = MySQL Database Test
Driver = MySQL
Server = localhost
Database = ccc2
Port = 3306
Charset = UTF8
I have my unixodbc odbc.ini configure file like this:
[test]
Driver = /usr/local/lib/libmyodbc5-5.1.8.so
Description = Connector/ODBC 5.1.8 Driver DSN
SERVER = 127.0.0.1
PORT = 3306
USER = root
Password =
DATABASE = test
OPTION =
SOCKET =
And the problem is that it will not use the database as specified above, which is 'test'.
What I have to do is to manually execute a direct sql to change to database and run my query:
SQLExecDirect(stmt, "USE test", SQL_NTS);
SQLExecDirect(stmt, "SELECT * FROM mytable", SQL_NTS);
Any idea on how should I get rid of the 'USE test' which is a mysql command.
Why is unixodbc not setting 'test' as the default db since it's already specified in the conf file?
As on windows the driver manager (unixODBC in this case) only acts on the Driver tag, all other entries in the DSN are up to the driver to interpret. It doesn't notice there is a database= entry and know by magic that in this driver it should execute a "USE" command, and for another call SQLSetConnectAttr( SQL_ATTR_CURRENT_CATALOG ).
On my copy of the MySQL driver, it certainly uses the database= entry. However, I would check that 1. The copy of the driver you are using is built to use the unixODBC lib to access the shared config file (libodbcinst.so), or the driver is reading it directly, and is reading the same ini file as unixODBC. Possibly check with strace to see what ini is opened after the driver is loaded. Maybe try setting ODBCINI=/path/to/your/odbc.ini
[test]
Driver = /usr/local/lib/libmyodbc5-5.1.8.so
Description = Connector/ODBC 5.1.8 Driver DSN
SERVER = 127.0.0.1
PORT = 3306
USER = root
Password =
**DATABASE** = test
**OPTION** = **3**
**SOCKET** =
and can change options between 1,2 too