Python 3.8.8 programm with Flask 2.0.1 and Flask-SQLAlchemy 2.5.1
MySql database, collation of the tables: utf8_general_ci.
I'm using two other sqlserver DB with SQLALCHEMY_BINDS. Everything runs on Windows 10.
Some chars from select queries on the MySql DB comes wrong: "situazione è decisamente migliorata"
should be: "situazione è decisamente migliorata"
This would solve the problem:
mystring.encode('cp1252').decode('utf8')
but I need a solution at program level. I tried:
appending to SQLALCHEMY_DATABASE_URI connection string:
"?charset=utf8" or "?charset=cp1215" and others
setting app.config['MYSQL_CHARSET'] and
app.config['MYSQL_DATABASE_CHARSET'] to 'utf8', 'utf8mb4', 'latin1', 'cp1252'
...
passing a parameter to SQLAlchemy like db = SQLAlchemy(use_native_unicode="utf8"), many variations here too
No attemp worked. Please I need suggestions.
Are you looking for a way to specify per database connection encoding ?
For all connections try to use
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {'encoding': 'cp1252'}
For specific connections to different DBs you can also use engine_options:
engine = create_engine('mysql://user:password#hostname/dbname',
encoding='cp1252')
Got the solution.
The problem was not a problem.
The person who build the original database (that is quite old) coded wrong some characters.
Some of my approaches and the one suggested by olegsv, worked, I checked that debugging deep down into into sqlalchemy data structures, the driver accepted the characters encoding, but the very chars in data were themself worong.
This was unespected.
Maybe I should delete the whole question.
Related
I am in the process of migrating databases from sqlite to mysql. Now that I've migrated the data to mysql, I'm not able to use my sqlalchemy code (in Python3) to access it in the new mysql db. I was under the impression that sqlalchemy syntax was database agnostic (i.e. the same syntax would work for accessing sqlite and mysql), but this appears not to be the case. So my question is: Is it absolutely required to use a DBAPI in addition to Sqlalchemy to read the data? Do I have to edit all of my sqlalchemy code to now read mysql?
The documentation says: The MySQL dialect uses mysql-python as the default DBAPI. There are many MySQL DBAPIs available, including MySQL-connector-python and OurSQL, which I think means that I DO need a DBAPI.
My old code with sqlite successfully worked like this with sqlite:
engine = create_engine('sqlite:///pmids_info.db')
def connection():
conn = engine.connect()
return conn
def load_tables():
metadata = MetaData(bind=engine) #init metadata. will be empty
metadata.reflect(engine) #retrieve db info for metadata (tables, columns, types)
inputPapers = Table('inputPapers', metadata)
return inputPapers
inputPapers = load_tables()
def db_inputPapers_retrieval(user_input):
result = engine.execute("select title, author, journal, pubdate, url from inputPapers where pmid = :0", [user_input])
for row in result:
title = row['title']
author = row['author']
journal = row['journal']
pubdate = row['pubdate']
url = row['url']
apa = str(author+' ('+pubdate+'). '+title+'. '+journal+'. Retrieved from '+url)
return apa
This worked fine and dandy. So then I tried to update it to work with the mysql db like this:
engine = create_engine('mysql://snarkshark#localhost/pmids_info')
At first when I tried to run my sample code like this, it complained because I didn't have MySqlDB. Some googling around informed me that MySqlDB does NOT work for Python 3. So then I tried pip installing pymysql and changing my engine statement to
engine = create_engine('mysql+pymysql://snarkshark#localhost/pmids_info')
which also ends up giving me various syntax errors when I try to adjust things.
So what I want to know, is if there is any way I can get my current syntax to work with mysql? Since the syntax is from sqlalchemy, I thought it would work perfectly for the exact same data in mysql that was previously in sqlite. Will I have to go through and update ALL of my db functions to use the syntax of the DBAPI?
This will sound like a dumb answer, but you'll need to change all the places where you're using database-specific behavior. SQLAlchemy does not guarantee that anything you do with it is portable across all backends. It leaks some abstractions on purpose to allow you to do things that are only available on certain backends. What you're doing is like using Python because it's cross-platform, then doing a bunch of os.fork()s everywhere, and then being surprised that it doesn't work on Windows.
For your specific case, at a minimum, you need to wrap all your raw SQL in text() so that you're not affected by the supported paramstyle of the DBAPI. However, there are still subtle differences between different dialects of SQL, so you'll need to use the SQLAlchemy SQL expression language instead of raw SQL if you want portability. After all that, you'll still need to be careful not to use backend-specific features in the SQL expression language.
I've made an application in Java EE that uses Hibernate to communicate with MySQL. It works perfectly on my Windows development machine, but I have problem on debian, where the application is deployed.
When I search for keyword with Polish letters(like ł, ą, ć, ó etc,) the result is ok on Windows, but on server, where I have imported the database it does not work.
Hibernate query looks like this:
#NamedQuery(name = "Keyword.findByKeyword", query = "SELECT k FROM Keyword k WHERE k.keyword = :keyword")
and is called like this:
myEntityManager.createNamedQuery("Keyword.findByKeyword").setParameter("keyword", keyword).getSingleResult();
When I use mysql on debian via SSH and type in SELECT query manually:
SELECT * FROM keywords WHERE keyword = 'ser żółty';
it also works and return single result. Encoding and collations of tables and columns are also ok. In datasource configuration I've added
?UseUnicode=true&characterEncoding=utf8
parameters, but it also did not help. I thought that maybe there is a problem with encoding in data from request send by form, but the problem appears even if the parameter i.e. "ser żółty" is hardcoded in my repository class.
I also use Hibernate Search for indexing and the FullTextEntityManager return correct results with Polish letters.
I assume that there is some problem between Hibernate and MySQL, but I have no more ideas what could I change. Any suggestions?
Server Wildfly9.0.1, MySQL 5.6
Ok the problem was in encoding on the mysql server level. It was set to latin1 by default. To fix this follow this question Change MySQL default character set to UTF-8 in my.cnf? and edit your my.cnf file.
I just updated the JDBC driver for my application from
mysql-connector-java-3.1.12-bin.jar
to
mysql-connector-java-5.1.34-bin.jar.
With the v3.x driver, this kind of a query works:
select concat("<a href>", count(sakila.payment.payment_id), "</a>")
from sakila.payment;
But now with the new v5.x driver, the query only works with a cast().
select cast(concat("<a href>", count(sakila.payment.payment_id), "</a>")
as char(30)) from sakila.payment;
Is there any property in the MySQL database I can change?
I don't want to change hundreds of queries like that.
I suspect that you will have to bite the bullet and update your code. There is a bug report here that seems to match your circumstances and the status of that bug report is "Won't fix". The response from the developers ([4 Apr 2007 17:43] Reggie Burnett) was:
This is something that we can't really fix. Let me explain.
MySQL has several issues when it comes to reporting whether a result if binary or not. This was very bad on MySQL versions prior to 5.0 but it's still a problem even today. The SQL you reported is returned by MySQL as binary when it obviously is not. The connector can't know for sure. With 5.0.5 and 5.0.6, we tried to make a "best guess" but that code caused more problems than it solved, so with 5.0.7 we have rolled it out. Your SQL will return string properly with 5.0.7, but that doesn't mean it's fixed. In fact, it returns string because we are ignoring the binary flag so that means you could generate valid SQL that should return binary and 5.0.7 will return string.
Until the server is fixed, the connector just can't always do the right thing. I hope this has cleared it up somewhat.
I am using the RJDBC package to connect to a MySQL (Maria DB) database in R on a Windows 7 machine and I am trying a statement like
select a as b
from table
but the column will always continue to be named "a" in the data frame.
This works normally with RODBC and RMySQL but doesn't work with RJDBC. Unfortunately, I have to use RJDBC as this is the only package that has no problem with the encoding of chinese, hebrew and so on letters (set names and so on don't seem to work with RODBC and RMySQL).
Has anybody experienced this problem?
I have run into the same frustrating issue. Sometimes the AS keyword would have its intended effect, but other times it wouldn't. I was unable to identify the conditions to make it work correctly.
Short Answer: (Thanks to Simon Urbanek (package maintainer for RJDBC), Yev, and Sebastien! See the Long Answer.) One thing that you may try is to open your JDBC connection using ?useOldAliasMetadataBehavior=true in your connection string. Example:
drv <- JDBC("com.mysql.jdbc.Driver", "C:/JDBC/mysql-connector-java-5.1.18-bin.jar", identifier.quote="`")
conn <- dbConnect(drv, "jdbc:mysql://server/schema?useOldAliasMetadataBehavior=true", "username", "password")
query <- "SELECT `a` AS `b` FROM table"
result <- dbGetQuery(conn, query)
dbDisconnect(conn)
This ended up working for me! See more details, including caveats, in the Long Answer.
Long Answer: I tried all sorts of stuff, including making views, changing queries, using JOIN statements, NOT using JOIN statements, using ORDER BY and GROUP BY statements, etc. I was never able to figure out why some of my queries were able to rename columns and others weren't.
I contacted the package maintainer (Simon Urbanek.) Here is what he said:
In the vast majority of cases this is an issue in the JBDC driver, because there is really not much RJDBC can do other than to call the driver.
He then recommended that I make sure I had the most recent JDBC driver for MySQL. I did have the most recent version. However, it got me thinking "maybe it IS a bug with the JDBC driver." So, I searched Google for: mysql jdbc driver bug alias.
The top result for this query was an entry at bugs.mysql.com. Yev, using MySQL 5.1.22, says that when he upgraded from driver version 5.0.4 to 5.1.5, his column aliases stopped working. Asked if it was a bug.
Sebastien replied, "No, it's not a bug! It's a documented change of behavior in all subsequent versions of the driver." and suggested using ?useOldAliasMetadataBehavior=true, citing documentation for the JDBC driver.
Caveat Lector: The documentation for the JDBC driver states that
useColumnNamesInFindColumn is preferred over useOldAliasMetadataBehavior unless you need the specific behavior that it provides with respect to ResultSetMetadata.
I haven't had the time to fully research what this means. In other words, I don't know what all of the ramifications are of using useOldAliasMetadataBehavior=true are. Use at your own risk. Does someone else have more information?
I don't know RJDBC, but in some cases when it is necessary to give permanent aliases to columns without renaming them, you can use VIEWs
CREATE OR REPLACE VIEW v_table AS
SELECT a AS b
FROM table
... and then ...
SELECT b FROM v_table
There is a separate function in the ResultSetMetaData interface for retrieving the column label vs the column name:
String getColumnLabel(int column) throws SQLException;
Gets the designated column's suggested title for use in printouts and
displays. The suggested title is usually specified by the SQL AS
clause. If a SQL AS is not specified, the value returned
fromgetColumnLabel will be the same as the value returned by the
getColumnName method.
Using getColumnLabel should resolve this issue (if not, check that your JDBC driver is following this spec).
e.g.
ResultSetMetaData rsmd = rs.getMetaData();
int columnCount = rsmd.getColumnCount();
while(rs.next()) {
for (int i = 1; i < columnCount + 1; i++) {
String label = rsmd.getColumnLabel(i);
System.out.println(rs.getString(label));
}
}
This is the work around we use for R and SAP HANA via RJDBC:
names(result)[1]<-"b"
It's not the nicest work around, but since Aaron's solution does work for us, we went with this "solution".
I've connected to a MySQL database using Perl DBI. I would like to find out which database I'm connected to.
I don't think I can use:
$dbh->{Name}
because I call USE new_database and $dbh->{Name} only reports the database that I initially connected to.
Is there any trick or do I need to keep track of the database name?
Try just executing the query
select DATABASE();
From what I could find, the DBH has access to the DSN that you initially connected with, but not after you made the change. (There's probably a better way to switch databases.)
$dbh->{Name} returns the db name from your db handle.
If you connected to another db after connected with your dbh, using mysql query "USE db_name", and you did not setup a new perl DBI db handle, of course, $dbh->{Name} will return the first you previously connected to... It's not spontaneic generation.
So to get the connected db name once the db handle is set up - for DBI mysql:
sub get_dbname {
my ($dbh) = #_;
my $connected_db = $dbh->{name};
$connected_db =~ s/^dbname=([^;].*);host.*$/$1/;
return $connected_db;
}
You can ask mysql:
($dbname) = (each %{$dbh->selectrow_hashref("show tables")}) =~ /^Tables_in_(.*)/;
Update: obviously select DATABASE() is a better way to do it :)
When you create a connection object it is for a certain database. In DBI's case anyway. I I don't believe doing the SQL USE database_name will affect your connection instance at all. Maybe there is a select_db (My DBI is rusty) function for the connection object or you'll have to create a new connection to the new database for the connection instance to properly report it.
FWIW - probably not much - DBD::Informix keeps track of the current database, which can change if you do operations such as CREATE DATABASE. The $dbh->{Name} attribute is specified by the DBI spec as the name used when the handle is established. Consequently, there is an Informix-specific attribute $dbh->{ix_DatabaseName} that provides the actual current database name. See: perldoc DBD::Informix.
You could consider requesting the maintainer(s) of DBD::MySQL add a similar attribute.