I ran the inspector on a table that has known default constraints but no results are showing up from the inspector. Is this some kind of issue with MSSQL dialect? I can see indexes and primary keys using the inspector.
from loguru import logger
import sqlalchemy as sa
logger.debug("begin engine create")
engine = sa.create_engine(DB_URI)
logger.debug("begin engine inspection")
insp = sa.inspect(engine)
logger.debug('list columns with default value')
columns = insp.get_columns(my_table_name)
[print(c["default"]) for c in columns if c["default"]]
logger.debug('index list')
indexes = insp.get_indexes(my_table_name)
[print(i) for i in indexes]
logger.debug(f'{my_table_name} PK')
pk_constraint = insp.get_pk_constraint(my_table_name)
print(pk_constraint)
https://github.com/sqlalchemy/sqlalchemy/discussions/6554#discussioncomment-799571
My issue was the user I was connecting with did not have table definition permissions.
Related
I had a django model field which was working in the default sqlite db:
uuid = models.TextField(default=uuid.uuid4, editable=False, unique=True).
However, when I tried to migrate to MySQL, I got the error:
django.db.utils.OperationalError: (1170, "BLOB/TEXT column 'uuid' used in key specification without a key length")
The first thing I tried was removing the unique=True, but I got the same error. Next, since I had another field (which successfully migrated ):
id = models.UUIDField(default=uuid.uuid4, editable=False)
I tried changing uuid to UUIDField, but I still get the same error. Finally, I changed uuid to:
uuid = models.TextField(editable=False)
But I am still getting the same error when migrating (DROP all the tables, makemigrations, migrate --run-syncdb). Ideally, I want to have a UUIDField or TextField with default = uuid.uuid4, editable = False, and unique = True, but I am fine doing these tasks in the view when creating the object.
You need to set max_length for your uuid field. Since UUID v4 uses 36 charactes, you can set it to max_length=36 (make sure you don't have longer values in the db already):
uuid = models.TextField(default=uuid.uuid4, editable=False, unique=True, max_length=36)
I'm using a linked server on a SQL 2016 server to read and write data in different MySQL tables.
With MySQL Connector/ODBC 5.3 everything works fine, after updating the MySQL Connector/ODBC to latest version 8.0.26 due to security reasons updating the MySQL data causes an error! Selecting MySQL data still works fine, as well as inserting new data; updating and deleting MySQL data is not possible any longer.
In different threads I found hints for correct structure of MySQL tables as requirement to maintain the data via linked server. Tables must have a primary key column (with no auto increment) and at least one column with type timestamp must exist. So I created following simple test table:
CREATE TABLE `test_odbc_3` (
`TDBC_ID` int(11) NOT NULL,
`TDBC_DESC` varchar(30) COLLATE utf8_unicode_ci NOT NULL,
`TDBC_TSU` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`TDBC_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_c
For maintenance of MySQL data I use the OPENQUERY syntax as follows:
Inserting a new row:
INSERT OPENQUERY(TDBAMYSQLTEST, 'SELECT TDBC_ID, TDBC_DESC, TDBC_TSU FROM admin_db.test_odbc_3')
VALUES (24,'row 4','2019-05-19 14:22:41)
works fine!
Selecting all rows:
SELECT *
FROM OPENQUERY( TDBAMYSQLTEST, 'SELECT TDBC_ID, TDBC_DESC, TDBC_TSU FROM admin_db.test_odbc_3')
works fine – result is:
TDBC_ID TDBC_DESC TDBC_TSU
21 row 1 2009-04-17 14:11:41.0000000
22 row 2 2009-04-17 14:11:41.0000000
23 row 3 2009-04-17 14:11:41.0000000
24 row 4 2019-05-19 14:22:41.0000000
Trying to update a row in this table causes the following error:
UPDATE OPENQUERY( TDBAMYSQLTEST, 'SELECT TDBC_ID, TDBC_DESC, TDBC_TSU FROM admin_db.test_odbc_3 WHERE TDBC_ID = 23')
SET TDBC_DESC = 'mydesc'
WHERE TDBC_ID = 23
Msg 7343, Level 16, State 4, Line 10
The OLE DB provider "MSDASQL" for linked server "TDBAMYSQLTEST" could not UPDATE table "[MSDASQL]". Unknown provider error
Based on different threads I checked and set the configuration of environment as well.
MySQL Connector/ODBC configuration:
ODBC driver information
ODBC connector configuration
Enabled Provider Options:
Dynamic parameter
Nested queries
Level zero only
Allow inprocess
Index as access path
Linked Server Properties set to True:
Data Access
RPC
RPC Out
Anybody has an idea about how to get running with update MySQL data again? Thnx in advance for any help!
Instead of using QPENQUERY it's necessary to use EXEC ... AT syntax as follows:
For selecting data:
EXEC('SELECT * FROM admin_db.test_odbc_3 WHERE TDBC_ID = 3') AT TDBAMYSQLTEST
For updating data:
EXEC('UPDATE admin_db.test_odbc_3 set TDBC_DESC = ''mynew_value'' WHERE TDBC_ID = 3') AT TDBAMYSQLTEST
I'm new on Python and trying to learn some basic Data Manipulation (the main focus is Data Science). So I'm still grasping Pandas and everything else.
What I'm trying to achieve, is to create a DataFrame and store it on a MySQL database. This is my script (that don't work):
from sqlalchemy.types import VARCHAR
from sqlalchemy import create_engine
import pandas as pd
import numpy as np
frame = pd.DataFrame(np.random.random((4,4)),
index=['val1','val2','val3','val4'],
columns=['col1','col2','col3','col4'])
engine = create_engine('mysql+pymysql://user:password#localhost/python_samples')
frame.to_sql('rnd_vals', engine, dtype={'index':VARCHAR(5)})
When I try to execute this, I get the error saying that MySQL won't allow to create a TEXT/BLOB index withouth the length:
InternalError: (pymysql.err.InternalError) (1170, "BLOB/TEXT column 'index' used in key specification without a key length") [SQL: 'CREATE INDEX ix_rnd_vals_index ON rnd_vals (`index`)']
I believed that I could fix this, by specifying the dtype option on the to_sql() function, but it didn't help.
I found a way of making this, by joining two DataFrames, one with the values, and the other one with the index:
from sqlalchemy.types import VARCHAR
import pandas as pd
import numpy as np
from sqlalchemy import create_engine
frame = pd.DataFrame(np.random.random(25).reshape(5,5),
columns=['Jan','Feb','Mar','Apr','May'])
idxFrame = pd.DataFrame({'index':['exp1','exp2','exp3','exp4','exp5']})
frame = frame.join(idxFrame)
frame=frame.set_index('index')
engine = create_engine('mysql+pymysql://user:password#localhost/python_samples')
frame.to_sql('indexes',engine,if_exists='replace', index_label='index',
dtype={'index':VARCHAR(5)})
This works as expected, but I really doubt that this is the correct way of making this, can someone help me? What did I did wrong?
Thank you
For whoever is having this issue, Ilja Everilä on the comments solved the issue. The index name was actually 'None', instead of 'index', so when I changed the dtype from
dtype={'index':VARCHAR(5)}
to
dtype={'None':VARCHAR(5)}
It solved the issue, and the table was created on MySQL as:
CREATE TABLE `rnd_vals` (
`index` text,
`col1` double DEFAULT NULL,
`col2` double DEFAULT NULL,
`col3` double DEFAULT NULL,
`col4` double DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8
as expected.
Thank you all!
I tried to find a direct way to enable pandas to directly import the index. In the end reset_index() seems the simplest method:
my_df.reset_index()
my_df.to_sql(name='my_table', con=engine, index=False, if_exists='replace')
With :
frame.to_sql('rnd_vals', engine, dtype={'None':VARCHAR(5)})
It was giving :
1170, "BLOB/TEXT column 'index' used in key specification without a key length") [SQL: 'CREATE INDEX ix_indexes_index ON indexes (index)'] (Background on this error at: http://sqlalche.me/e/e3q8)
This resolved the issue :
frame.to_sql('indexes',engine,if_exists='replace', index_label='index',dtype={frame.index.name:VARCHAR(5)})
You are trying to make an index from a column of type text/blob. In this case, MySQL unable to put uniqueness to the columns because of dynamic nature. There is no length associated with this. You can provide the type of column while saving dataframe to MySQL or (if you don't need an index) just make index=False.
Use VARCHAR(...) instead of TEXT whenever practical.
In general, it is not useful to index TEXT columns.
I can't provide the sqlalchemy; I am not familiar with how it obfuscates the SQL code.
I have a bunch of MySQL tables I work with where the ultimate data source from a very slow SQL server administered by someone else. My predecessors' solution to dealing with this is to do queries more-or-less like:
results = python_wrapper('SELECT primary_key, col2, col3 FROM foreign_table;')
other_python_wrapper('DELETE FROM local_table;')
other_python_wrapper('INSERT INTO local_table VALUES() %s;' % results)
The problem is this means you can never use values in local_table as foreign key constraints for other tables because they are constantly being deleted and added back into the table whenever you update it from the foreign source. However, if a record really does dis sapper in the results to the query on the foreign server, than that usually means you would want to trigger a cascade effect to drop records in other local tables that you've linked with a foreign key constraint to data propagated from the foreign table.
The only semi-reasonable solution I've come up with is to do something like:
results = python_wrapper('SELECT primary_key, col2, col3 FROM foreign_table;')
other_python_wrapper('DELETE FROM local_table_temp;')
other_python_wrapper('INSERT INTO local_table_temp VALUES() %s;' % results)
other_python_wrapper('DELETE FROM local_table WHERE primary_key NOT IN local_table_temp;')
other_python_wrapper('INSERT INTO local_table SELECT * FROM local_table_temp ON DUPLICATE KEY UPDATE local_table.col2 = local_table_temp.col2, local_table.col3 = local_table_temp.col3
The problem is there's a fair number of these tables and many of the tables have a large number of columns that need to be updated so it's tedious to write the same boiler-plate over & over. And if the table schema changes, there's more than one place you need to update the listing of all columns.
Is there any more concise way to do this with the SQL code?
Thanks!
I have a somewhat un-satisfactory answer to my own question. Since I'm using python to query the foreign Oracle database and put that into SQL, and I trust the format of the table and column names to be pretty well behaved, I can just wrap the whole procedure in python code and have python generate the update SQL update queries based off inspecting the tables.
For a number of reasons, I'd still like to see a better way to do this, but it works for me because:
I'm using an external scripting language that can inspect the database schema anyway.
I trust the database, column, and table names I'm working with to be well-behaved because these are all things I have direct control over.
My solution depends on the local SQL table structure; specifically which keys are primary keys. The code won't work without properly structured tables. But that's OK, because I can restructure the MySQL tables to make my python code work.
While I do hope someone else can think up a more-elegant and/or general-purpose solution, I will offer up my own python code to anyone who is working on a similar problem who can safely make the same assumptions I did above.
Below is a python wrapper I use to do simple SQL queries in python:
import config, MySQLdb
class SimpleSQLConn(SimpleConn):
'''simplified wrapper around a MySQLdb.connection'''
def __init__(self, **kwargs):
self._connection = MySQLdb.connect(host=config.mysql_host,
user=config.mysql_user,
passwd=config.mysql_pass,
**kwargs)
self._cursor = self._connection.cursor()
def query(self, query_str):
self._cursor.execute(query_str)
self._connection.commit()
return self._cursor.fetchall()
def columns(self, database, table):
return [x[0] for x in self.query('DESCRIBE `%s`.`%s`' % (database, table))g]
def primary_keys(self, database, table):
return [x[0] for x in self.query('DESCRIBE `%s`.`%s`' % (database, table)) if 'PRI' in x]
And here is the actual update function, using the SQL wrapper class above:
def update_table(database,
table,
mysql_insert_with_dbtable_placeholder):
'''update a mysql table without first deleting all the old records
mysql_insert_with_dbtable_placeholder should be set to a string with
placeholders for database and table, something like:
mysql_insert_with_dbtable_placeholder = "
INSERT INTO `%(database)s`.`%(table)s` VALUES (a, b, c);
note: code as is will update all the non-primary keys, structure
your tables accordingly
'''
sql = SimpleSQLConn()
query ='DROP TABLE IF EXISTS `%(database)s`.`%(table)s_temp_for_update`' %\
{'database': database, 'table': table}
sql.query(query)
query ='CREATE TABLE `%(database)s`.`%(table)s_temp_for_update` LIKE `%(database)s`.`%(table)s`'%\
{'database': database, 'table': table}
sql.query(query)
query = mysql_insert_with_dbtable_placeholder %\
{'database': database, 'table': '%s_temp_for_update' % table}
sql.query(query)
query = '''DELETE FROM `%(database)s`.`%(table)s` WHERE
(%(primary_keys)s) NOT IN
(SELECT %(primary_keys)s FROM `%(database)s`.`%(table)s_temp_for_update`);
''' % {'database': database,
'table': table,
'primary_keys': ', '.join(['`%s`' % key for key in sql.primary_keys(database, table)])}
sql.query(query)
update_columns = [col for col in sql.columns(database, table)
if col not in sql.primary_keys(database, table)]
query = '''INSERT into `%(database)s`.`%(table)s`
SELECT * FROM `%(database)s`.`%(table)s_temp_for_update`
ON DUPLICATE KEY UPDATE
%(update_cols)s
''' % {'database': database,
'table': table,
'update_cols' : ',\n'.join(['`%(table)s`.`%(col)s` = `%(table)s_temp_for_update`.`%(col)s`' \
% {'table': table, 'col': col} for col in update_columns])}
sql.query(query)
I've recently picked up Slick and Scala, and realised my database tables created by Slick where using MYISAM, seems this is the default engine on my MySQL config and Slick create table statements do not specify engine. That's all good but I wanted to change to InnoDB without changing my default table engine in MySQL. Slick documentation has nothing on the matter. Is there anyway to change table engine with Slick (before tables creation) without modying default engine in MySQL my.ini ?
You can define the engine or otherwise extend the query by manipulating the statements:
val statements: Iterator[String] = myTableQuery.schema.createStatements
val action: SqlAction[Int, NoStream, Effect] =
sqlu"#${createStatement.mkString(" ")} engine=InnoDB"
db.run(createAction)
Take care to use the # verbatim interpolation prefix for the sqlu interpolator - without it, the statements will be escaped, leading to a syntax error.
see related docs: http://slick.lightbend.com/doc/3.2.3/schemas.html#data-definition-language
EDIT:
I have noticed that the approach does not work if the DDL generates multiple statements, like additional alter table.... In this case, there is an even less elegant solution:
def createTable(ddl: profile.DDL, db: Database)(implicit ec: ExecutionContext): Future[Int] = {
val mapped = ddl.createStatements.toList map (
s => if (s.startsWith("create table")) s"$s engine=InnoDB" else s
)
mapped.foldLeft(Future.successful(1)) {
case (acc: Future[Int], s: String) =>
acc.flatMap(_ => db.run(sqlu"#$s"))
}
}
You cannot use traverse or map to create the futures, because alter table depends on the table being already present. Use flatMap for sequential execution.