I have a database in Postgres that I would like to migrate to MySQL, but whenever I get to the reverse engineering stage of the MySQL Workbench Migration Wizard I get an error. The error is:
ERROR: Reverse engineer selected schemas: ProgrammingError("('42703', '[42703] ERROR: column "min_value" does not exist;\nError while executing the query (1) (SQLExecDirectW)')"): error calling Python module function DbPostgresqlRE.reverseEngineer
Failed
The full error log is:
File "C:\Program Files\MySQL\MySQL Workbench 6.3 CE\modules\db_generic_re_grt.py", line 237, in reverseEngineer
catalog = cls.reverseEngineerCatalog(connection, catalog_name)
File "C:\Program Files\MySQL\MySQL Workbench 6.3 CE\modules\db_generic_re_grt.py", line 397, in reverseEngineerCatalog
cls.reverseEngineerSequences(connection, schema)
File "C:\Program Files\MySQL\MySQL Workbench 6.3 CE\modules\db_postgresql_re_grt.py", line 76, in reverseEngineerSequences
min_value, max_value, start_value, increment_by, last_value, is_cycled, ncache = cls.execute_query(connection, seq_details_query % (schema.name, seq_name)).fetchone()
File "C:\Program Files\MySQL\MySQL Workbench 6.3 CE\modules\db_generic_re_grt.py", line 76, in execute_query
return cls.get_connection(connection_object).cursor().execute(query, *args, **kwargs)
pyodbc.ProgrammingError: ('42703', '[42703] ERROR: column "min_value" does not exist;\nError while executing the query (1) (SQLExecDirectW)')
Traceback (most recent call last):
File "C:\Program Files\MySQL\MySQL Workbench 6.3 CE\workbench\wizard_progress_page_widget.py", line 192, in thread_work
self.func()
File "C:\Program Files\MySQL\MySQL Workbench 6.3 CE\modules\migration_schema_selection.py", line 175, in task_reveng
self.main.plan.migrationSource.reverseEngineer()
File "C:\Program Files\MySQL\MySQL Workbench 6.3 CE\modules\migration.py", line 369, in reverseEngineer
self.state.sourceCatalog = self._rev_eng_module.reverseEngineer(self.connection, self.selectedCatalogName, self.selectedSchemataNames, self.state.applicationData)
SystemError: ProgrammingError("('42703', '[42703] ERROR: column "min_value" does not exist;\nError while executing the query (1) (SQLExecDirectW)')"): error calling Python module function DbPostgresqlRE.reverseEngineer
ERROR: Reverse engineer selected schemas: ProgrammingError("('42703', '[42703] ERROR: column "min_value" does not exist;\nError while executing the query (1) (SQLExecDirectW)')"): error calling Python module function DbPostgresqlRE.reverseEngineer
Failed
I tried modifying the Python script as described in the top comment here, but that did not stop the error. Does anyone know how I could resolve this?
Fix File "C:\Program Files\MySQL\MySQL Workbench 6.3 CE\modules\db_postgresql_re_grt.py" lines ~70 change to
seq_details_query = """SELECT min_value, max_value, start_value,
increment_by, last_value, cycle as is_cycled, cache_size as cache_value
FROM pg_catalog.pg_sequences
WHERE schemaname = '%s' AND sequencename = '%s' """
I have PostgreSQL 10.5
Replaced in db_postgresql_re_grt.py:80
min_value, max_value, start_value, increment_by, last_value, is_cycled, ncache = cls.execute_query(connection, seq_details_query % (schema.name, seq_name)).fetchone()
with:
last_value, log_cnt, is_called = cls.execute_query(connection, """SELECT last_value, log_cnt, is_called FROM "%s"."%s" """ % (schema.name, seq_name)).fetchone()
seqrelid, seqtypid, seqstart, seqincrement, seqmax, seqmin, seqcache, seqcycle = cls.execute_query(connection, "select seqrelid, seqtypid, seqstart, seqincrement, seqmax, seqmin, seqcache, seqcycle from pg_sequence where seqrelid = '%s'::regclass" % seq_name).fetchone()
min_value = seqmin
max_value = seqmax
start_value = seqstart
increment_by = seqincrement
is_cycled = seqcycle
ncache = seqcache
Related
I am having problems using a MySQL database as a metadata database in a TensorFlow Extended pipeline.
I used the penguin template to set up a very simple pipeline and also set up a MySQL database locally.
The only thing I changed in the code was using
tfx.orchestration.metadata.mysql_metadata_connection_config as the metadata_connection_config input to the pipeline instead of tfx.orchestration.metadata.sqlite_metadata_connection_config
from tfx import v1 as tfx
# [...] Add a simple CsvExampleGen
tfx.dsl.Pipeline(
pipeline_name=pipeline_name,
pipeline_root=pipeline_root,
components=components,
metadata_connection_config=tfx.orchestration.metadata
.mysql_metadata_connection_config(
host="localhost",
port=3306,
database="ml_metadata",
username="root",
password="password"),
beam_pipeline_args=beam_pipeline_args,
)
Running this code results in the following error message:
[2021-11-17 12:02:16,948] {taskinstance.py:1270} INFO - Marking task as FAILED. dag_id=penguin_pipeline, task_id=CsvExampleGen, execution_date=20211117T110208, start_date=20211117T110212, end_date=20211117T110216
[2021-11-17 12:02:16,966] {standard_task_runner.py:88} ERROR - Failed to execute job 101 for task CsvExampleGen
Traceback (most recent call last):
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
args.func(args, dag=self.dag)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
_run_raw_task(args, ti)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 180, in _run_raw_task
ti._run_raw_task(
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
result = execute_callable(context=context)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/operators/python.py", line 151, in execute
return_value = self.execute_callable()
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/operators/python.py", line 162, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/user/anaconda3/lib/python3.8/site-packages/tfx/orchestration/airflow/airflow_component.py", line 76, in _airflow_component_launcher
launcher.launch()
File "/home/user/anaconda3/lib/python3.8/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 191, in launch
execution_decision = self._run_driver(self._input_dict, self._output_dict,
File "/home/user/anaconda3/lib/python3.8/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 152, in _run_driver
with self._metadata_connection as m:
File "/home/user/anaconda3/lib/python3.8/site-packages/tfx/orchestration/metadata.py", line 152, in __enter__
raise RuntimeError(
RuntimeError: Failed to establish connection to Metadata storage with error: mysql_real_connect failed: errno: , error:
In this log I was using AirFlow, but the same exception message is shown if I run a LocalDagRunner:
Exception has occurred: RuntimeError
Failed to establish connection to Metadata storage with error: mysql_real_connect failed: errno: , error:
I have tried changing the host to "127.0.0.1", but this didn't change anything. Anyone had a similar issue or maybe see an apparent error in my approach?
The MySQL server is version 8.0 and I am using TensorFlow Extended version 1.4.0
i have a database and i give it name "test" then
i try to change the database name using python code, but when i run the program the result give me an error warning. here is my code :
def changedb_name():
import mysql.connector
mydb = mysql.connector.connect(host = 'localhost', user='root', password ='')
mycursor = mydb.cursor()
mycursor.execute('ALTER DATABASE test RENAME to test1')
changedb_name()
here is the error warning :
Traceback (most recent call last):
File "c:\Users\NBUSER\Documents\ARTOFWAR\PITON\02-piton\aldo\chgdbname.py", line 18, in <module>
viewdb1()
File "c:\Users\NBUSER\Documents\ARTOFWAR\PITON\02-piton\aldo\chgdbname.py", line 15, in viewdb1
mycursor.execute('RENAME DATABASE test to test1')
File "C:\Users\NBUSER\AppData\Local\Programs\Python\Python39\lib\site-packages\mysql\connector\cursor.py", line 568, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\NBUSER\AppData\Local\Programs\Python\Python39\lib\site-packages\mysql\connector\connection.py", line 846, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
ion.py", line 656, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'DATABASE test to test1' at line 1
PS C:\Users\NBUSER\Documents\ARTOFWAR\PITON\02-piton> & C:/Users/NBUSER/AppData/Local/Programs/Python/Python39/python.exe c:/Users/NBUSER/Documents/ARTOFWAR/PITON/02-piton/aldo/chgdbname.py
Traceback (most recent call last):
File "c:\Users\NBUSER\Documents\ARTOFWAR\PITON\02-piton\aldo\chgdbname.py", line 18, in <module>
viewdb1()
File "c:\Users\NBUSER\Documents\ARTOFWAR\PITON\02-piton\aldo\chgdbname.py", line 15, in viewdb1
mycursor.execute('ALTER DATABASE test RENAME to test1')
File "C:\Users\NBUSER\AppData\Local\Programs\Python\Python39\lib\site-packages\mysql\connector\cursor.py", line 568, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\NBUSER\AppData\Local\Programs\Python\Python39\lib\site-packages\mysql\connector\connection.py", line 846, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\NBUSER\AppData\Local\Programs\Python\Python39\lib\site-packages\mysql\connector\connection.py", line 656, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'RENAME to test1' at line 1
From where did you find the SQL syntax to ALTER DATABASE...RENAME... ? It seems to me your reference is flawed; that isn't any MySQL syntax I'm familiar with. I believe the ability briefly existed for the blink of an eye. A bit of an internet history search confirms that it hasn't been part of MySQL (pdf) syntax in over 10 years (hat tip to #zloctb).
You can't rename a database in pure SQL. The usual workaround is to export to SQL and re-import to the new database.
When i run pwiz to generate a peewee modem from existing database it shows following error:
root#server:~# python -m pwiz -e mysql -P -H 127.0.0.1 mysql
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/pwiz.py", line 202, in <module>
print_models(introspector, tables, preserve_order=options.preserve_order)
File "/usr/local/lib/python2.7/dist-packages/pwiz.py", line 47, in print_models
database = introspector.introspect(table_names=tables)
File "/usr/local/lib/python2.7/dist-packages/playhouse/reflection.py", line 440, in introspect
tables = self.metadata.database.get_tables()
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3089, in get_tables
return [table for table, in self.execute_sql('SHOW TABLES')]
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2459, in execute_sql
cursor = self.cursor(commit)
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2445, in cursor
self.connect()
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2411, in connect
self._state.set_connection(self._connect())
File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3083, in _connect
return mysql.connect(db=self.database, **self.connect_params)
AttributeError: 'NoneType' object has no attribute 'connect'
What is the problem? The username and password of database is correct.
PyMySql is not installed. Install it using pip or other available options:
pip install PyMySql
Unfortunately, peewee does not write `PyMySql1 module of python in its dependency list, so it should be installed separately.
I have a bash script that starts some python code, running on a RasPi under latest raspbian. It runs fine if I run it manually with sudo, but when it auto runs at boot I get a mysql error. The script is called from a line in /etc/rc.local
The script is
#!/bin/bash
/home/control/solar/v10_1/control.py >> '/home/control/solar/logs/control_X.X_start.log' 2>&1 &
echo "control started" >> '/home/control/solar/logs/control_X.X_start.log'
echo "UID is $UID , EUID is $EUID" >> '/home/control/solar/logs/control_X.X_start.log'
The output to the log after bootup is
UID is 0 , EUID is 0
Traceback (most recent call last):
File "/home/control/solar/v10_1/control.py", line 42, in <module>
import variables # global variables
File "/home/control/solar/v10_1/variables.py", line 13, in <module>
gv.db = MySQLdb.connect("localhost", "solar", "solar", db='solar') # database for logging
File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")
I have made all the paths absolute, the UID is 0 in both cases, what can cause the difference in behaviour?
My legacy mysql 4.0.20 server is on a windows machine.
I'm developing a new system (python based) on a linux that needs to connect to the legacy server and make queries etc.
I have successfully connected using both plain MySQLdb and django. I'm having trouble connecting using sqlalchemy. Here is the code:
conn_str = "mysql://user:pass#192.168.171.233/dbd"
engine = create_engine(conn_str, echo=True)
metadata = MetaData(engine)
connection = engine.connect()
and the error stack I'm getting:
2011-03-01 08:35:04,613 INFO sqlalchemy.engine.base.Engine.0x...b42c SELECT DATABASE()
2011-03-01 08:35:04,613 INFO sqlalchemy.engine.base.Engine.0x...b42c ()
Traceback (most recent call last):
connection = engine.connect()
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line 1811, in connect
return self.Connection(self, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line 832, in init
self.connection = connection or engine.raw_connection()
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line 1874, in raw_connection
return self.pool.unique_connection()
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 142, in unique_connection
return _ConnectionFairy(self).checkout()
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 369, in __init
rec = self._connection_record = pool.get()
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 213, in get
return self.do_get()
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 732, in do_get
con = self.create_connection()
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 147, in create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 258, in init
l.first_connect(self.connection, self)
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/strategies.py", line 151, in first_connect
dialect.initialize(c)
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/dialects/mysql/base.py", line 1753, in initialize
default.DefaultDialect.initialize(self, connection)
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/default.py", line 159, in initialize
self.returns_unicode_strings = self._check_unicode_returns(connection)
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/default.py", line 205, in _check_unicode_returns
unicode_for_varchar = check_unicode(sqltypes.VARCHAR(60))
File "/usr/local/lib/python2.6/dist-packages/sqlalchemy/engine/default.py", line 195, in check_unicode
]).compile(dialect=self)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.3c1-py2.6-linux-i686.egg/MySQLdb/cursors.py", line 173, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.3c1-py2.6-linux-i686.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
ProgrammingError: (1064, "You have an error in your SQL syntax. Check the manual that corresponds to your MySQL server version for the right syntax to use near '(60)) AS anon_1' at line 1")
Looks like MySQL is reporting a syntax error in one of your SQL statements:
<snip>
1064, "You have an error in your SQL syntax.
Check the manual that corresponds to your MySQL server
version for the right syntax to use near
'(60)) AS anon_1' at line 1
</snip>
It looks like MySQL is objecting to a statement that includes the string:
(60)) AS anon_1
NOTE: I am not familiar with Sqlalchemy, but it looks to me like the exception (the SQL error from the statement) is getting propagated up to the handler specified in the connection.
FOLLOWUP: It's likely that a query that Sqlalchemy is executing is not compatible with the older version of MySQL. Looks like it's in the "_check_unicode_returns" function.
Have you tried disabling unicode support?
conn_str = "mysql://user:pass#192.168.171.233/dbd?use_unicode=0"