I am having problems using a MySQL database as a metadata database in a TensorFlow Extended pipeline.
I used the penguin template to set up a very simple pipeline and also set up a MySQL database locally.
The only thing I changed in the code was using
tfx.orchestration.metadata.mysql_metadata_connection_config as the metadata_connection_config input to the pipeline instead of tfx.orchestration.metadata.sqlite_metadata_connection_config
from tfx import v1 as tfx
# [...] Add a simple CsvExampleGen
tfx.dsl.Pipeline(
pipeline_name=pipeline_name,
pipeline_root=pipeline_root,
components=components,
metadata_connection_config=tfx.orchestration.metadata
.mysql_metadata_connection_config(
host="localhost",
port=3306,
database="ml_metadata",
username="root",
password="password"),
beam_pipeline_args=beam_pipeline_args,
)
Running this code results in the following error message:
[2021-11-17 12:02:16,948] {taskinstance.py:1270} INFO - Marking task as FAILED. dag_id=penguin_pipeline, task_id=CsvExampleGen, execution_date=20211117T110208, start_date=20211117T110212, end_date=20211117T110216
[2021-11-17 12:02:16,966] {standard_task_runner.py:88} ERROR - Failed to execute job 101 for task CsvExampleGen
Traceback (most recent call last):
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
args.func(args, dag=self.dag)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
_run_raw_task(args, ti)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 180, in _run_raw_task
ti._run_raw_task(
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
result = execute_callable(context=context)
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/operators/python.py", line 151, in execute
return_value = self.execute_callable()
File "/home/user/anaconda3/lib/python3.8/site-packages/airflow/operators/python.py", line 162, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/user/anaconda3/lib/python3.8/site-packages/tfx/orchestration/airflow/airflow_component.py", line 76, in _airflow_component_launcher
launcher.launch()
File "/home/user/anaconda3/lib/python3.8/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 191, in launch
execution_decision = self._run_driver(self._input_dict, self._output_dict,
File "/home/user/anaconda3/lib/python3.8/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 152, in _run_driver
with self._metadata_connection as m:
File "/home/user/anaconda3/lib/python3.8/site-packages/tfx/orchestration/metadata.py", line 152, in __enter__
raise RuntimeError(
RuntimeError: Failed to establish connection to Metadata storage with error: mysql_real_connect failed: errno: , error:
In this log I was using AirFlow, but the same exception message is shown if I run a LocalDagRunner:
Exception has occurred: RuntimeError
Failed to establish connection to Metadata storage with error: mysql_real_connect failed: errno: , error:
I have tried changing the host to "127.0.0.1", but this didn't change anything. Anyone had a similar issue or maybe see an apparent error in my approach?
The MySQL server is version 8.0 and I am using TensorFlow Extended version 1.4.0
Related
I'm using peewee for a Python ORM to a MySQL database and Flask to serve a web site from my local desktop running Ubuntu. I can connect to the database from the command line (mysql test -u user -p), but am getting a connection refused error when trying from Python. This code has worked on my previous Ubuntu PC, and I dumped the database from there, imported into the new PC and granted permissions, so I'm really not sure why it's giving me this.
Model.py:
from peewee import MySQLDatabase, Model, AutoField, CharField, IntegerField, \
ForeignKeyField, BooleanField, TextField, DateTimeField
from flask_security import UserMixin, RoleMixin
class BaseModel(Model):
"""
Base model for the model classes; prevents having to define Meta class with the
database for each Model class. Must call BaseModel._meta.database.init() before
connecting - should be able to just call once on BaseModel before
using derived model classes.
"""
class Meta:
database = MySQLDatabase(None) #init before calling connect()
#Additional code to define various tables
Trying to connect in Python command line:
import Model as mmm
db = mmm.BaseModel._meta.database
db.init('test', host = 'localhost', user = 'mike', passwd = password, port = 3306)
db.connect()
Error stack trace:
Traceback (most recent call last):
File "/home/mike/.local/lib/python3.8/site-packages/pymysql/connections.py", line 569, in connect
sock = socket.create_connection(
File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mike/.local/lib/python3.8/site-packages/peewee.py", line 3035, in connect
self._state.set_connection(self._connect())
File "/home/mike/.local/lib/python3.8/site-packages/peewee.py", line 3933, in _connect
conn = mysql.connect(db=self.database, **self.connect_params)
File "/home/mike/.local/lib/python3.8/site-packages/pymysql/__init__.py", line 94, in Connect
return Connection(*args, **kwargs)
File "/home/mike/.local/lib/python3.8/site-packages/pymysql/connections.py", line 327, in __init__
self.connect()
File "/home/mike/.local/lib/python3.8/site-packages/pymysql/connections.py", line 619, in connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'aspire' ([Errno 111] Connection refused)")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/mike/.local/lib/python3.8/site-packages/peewee.py", line 3038, in connect
self._initialize_connection(self._state.conn)
File "/home/mike/.local/lib/python3.8/site-packages/peewee.py", line 2873, in __exit__
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/home/mike/.local/lib/python3.8/site-packages/peewee.py", line 183, in reraise
raise value.with_traceback(tb)
File "/home/mike/.local/lib/python3.8/site-packages/peewee.py", line 3035, in connect
self._state.set_connection(self._connect())
File "/home/mike/.local/lib/python3.8/site-packages/peewee.py", line 3933, in _connect
conn = mysql.connect(db=self.database, **self.connect_params)
File "/home/mike/.local/lib/python3.8/site-packages/pymysql/__init__.py", line 94, in Connect
return Connection(*args, **kwargs)
File "/home/mike/.local/lib/python3.8/site-packages/pymysql/connections.py", line 327, in __init__
self.connect()
File "/home/mike/.local/lib/python3.8/site-packages/pymysql/connections.py", line 619, in connect
raise exc
peewee.OperationalError: (2003, "Can't connect to MySQL server on 'aspire' ([Errno 111] Connection refused)")
Hi I'm new to the PYTHON and MYSQL. I'm trying to update the data base using the python where I'm taking user input and trying to update into the database.
Can any buddy help me in this error.
Traceback (most recent call last):
File "C:\Users\YGIN\Anaconda3\envs\MyTest\lib\site-packages\mysql\connector\network.py", line 148, in send_plain
self.sock.sendall(packet)
File "C:\Users\YGIN\Anaconda3\envs\MyTest\lib\ssl.py", line 1037, in sendall
return super().sendall(data, flags)
OSError: [WinError 10038] An operation was attempted on something that is not a socket
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/YGIN/PycharmProjects/MyTest/BankApplication.py", line 181, in <module>
c1.customer_closing_account()
File "C:/Users/YGIN/PycharmProjects/MyTest/BankApplication.py", line 146, in customer_closing_account
dbq.delete_cust_account(self.account_no)
File "C:\Users\YGIN\PycharmProjects\MyTest\DB_queryies.py", line 71, in delete_cust_account
my_cursor.execute(mysqlinsertquery, recordstodel)
File "C:\Users\YGIN\Anaconda3\envs\MyTest\lib\site-packages\mysql\connector\cursor.py", line 569, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\YGIN\Anaconda3\envs\MyTest\lib\site-packages\mysql\connector\connection.py", line 590, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\YGIN\Anaconda3\envs\MyTest\lib\site-packages\mysql\connector\connection.py", line 344, in _send_cmd
packet_number, compressed_packet_number)
File "C:\Users\YGIN\Anaconda3\envs\MyTest\lib\site-packages\mysql\connector\network.py", line 151, in send_plain
errno=2055, values=(self.get_address(), _strioerror(err)))
mysql.connector.errors.OperationalError: 2055: Lost connection to MySQL server at 'localhost:3306', system error: 10038 An operation was attempted on something that is not a socket
I am using Django 1.11 for making an app 'cnfs', and I am using MYSQL database with it. I am constantly facing this issue where I am getting an error like this when I type the following code:
$python manage.py migrate
System check identified some issues: WARNINGS: ?: (mysql.W002) MySQL Strict Mode is not set for database connection 'default' HINT: MySQL's Strict Mode fixes many data integrity problems in MySQL, such as data truncation upon insertion, by escalating warnings into errors. It is strongly recommended you activate it. See: https://docs.djangoproject.com/en/1.11/ref/databases/#mysql-sql-mode Operations to perform: Apply all migrations: admin, auth, cnfs, contenttypes, sites Running migrations: No migrations to apply. Traceback (most recent call last): File "manage.py", line 22, in execute_from_command_line(sys.argv) File "/home/ubuntu/.local/lib/python2.7/site-packages/django/core/management/init.py", line 363, in execute_from_command_line utility.execute() File "/home/ubuntu/.local/lib/python2.7/site-packages/django/core/management/init.py", line 355, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/ubuntu/.local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(*args, **cmd_options) File "/home/ubuntu/.local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute output = self.handle(*args, **options) File "/home/ubuntu/.local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 227, in handle self.verbosity, self.interactive, connection.alias, apps=post_migrate_apps, plan=plan, File "/home/ubuntu/.local/lib/python2.7/site-packages/django/core/management/sql.py", line 53, in emit_post_migrate_signal **kwargs File "/home/ubuntu/.local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in send for receiver in self._live_receivers(sender) File "/home/ubuntu/.local/lib/python2.7/site-packages/django/contrib/sites/management.py", line 20, in create_default_site if not Site.objects.using(using).exists(): File "/home/ubuntu/.local/lib/python2.7/site-packages/django/db/models/query.py", line 670, in exists return self.query.has_results(using=self.db) File "/home/ubuntu/.local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 517, in has_results return compiler.has_results() File "/home/ubuntu/.local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 845, in has_results return bool(self.execute_sql(SINGLE)) File "/home/ubuntu/.local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 886, in execute_sql raise original_exception django.db.utils.ProgrammingError: (1146, "Table 'cnfs.django_site' doesn't exist")
I have literally spent days on this, I did not face this issue when i was using the default django database engine, sqlite3.
I have tried doing this:
$python manage.py migrate sites
My SITE_ID setting is set to 1.
I tried removing the 'django.contrib.sites' thing from the INSTALLED_APPS setting, but that throws this error:
Internal Server Error: / Traceback (most recent call last): File "/home/ubuntu/.local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 41, in inner response = get_response(request) File "/home/ubuntu/.local/lib/python2.7/site-packages/django/utils/deprecation.py", line 138, in call response = self.process_request(request) File "/home/ubuntu/.local/lib/python2.7/site-packages/subdomains/middleware.py", line 62, in process_request super(SubdomainURLRoutingMiddleware, self).process_request(request) File "/home/ubuntu/.local/lib/python2.7/site-packages/subdomains/middleware.py", line 38, in process_request (self.get_domain_for_request(request), request.get_host())) File "/home/ubuntu/.local/lib/python2.7/site-packages/subdomains/middleware.py", line 31, in get_domain_for_request return get_domain() File "/home/ubuntu/.local/lib/python2.7/site-packages/subdomains/utils.py", line 12, in current_site_domain from django.contrib.sites.models import Site File "/home/ubuntu/.local/lib/python2.7/site-packages/django/contrib/sites/models.py", line 84, in class Site(models.Model): File "/home/ubuntu/.local/lib/python2.7/site-packages/django/db/models/base.py", line 118, in new "INSTALLED_APPS." % (module, name) RuntimeError: Model class django.contrib.sites.models.Site doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.
Any suggestions would be much appreciated.
Thanks.
uncomment the django.contrib.sites from installed apps
then
python manage.py migrate sites
python manage.py migrate
if you have few table in the database, drop the tables and then run this
I run into such problems when I running my Django project(a week ago, the project work properly, today find this problem):
My Django version is 1.10.2 with python version 3.5.2, MySQL version is 5.5 on ubuntu 14.0.
/Users/deja/Virtualenv/python3.5/bin/python3.5 /Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py --multiproc --qt-support --client 127.0.0.1 --port 60850 --file /Users/mozat/project/crawler_management_system/manage.py runserver 0.0.0.0:6380
pydev debugger: process 81527 is connecting
Connected to pydev debugger (build 145.1504)
pydev debugger: process 81528 is connecting
Performing system checks...
System check identified no issues (0 silenced).
You have 13 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
January 03, 2017 - 07:55:47
Django version 1.10.2, using settings 'crawler_management_system.settings'
Starting development server at http://0.0.0.0:6380/
Quit the server with CONTROL-C.
Internal Server Error: /
Traceback (most recent call last):
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/network.py", line 226, in recv_plain
chunk = self.sock.recv(4 - packet_len)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/mozat/project/crawler_management_system/crawler_management_system/mysql_utility.py", line 32, in open_db
raise err
File "/Users/mozat/project/crawler_management_system/crawler_management_system/mysql_utility.py", line 28, in open_db
yield cursor
File "/Users/mozat/project/crawler_management_system/crawler_management_system/views.py", line 40, in select_batch_records
cursor.execute(sql.format(table=self.table,times=times))
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/cursor.py", line 515, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 488, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 267, in _send_cmd
return self._socket.recv()
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/network.py", line 255, in recv_plain
errno=2055, values=(self.get_address(), _strioerror(err)))
mysql.connector.errors.OperationalError: 2055: Lost connection to MySQL server at '127.0.0.1:10189', system error: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 710, in reset_session
self.cmd_reset_connection()
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 1046, in cmd_reset_connection
raise errors.NotSupportedError("MySQL version 5.7.2 and "
mysql.connector.errors.NotSupportedError: MySQL version 5.7.2 and earlier does not support COM_RESET_CONNECTION.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/protocol.py", line 267, in parse_eof
unpacked = struct_unpack('<xxxBBHH', packet)
struct.error: unpack requires a bytes object of length 9
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/django/core/handlers/exception.py", line 39, in inner
response = get_response(request)
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
response = self._get_response(request)
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/mozat/project/crawler_management_system/crawler_management_system/views.py", line 99, in mainpage
result = batch_record_repo.select_batch_records(times)
File "/Users/mozat/project/crawler_management_system/crawler_management_system/views.py", line 41, in select_batch_records
return cursor.fetchall()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/mozat/project/crawler_management_system/crawler_management_system/mysql_utility.py", line 40, in open_db
connection.close()
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/pooling.py", line 117, in close
cnx.reset_session()
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 713, in reset_session
self._database, self._charset_id)
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 661, in cmd_change_user
self._post_connection()
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/abstracts.py", line 695, in _post_connection
self.set_charset_collation(self._charset_id)
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/abstracts.py", line 654, in set_charset_collation
charset_name, collation_name))
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 869, in _execute_query
self.cmd_query(query)
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 488, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 393, in _handle_result
return self._handle_eof(packet)
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/connection.py", line 344, in _handle_eof
eof = self._protocol.parse_eof(packet)
File "/Users/deja/Virtualenv/python3.5/lib/python3.5/site-packages/mysql/connector/protocol.py", line 269, in parse_eof
raise errors.InterfaceError(err_msg)
mysql.connector.errors.InterfaceError: Failed parsing EOF packet.
[03/Jan/2017 07:56:40] "GET / HTTP/1.1" 500 172448
anyone happen to see same problem?Could you please give me some suggestion.
all ,nothing wrong with mysql, and Django. It's for I use a complex SQL to access mysql:
SELECT *
FROM
(SELECT *,
#batch_rank := IF(#current_batch = spider , #batch_rank + 1, 1) AS batch_rank,
#current_batch := spider
FROM deja_crawler.crawl_batch_record
where spider in (select distinct spider from deja_crawler.crawl_batch_record where spider like '%_new' or spider like '%_update')
ORDER BY spider, create_time DESC)
ranked
WHERE batch_rank <= 3
the tables used to have not too much data, so it works, later, the table became huge, and the access time cost is almost 30s, but the connection time out is set as 20s, so it will raise time out error.
I have an issue with my deployed application on an apache2 with mod_wsgi.
The connection process to the database is exactly successful only once out of 2!
I've named the database 'CKP' and for some reason have to change to 'ckp'.
So when I request a page of the application once out of two, I can read in the error log of apache:
mod_wsgi (pid=16098): Target WSGI script '/u1/msc/vg55/public_html/core_knowledge_web_platform/wsgi.py' cannot be loaded as Python module.
mod_wsgi (pid=16098): Exception occurred processing WSGI script '/u1/msc/vg55/public_html/core_knowledge_web_platform/wsgi.py'.
Traceback (most recent call last):
File "/u1/msc/vg55/public_html/core_knowledge_web_platform/wsgi.py", line 20, in <module>
command.validate()
File "/usr/lib/python2.6/site-packages/django/core/management/base.py", line 249, in validate
num_errors = get_validation_errors(s, app)
File "/usr/lib/python2.6/site-packages/django/core/management/validation.py", line 102, in get_validation_errors
connection.validation.validate_field(e, opts, f)
File "/usr/lib/python2.6/site-packages/django/db/backends/mysql/validation.py", line 14, in validate_field
db_version = self.connection.get_server_version()
File "/usr/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 338, in get_server_version
self.cursor()
File "/usr/lib/python2.6/site-packages/django/db/backends/__init__.py", line 250, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/usr/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 322, in _cursor
self.connection = Database.connect(**kwargs)
File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (1044, "Access denied for user 'ckpuser'#'%' to database 'CKP'")
If it could help, I've integrated the improved WSGI script for use with Django given by Graham Dumpleton in his blog.
Thanks for any suggestion.