I am working on a project that loads local data onto a local server using SQLAlchemy and a pyramid framework. The following describes the setup:
-The local data are divided in multiple CSV files
-A separate SQL object (called DataSource) is created from each individual CSV file
-The parent of all DataSources is also a DataSource. Thus all CSV data are children
So, the parent DataSource is created, then all the children DataSources are created and linked to the parent. A separate function checks if the DataSource already exists. If not, it is created.
So, firstly we create the parent DS:
sName = 'Parent DS'
oDS = fetch_or_create_data_source(name=sName,author="Parent_DS_author"
)
Then we loop through all CSVs and create the children DS:
for sCSV_name in lCSVs:
oChildDS = fetch_or_create_data_source(name=sCSV_name,parent=oDS)
Here is the problem:
fetch_or_create_data_source creates a keyerror on the second iteration of CSV loading. So the parent and the first child DataSources are created without a problem. The error log below is created on the second child DS. A new DS should be created, since no there exists no DS with the given parent and name. All CSV filenames are unique.
Can anyone see why I get a mysterious key error on the second iteration of the DS creation? I could not find anything on the webs.
(EDIT): I know the error is created in lDS = list(oDS.filter_by(name=name).filter_by(parent=parent)), but I do not understand why, especially after the first two times it runs without a problem. I changed the code so that the parent_id is called instead of the parent object, but the KeyError persists. The error is caused by any filter_by() statement. To check for primitive values, I added some print statements to check the arguments in fetch_or_create_data_source. The results:
name = 'Parent DS' type <type 'str'>
author = 'Parent_DS_author' type <type 'str'>
parent = None
Current DS id = '352' type <type 'int'>
.
.
name = 'tab19' type <type 'str'>
author = '' type <type 'str'>
parent = <pyramidapp.models.DataSource object at 0x7fcb28066b50>
parent_id = 352 type <type 'int'>
Current DS id = '353' type <type 'int'>
.
.
name = 'tab42' type <type 'str'>
author = '' type <type 'str'>
parent = <pyramidapp.models.DataSource object at 0x7fcb28066b50>
parent_id = 352 type <type 'int'>
When I explicitely state that oDS is None, the key error originates from DBSession.flush() in the 3rd iteration.
Why would the first two instances of fetch_or_create_data_source run without problems, but the third not?
Code and traceback:
fetch_or_create_data_source:
def fetch_or_create_data_source(name,parent=None,author=""):
from pyramidapp.models import DBSession,DataSource
oDS = DBSession.query(DataSource)
print "name = '{0}' type {1}".format(name,type(name))
print "author = '{0}' type {1}".format(author,type(author))
print "parent = {0}".format(parent)
if parent:
print "parent_id = {0} type {1}".format(parent.id,type(parent.id))
if parent is None:
lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent))
else:
lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent.id)) <=== Key error from here
if len(lDS)==0:
oDS = None
else:
oDS = lDS[0]
if not oDS:
oDS = DataSource()
oDS.name = name
if parent:
oDS.parent = parent
if parent.author:
oDS.author = parent.author
if author:
oDS.author = author
DBSession.add(oDS)
DBSession.flush()
print "Current DS id = '{0}' type {1}".format(oDS.id,type(oDS.id))
return oDS
Pyramidapp.models.DataSource:
class DataSource(Base):
"""
this represents a release of data. It may be a group of documents
or a single spreadsheet
"""
__tablename__ = 'data_sources'
id = Column(Integer,Sequence('data_src_seq'),primary_key=True)
name = Column(String(100))
notes = Column(Text)
title = Column(String(100))
author = Column(String(100))
parent_id = Column(Integer,ForeignKey('data_sources.id'))
parent = relationship('DataSource',backref=backref("children",cascade="all, delete"),remote_side=[id,])
.
.
...
Traceback (most recent call last):
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_debugtoolbar-2.0.2-py2.7.egg/pyramid_debugtoolbar/panels/performance.py", line 55, in resource_timer_handler
result = handler(request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_tm-0.7-py2.7.egg/pyramid_tm/__init__.py", line 82, in tm_tween
reraise(*exc_info)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_tm-0.7-py2.7.egg/pyramid_tm/__init__.py", line 63, in tm_tween
response = handler(request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/router.py", line 163, in handle_request
response = view_callable(context, request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/config/views.py", line 355, in rendered_view
result = view(context, request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/config/views.py", line 477, in _class_requestonly_view
response = getattr(inst, attr)()
File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/views/basic_views.py", line 108, in run_ds_script
oScript.run(dSettings)
File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/scripts/data_source_specific/load_prescriptions_dispensed_in_the_community_data.py", line 82, in run
oChildDS = fetch_or_create_data_source(name=sCSV_name,parent=oDS)
File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/tools/data_source_script_tools.py", line 22, in fetch_or_create_data_source
lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent.id))
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2397, in __iter__
self.session._autoflush()
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1184, in _autoflush
self.flush()
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1879, in flush
self._flush(objects)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1997, in _flush
transaction.rollback(_capture_exception=True)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 57, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1961, in _flush
flush_context.execute()
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 370, in execute
rec.execute(self)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 523, in execute
uow
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 64, in save_obj
mapper, table, insert)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 594, in _emit_insert_statements
execute(statement, params)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 717, in execute
return meth(self, multiparams, params)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 814, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 927, in _execute_context
context)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1079, in _handle_dbapi_exception
util.reraise(*exc_info)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 920, in _execute_context
context)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 425, in do_execute
cursor.execute(statement, parameters)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/cursors.py", line 93, in execute
escaped_args = tuple(conn.escape(arg) for arg in args)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/cursors.py", line 93, in <genexpr>
escaped_args = tuple(conn.escape(arg) for arg in args)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/connections.py", line 698, in escape
return escape_item(obj, self.charset)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/converters.py", line 24, in escape_item
encoder = encoders[type(val)]
KeyError: <class 'pyramidapp.models.DataSource'>
If you start from the very end of the traceback and go a few lines up, you'll see that the error is raised in the PyMySQL lib when it tries to escape the value - supposedly it has some kind of a dict called encoders which contains a mapping of encoders. It looks like it fails when passed an instance if pyramidapp.models.DataSource class.
Surely, PyMySQL knows nothing about SQLAlchemy declarative classes and can only escape primitive types (ints, strings etc). So it's kinda unexpected it even sees the DataSource instance instead.
From your code samples it's not quite clear what causes the problem, but my best guess would be that you're either assigning a class instance to an attribute which is supposed to hold a primitive value, or maybe using a class instance in a query where it expects a primitive value. Something like this:
ds = DataSource(...)
other_ds.parent_id = ds # instead of .parent
or
session.query(DataSource).filter(DataSource.parent_id==ds)
Actually, here it is:
lDS = list(oDS.filter_by(name=name).filter_by(parent=parent))
Related
Good day.
I have a small password generator program and I want to save the created passwords into a json file (append each time) so I can add them to an SQLITE3 database.
Just trying to do the append functionality I receive several errors that I don't understand.
Here are the errors I receive and below that is the code itself.
I'm quite new to Python so additional details are welcomed.
Traceback (most recent call last):
File "C:\Users\whitmech\OneDrive - Six Continents Hotels, Inc\04 - Python\02_Mosh_Python_Course\Py_Projects\PWGenerator.py", line 32, in
data = json.load(file)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
import random
import string
import sqlite3
import json
from pathlib import Path
print('hello, Welcome to Password generator!')
# input the length of password
length = int(input('\nEnter the length of password: '))
# define data
lower = string.ascii_lowercase
upper = string.ascii_uppercase
num = string.digits
symbols = string.punctuation
# string.ascii_letters
# combine the data
all = lower + upper + num + symbols
# use random
temp = random.sample(all, length)
# create the password
password = "".join(temp)
filename = 'saved.json'
entry = {password}
with open(filename, "r+") as file:
data = json.load(file)
data.append(entry)
file.seek(0)
json.dump(data, file)
# print the password
print(password)
Update: I've changed the JSON code as directed and it works but when trying to do the SQLite3 code I'm knowing receiving a typeerror
Code:
with open(filename, "r+") as file:
try:
data = json.load(file)
data.append(entry)
except json.decoder.JSONDecodeError as e:
data = entry
file.seek(0)
json.dump(data, file)
# print the password
print(password)
store = input('Would you like to store the password? ')
if store == "Yes":
pwStored = json.loads(Path("saved.json").read_text())
with sqlite3.connect("db.pws") as conn:
command = "INSERT INTO Passwords VALUES (?)"
for i in pwStored:
conn.execute(command, tuple(i.values)) # Error with this code
conn.commit()
else:
exit()
Error:
AttributeError: 'str' object has no attribute 'values'
The error is because
Your json file is empty, you need to update the following block
entry = [password]
with open(filename, "r+") as file:
try:
data = json.load(file)
data.extend(entry)
except json.decoder.JSONDecodeError as e:
data = entry
file.seek(0)
json.dump(data, file)
Also you are adding password in a set ie., entry, and it will again throw you an error TypeError: Object of type set is not JSON serializable
So you need to convert that to either a list or dict
Note: Here I have used entry as a list
In my application, I use an internal db and an external db where I use the external db to load some details from the main/global database.
From time to time when I try to query some data or commit a transaction, I get an error sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back which causes whole application to break down and it's not fixed until I restart it. This mostly happen when the external database is being restarted. We lose connection, the transaction gets invalid and it hangs.
In the FastAPI app, I use SQLAlchemy. I set the external DB as follows (with some redundant details omitted):
class _ExternalDB(DBSetup):
def __init__(self):
self._database = Database(self.url, ssl=ctx)
self._engine = create_engine(self.url, echo=True, echo_pool='debug')
self._session = scoped_session(
sessionmaker(autocommit=False, autoflush=False, bind=self._engine, class_=SafeSession)
)
where DBSetup is a base class which contains url, database and session. I wrote SafeSession class to try and solve the issue I am having:
class SafeSession(Session):
def commit(self):
try:
return super().commit()
except Exception as e:
logger.error("Could not commit the SQL operation!", e)
self.rollback()
raise e
def query(self, *entities, **kwargs):
try:
return super().query(*entities, **kwargs)
except Exception as e:
logger.error("Could not complete the transaction!", e)
self.rollback()
raise e
An example query is as follows:
external_db_setup.session.query(User).get(user_id)
My goal was to roll the invalid transaction back whenever it happens, but the application never gets to the except block because I have never seen any logs for "Could not complete the transaction". Am I missing something?
Full error log:
File "/***/repositories/user.py", line 8, in get_user_by_id
return external_db_setup.session.query(User).get(user_id)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1004, in get
return self._get_impl(ident, loading.load_on_pk_identity)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1119, in _get_impl
return db_load_fn(self, primary_key_identity)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 284, in load_on_pk_identity
return q.one()
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3358, in one
ret = self.one_or_none()
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3327, in one_or_none
ret = list(self)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3403, in __iter__
return self._execute_and_instances(context)
File "/***/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3428, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 984, in execute
return meth(self, multiparams, params)
File "/***/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_clauseelement
ret = self._execute_context(
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1177, in _execute_context
self._handle_dbapi_exception(
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _handle_dbapi_exception
util.raise_(
File "/***/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1173, in _execute_context
conn = self._revalidate_connection()
File "/***/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 456, in _revalidate_connection
raise exc.InvalidRequestError(
graphql.error.located_error.GraphQLLocatedError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
I'm trying to use marshmallow-sqlalchemy with aiohttp and I have followed their docs with the basic example and I'm getting an error.
I have this schema:
from marshmallow_sqlalchemy import SQLAlchemyAutoSchema
from db.customer import Customer
class CustomerSchema(SQLAlchemyAutoSchema):
class Meta:
model = Customer
include_relationships = True
load_instance = True
And then the following code for the query:
from sqlalchemy import select
from db import db_conn
from db.customer import Customer
from queries.schema import CustomerSchema
customer_schema = CustomerSchema()
async def get_all_users():
async with db_conn.get_async_sa_session() as session:
statement = select(Customer)
results = await session.execute(statement)
_ = (results.scalars().all())
print(_)
response = customer_schema.dump(_, many=True)
print(response)
For the first print statement I'm getting
[<db.customer.Customer object at 0x10a183340>, <db.customer.Customer object at 0x10a183940>, <db.customer.Customer object at 0x10b0cd9d0>]
But then it fails with
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 60, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: http://sqlalche.me/e/14/xd2s)
So how can I use marshmallow-sqlalchemy to serialize the SqlAlchemy reponse?
Another options (packages, etc) or a generic custom solutions are OK too.
For the time being I'm using this:
statement = select(Customer)
results = await session.execute(statement)
_ = (results.scalars().all())
response = {}
for result in _:
value = {k: (v if not isinstance(v, sqlalchemy.orm.state.InstanceState) else '_') for k, v in result.__dict__.items()}
response[f'customer {value["id"]}'] = value
return response
Full traceback:
Traceback (most recent call last):
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
resp = await self._request_handler(request)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/aiohttp/web_app.py", line 499, in _handle
resp = await handler(request)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/aiohttp/web_urldispatcher.py", line 948, in _iter
resp = await method()
File "/Users/ruslan/OneDrive/Home/Dev/projects/code/education/other/cft/views/user.py", line 24, in get
await get_all_users()
File "/Users/ruslan/OneDrive/Home/Dev/projects/code/education/other/cft/queries/user.py", line 18, in get_all_users
response = customer_schema.dump(_, many=True)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 547, in dump
result = self._serialize(processed_obj, many=many)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 509, in _serialize
return [
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 510, in <listcomp>
self._serialize(d, many=False)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 515, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/fields.py", line 310, in serialize
value = self.get_value(obj, attr, accessor=accessor)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow_sqlalchemy/fields.py", line 27, in get_value
return super(fields.List, self).get_value(obj, attr, accessor=accessor)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/fields.py", line 239, in get_value
return accessor_func(obj, check_key, default)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/schema.py", line 472, in get_attribute
return get_value(obj, attr, default)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/utils.py", line 239, in get_value
return _get_value_for_key(obj, key, default)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/marshmallow/utils.py", line 253, in _get_value_for_key
return getattr(obj, key, default)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 480, in __get__
return self.impl.get(state, dict_)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 931, in get
value = self.callable_(state, passive)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 879, in _load_for_state
return self._emit_lazyload(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 1036, in _emit_lazyload
result = session.execute(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/sql/lambdas.py", line 481, in _execute_on_connection
return connection._execute_clauseelement(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1451, in _execute_clauseelement
ret = self._execute_context(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
self._handle_dbapi_exception(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1998, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1770, in _execute_context
self.dialect.do_execute(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
cursor.execute(statement, parameters)
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 449, in execute
self._adapt_connection.await_(
File "/Users/ruslan/.local/share/virtualenvs/cft-RKlbQ9iX/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 60, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: http://sqlalche.me/e/14/xd2s)
The problem in this case is that the Marshmallow schema is configured to load related models (include_relationships=True). Since the initial query doesn't load them automatically, the schema triggers a query to fetch them, and this causes the error.
The simplest solution, demonstrated in the docs, is to eagerly load the related objects with their "parent":
async def get_all_users():
async with db_conn.get_async_sa_session() as session:
# Let's assume a Customer has a 1 to many relationship with an Order model
statement = select(Customer).options(orm.selectinload(Customer.orders))
results = await session.execute(statement)
_ = (results.scalars().all())
print(_)
response = customer_schema.dump(_, many=True)
print(response)
There is more discussion in the Preventing Implicit IO when Using AsyncSession section of the docs.
Following this question:
As Ilja Everilä mentioned in his answer, I created a table object:
from sqlalchemy import *
metadata = MetaData()
idTagTable = Table('id_tag', metadata,
Column('id', String(255), primary_key = True),
Column('category', String(20), nullable = False),
Column('createddate', Date, nullable = False),
Column('updatedon', Date, nullable = False)
)
After creating a table object, I changed insert and update statements:
insert_statement = sqlalchemy.dialects.postgresql.insert(idTagTable)
upsert_statement = insert_statement.on_conflict_do_update(
constraint=PrimaryKeyConstraint('id'),
set_={"updatedon": insert_statement.excluded.updateon,
"category":insert_statement.excluded.category}
)
insert_values = df.to_dict(orient='records')
conn.execute(upsert_statement, insert_values)
Now I am getting Programming Error:
Traceback (most recent call last):
File "<ipython-input-66-0fc6a1bf9c6b>", line 7, in <module>
conn.execute(upsert_statement, insert_values)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
exc_info
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _execute_context
context)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 467, in do_executemany
cursor.executemany(statement, parameters)
ProgrammingError: (psycopg2.ProgrammingError) syntax error at or near
")"
LINE 1: ...category) VALUES ('sports') ON CONFLICT () DO UPDAT...
^
Not Able to understand why I am getting this error.
The PrimaryKeyConstraint object you're using as constraint= argument is not bound to any table and would seem to produce nothing when rendered, as seen in ON CONFLICT (). Instead pass the primary key(s) of your table as the conflict_target and Postgresql will perform unique index inference:
upsert_statement = insert_statement.on_conflict_do_update(
constraint=idTagTable.primary_key,
set_={"updatedon": insert_statement.excluded.updateon,
"category":insert_statement.excluded.category}
)
I have a very simple helper function that helps creates column definitions for a javascript framework I"m using.
def tocol(fields,title=None,width=None,attributes=None):
cols = dict((name,eval(name)) for name in ['title','width','attributes'] if eval(name) is not None)
cols["field"] = fields
return cols
when I try this in a web2py shell the result is as I'd expect:
In [15]: col = tocol(attr.columns.tolist())
In [16]: col
Out[16]: {'field': ['l1', 'pw', 'bw', 'tilt']}
but when I try the same thing in a view I get the following traceback:
Traceback (most recent call last):
File "/home/tahnoon/web2py/gluon/restricted.py", line 224, in restricted
exec ccode in environment
File "/home/tahnoon/web2py/applications/apollo/controllers/performance.py", line 788, in <module>
File "/home/tahnoon/web2py/gluon/globals.py", line 392, in <lambda>
self._caller = lambda f: f()
File "/home/tahnoon/web2py/applications/apollo/controllers/performance.py", line 561, in pa_equity
cols = tocol(attr.columns.tolist())
File "applications/apollo/modules/helpers.py", line 33, in tocol
cols = dict((name,eval(name)) for name in ['title','width','attributes'] if eval(name) is not None)
File "applications/apollo/modules/helpers.py", line 33, in <genexpr>
cols = dict((name,eval(name)) for name in ['title','width','attributes'] if eval(name) is not None)
File "<string>", line 1, in <module>
NameError: name 'title' is not defined
Might anyone have any idea what is going wrong here? Pretty mystified.
Thanks
It looks like you're just trying to remove None values from a dictionary, so instead, why not just create a function to do that:
def remove_none(d):
[d.pop(key) for key in d.keys() if d[key] is None]
Then you can do:
col = remove_none(dict(field=attr.columns.tolist(),
title=None, width=None, attributes=None))
Or if you want to enter the arguments directly::
def remove_none_dict(**d):
[d.pop(key) for key in d.keys() if d[key] is None]
return d
col = remove_none_dict(field=attr.columns.tolist(),
title=None, width=None, attributes=None))