web2py: module function that behaves normally in shell fails browser mode - function

I have a very simple helper function that helps creates column definitions for a javascript framework I"m using.
def tocol(fields,title=None,width=None,attributes=None):
cols = dict((name,eval(name)) for name in ['title','width','attributes'] if eval(name) is not None)
cols["field"] = fields
return cols
when I try this in a web2py shell the result is as I'd expect:
In [15]: col = tocol(attr.columns.tolist())
In [16]: col
Out[16]: {'field': ['l1', 'pw', 'bw', 'tilt']}
but when I try the same thing in a view I get the following traceback:
Traceback (most recent call last):
File "/home/tahnoon/web2py/gluon/restricted.py", line 224, in restricted
exec ccode in environment
File "/home/tahnoon/web2py/applications/apollo/controllers/performance.py", line 788, in <module>
File "/home/tahnoon/web2py/gluon/globals.py", line 392, in <lambda>
self._caller = lambda f: f()
File "/home/tahnoon/web2py/applications/apollo/controllers/performance.py", line 561, in pa_equity
cols = tocol(attr.columns.tolist())
File "applications/apollo/modules/helpers.py", line 33, in tocol
cols = dict((name,eval(name)) for name in ['title','width','attributes'] if eval(name) is not None)
File "applications/apollo/modules/helpers.py", line 33, in <genexpr>
cols = dict((name,eval(name)) for name in ['title','width','attributes'] if eval(name) is not None)
File "<string>", line 1, in <module>
NameError: name 'title' is not defined
Might anyone have any idea what is going wrong here? Pretty mystified.
Thanks

It looks like you're just trying to remove None values from a dictionary, so instead, why not just create a function to do that:
def remove_none(d):
[d.pop(key) for key in d.keys() if d[key] is None]
Then you can do:
col = remove_none(dict(field=attr.columns.tolist(),
title=None, width=None, attributes=None))
Or if you want to enter the arguments directly::
def remove_none_dict(**d):
[d.pop(key) for key in d.keys() if d[key] is None]
return d
col = remove_none_dict(field=attr.columns.tolist(),
title=None, width=None, attributes=None))

Related

Getting an AttributeError when I try to create a Oracle BMC SecurityList

I am trying to automate the creation of resources in Oracle BMC. I have this python code:
import oraclebmc
config = oraclebmc.config.from_file()
network = oraclebmc.core.virtual_network_client.VirtualNetworkClient(config)
compartment_id = ...
vcn_id = ....
details = oraclebmc.core.models.CreateSecurityListDetails()
details.compartment_id = compartment_id
details.display_name = "baseline"
details.ingress_security_rules = ()
details.egress_security_rules = ()
details.vcn_id = vcn_id
network.create_security_list(details)
But when I run this code I get:
Traceback (most recent call last):
File "deploy/cloudresources/foo.py", line 16, in <module>
network.create_security_list(details)
File "/Users/jwmcclai/bmcs_env/lib/python2.7/site-packages/oraclebmc/core/virtual_network_client.py", line 668, in create_security_list
response_type="SecurityList")
File "/lib/python2.7/site-packages/oraclebmc/base_client.py", line 124, in call_api
body = self.sanitize_for_serialization(body)
File "/lib/python2.7/site-packages/oraclebmc/base_client.py", line 230, in sanitize_for_serialization
for key, val in obj_dict.items()}
File "/lib/python2.7/site-packages/oraclebmc/base_client.py", line 230, in <dictcomp>
for key, val in obj_dict.items()}
File "/lib/python2.7/site-packages/oraclebmc/base_client.py", line 226, in sanitize_for_serialization
for attr, _ in obj.swagger_types.items()
AttributeError: 'tuple' object has no attribute 'swagger_types'
I can create security lists through console and I can create other resources (VCN, instances, etc.) using the Python API. Any ideas?
Thanks
This is because you are defining the security rules fields as tuples, not as lists.
Your code:
details.ingress_security_rules = ()
details.egress_security_rules = ()
Should be:
details.ingress_security_rules = []
details.egress_security_rules = []
As the docs mention, these fields should be of type list, not type tuple.

Sqlalchemy class _MatchType(sqltypes.Float, sqltypes.MatchType): AttributeError

sometimes execute query and got this error:
ct = db.session.query(CIType).filter(
CIType.type_name == key).first() or \
db.session.query(CIType).filter(CIType.type_id == key).first()
full error info
2016-08-11 14:27:26,177 ERROR /usr/lib/python2.6/site-packages/flask/app.py 1306 - Exception on /api/v0.1/projects/search-indexer-rafael/product [GET]
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/flask/app.py", line 1687, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python2.6/site-packages/flask/app.py", line 1360, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python2.6/site-packages/flask/app.py", line 1358, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python2.6/site-packages/flask/app.py", line 1344, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/data/webapps/cmdb-api/core/special.py", line 175, in get_project_product
product = ProjectManager().get_for_product(project_name)
File "/data/webapps/cmdb-api/lib/special/project.py", line 18, in __init__
self.ci_type = CITypeCache.get("project")
File "/data/webapps/cmdb-api/models/cmdb.py", line 458, in get
ct = db.session.query(CIType).filter(
File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/scoping.py", line 149, in do
def do(self, *args, **kwargs):
File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/_collections.py", line 903, in __call__
item = dict.get(self, key)
File "/usr/lib/python2.6/site-packages/flask_sqlalchemy.py", line 201, in __init__
bind=db.engine,
File "/usr/lib/python2.6/site-packages/flask_sqlalchemy.py", line 754, in engine
return self.get_engine(self.get_app())
File "/usr/lib/python2.6/site-packages/flask_sqlalchemy.py", line 771, in get_engine
return connector.get_engine()
File "/usr/lib/python2.6/site-packages/flask_sqlalchemy.py", line 451, in get_engine
self._engine = rv = sqlalchemy.create_engine(info, **options)
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/__init__.py", line 344, in create_engine
of 0 indicates no limit; to disable pooling, set ``poolclass`` to
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 50, in create
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/url.py", line 116, in get_dialect
return self.get_dialect().driver
File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 170, in load
fn.__func__.__doc__ = doc
File "/usr/lib64/python2.6/site-packages/sqlalchemy/dialects/__init__.py", line 33, in _auto_fn
try:
File "/usr/lib64/python2.6/site-packages/sqlalchemy/dialects/mysql/__init__.py", line 8, in <module>
from . import base, mysqldb, oursql, \
File "/usr/lib64/python2.6/site-packages/sqlalchemy/dialects/mysql/base.py", line 681, in <module>
class _MatchType(sqltypes.Float, sqltypes.MatchType):
AttributeError: 'module' object has no attribute 'MatchType'
code
#special.route("/api/v0.1/projects/<string:project_name>/product",
methods=["GET"])
def get_project_product(project_name):
product = ProjectManager().get_for_product(project_name)
return jsonify(product=product)
...
goto
class ProjectManager(object):
def __init__(self):
self.ci_type = CITypeCache.get("project")
...
then
class CITypeCache(object):
#classmethod
def get(cls, key):
if key is None:
return
ct = cache.get("CIType::ID::%s" % key) or \
cache.get("CIType::Name::%s" % key)
if ct is None:
ct = db.session.query(CIType).filter(
CIType.type_name == key).first() or \
db.session.query(CIType).filter(CIType.type_id == key).first()
if ct is not None:
CITypeCache.set(ct)
return ct
The sqlalchemy's version is SQLAlchemy-1.0.8-py2.6.egg-info
and after many same error, I can't catch this error any more. What's the reason of this error?
I assume CIType.type_name and CIType.type_id have different data types (perhaps string and numeric types). It may lead to situation when:
db.session.query(CIType).filter(CIType.type_name == key).first()
is valid expression but:
db.session.query(CIType).filter(CIType.type_id == key).first()
produces error because of type mismatch. You need to convert key to the type_id column type in this expression.
The second expression is calculated when first expression returns no results. As written in Python documentation:
The expression x or y first evaluates x; if x is true, its value is returned; otherwise, y is evaluated and the resulting value is returned.
For example:
>>> a = 1 or 2 + '2'
>>> print a
1
>>> a = 0 or 2 + '2'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

Python Return a list in Bottle

I have a list from an mysql query that I'm trying to return in my bottle website. Is this possible? Here's what I have:
def create_new_location():
kitchen_locations = select_location()
return template('''
% for kitchen_location in {{kitchen_locations}}:
{{kitchen_location}} Kitchen
<br/>
% end''',kitchen_locations=kitchen_locations)
This is the error that I get.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/local/lib/python2.7/site-packages/bottle.py", line 1732, in wrapper
rv = callback(*a, **ka)
File "index.py", line 32, in create_new_location
</form>''',kitchen_locations=kitchen_locations)
File "/usr/local/lib/python2.7/site-packages/bottle.py", line 3609, in template
return TEMPLATES[tplid].render(kwargs)
File "/usr/local/lib/python2.7/site-packages/bottle.py", line 3399, in render
self.execute(stdout, env)
File "/usr/local/lib/python2.7/site-packages/bottle.py", line 3386, in execute
eval(self.co, env)
File "<string>", line 6, in <module>
TypeError: unhashable type: 'set'
Got It (took me a while...)
% for kitchen_location in {{kitchen_locations}}:
Should be
% for kitchen_location in kitchen_locations:
When using the % at the beginning you don't need the {{}}.
This error:
TypeError: unhashable type: 'set'
is trying to use a set literal {{kitchen_locations}} ==>
kitchen_locations in a set in another set. since set is not hash-able you got the error

error loading json using topsy

When i load single record json is created just fine when i try to load multiple records i get this error. Sorry i am new to python http://tny.cz/ce1baaba
Traceback (most recent call last):
File "TweetGraber.py", line 26, in <module>
get_tweets_by_query(topic)
File "TweetGraber.py", line 15, in get_tweets_by_query
json_tree = json.loads(source)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 368, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 2 column 1 - line 11 column 1 (char 2380 - 46974)
Here is my code
def get_tweets_by_query(query, count=10):
"""A function that gets the tweets by a query."""
Tweets=[]
queryEncoded=urllib.quote(query)
api_key = "xxxxx"
source=urllib.urlopen("http://api.topsy.com/v2/content/bulktweets.json?q=%s&type=tweet&offset=0&perpage=%s&window=realtime&apikey=%s" % (queryEncoded, count, api_key)).read()
json_tree = json.loads(source)
pprint(json_tree)
topic = raw_input("Please enter a topic: ")
get_tweets_by_query(topic)
Thanks Timusan I was able to correct my json The problem with the original it was missing the root element "[" which showed we are expecting array and there "," was missing after end of each object. So here is fixed code.
So here is the code
def get_tweets_by_query(query, count=10):
"""A function that gets the tweets by a query."""
Tweets=[]
queryEncoded=urllib.quote(query)
api_key = "xx"
source=urllib.urlopen("http://api.topsy.com/v2/content/bulktweets.json?q=%s&type=tweet&offset=0&perpage=%s&window=realtime&apikey=%s" % (queryEncoded, count, api_key)).read()
source="["+source+"]"
source=source.replace("}\n{","},{")
json_tree = json.loads(source)
pprint(json_tree)

sqlalchemy pyramid query keyerror

I am working on a project that loads local data onto a local server using SQLAlchemy and a pyramid framework. The following describes the setup:
-The local data are divided in multiple CSV files
-A separate SQL object (called DataSource) is created from each individual CSV file
-The parent of all DataSources is also a DataSource. Thus all CSV data are children
So, the parent DataSource is created, then all the children DataSources are created and linked to the parent. A separate function checks if the DataSource already exists. If not, it is created.
So, firstly we create the parent DS:
sName = 'Parent DS'
oDS = fetch_or_create_data_source(name=sName,author="Parent_DS_author"
)
Then we loop through all CSVs and create the children DS:
for sCSV_name in lCSVs:
oChildDS = fetch_or_create_data_source(name=sCSV_name,parent=oDS)
Here is the problem:
fetch_or_create_data_source creates a keyerror on the second iteration of CSV loading. So the parent and the first child DataSources are created without a problem. The error log below is created on the second child DS. A new DS should be created, since no there exists no DS with the given parent and name. All CSV filenames are unique.
Can anyone see why I get a mysterious key error on the second iteration of the DS creation? I could not find anything on the webs.
(EDIT): I know the error is created in lDS = list(oDS.filter_by(name=name).filter_by(parent=parent)), but I do not understand why, especially after the first two times it runs without a problem. I changed the code so that the parent_id is called instead of the parent object, but the KeyError persists. The error is caused by any filter_by() statement. To check for primitive values, I added some print statements to check the arguments in fetch_or_create_data_source. The results:
name = 'Parent DS' type <type 'str'>
author = 'Parent_DS_author' type <type 'str'>
parent = None
Current DS id = '352' type <type 'int'>
.
.
name = 'tab19' type <type 'str'>
author = '' type <type 'str'>
parent = <pyramidapp.models.DataSource object at 0x7fcb28066b50>
parent_id = 352 type <type 'int'>
Current DS id = '353' type <type 'int'>
.
.
name = 'tab42' type <type 'str'>
author = '' type <type 'str'>
parent = <pyramidapp.models.DataSource object at 0x7fcb28066b50>
parent_id = 352 type <type 'int'>
When I explicitely state that oDS is None, the key error originates from DBSession.flush() in the 3rd iteration.
Why would the first two instances of fetch_or_create_data_source run without problems, but the third not?
Code and traceback:
fetch_or_create_data_source:
def fetch_or_create_data_source(name,parent=None,author=""):
from pyramidapp.models import DBSession,DataSource
oDS = DBSession.query(DataSource)
print "name = '{0}' type {1}".format(name,type(name))
print "author = '{0}' type {1}".format(author,type(author))
print "parent = {0}".format(parent)
if parent:
print "parent_id = {0} type {1}".format(parent.id,type(parent.id))
if parent is None:
lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent))
else:
lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent.id)) <=== Key error from here
if len(lDS)==0:
oDS = None
else:
oDS = lDS[0]
if not oDS:
oDS = DataSource()
oDS.name = name
if parent:
oDS.parent = parent
if parent.author:
oDS.author = parent.author
if author:
oDS.author = author
DBSession.add(oDS)
DBSession.flush()
print "Current DS id = '{0}' type {1}".format(oDS.id,type(oDS.id))
return oDS
Pyramidapp.models.DataSource:
class DataSource(Base):
"""
this represents a release of data. It may be a group of documents
or a single spreadsheet
"""
__tablename__ = 'data_sources'
id = Column(Integer,Sequence('data_src_seq'),primary_key=True)
name = Column(String(100))
notes = Column(Text)
title = Column(String(100))
author = Column(String(100))
parent_id = Column(Integer,ForeignKey('data_sources.id'))
parent = relationship('DataSource',backref=backref("children",cascade="all, delete"),remote_side=[id,])
.
.
...
Traceback (most recent call last):
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_debugtoolbar-2.0.2-py2.7.egg/pyramid_debugtoolbar/panels/performance.py", line 55, in resource_timer_handler
result = handler(request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_tm-0.7-py2.7.egg/pyramid_tm/__init__.py", line 82, in tm_tween
reraise(*exc_info)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_tm-0.7-py2.7.egg/pyramid_tm/__init__.py", line 63, in tm_tween
response = handler(request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/router.py", line 163, in handle_request
response = view_callable(context, request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/config/views.py", line 355, in rendered_view
result = view(context, request)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/config/views.py", line 477, in _class_requestonly_view
response = getattr(inst, attr)()
File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/views/basic_views.py", line 108, in run_ds_script
oScript.run(dSettings)
File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/scripts/data_source_specific/load_prescriptions_dispensed_in_the_community_data.py", line 82, in run
oChildDS = fetch_or_create_data_source(name=sCSV_name,parent=oDS)
File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/tools/data_source_script_tools.py", line 22, in fetch_or_create_data_source
lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent.id))
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2397, in __iter__
self.session._autoflush()
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1184, in _autoflush
self.flush()
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1879, in flush
self._flush(objects)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1997, in _flush
transaction.rollback(_capture_exception=True)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 57, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1961, in _flush
flush_context.execute()
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 370, in execute
rec.execute(self)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 523, in execute
uow
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 64, in save_obj
mapper, table, insert)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 594, in _emit_insert_statements
execute(statement, params)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 717, in execute
return meth(self, multiparams, params)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 814, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 927, in _execute_context
context)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1079, in _handle_dbapi_exception
util.reraise(*exc_info)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 920, in _execute_context
context)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 425, in do_execute
cursor.execute(statement, parameters)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/cursors.py", line 93, in execute
escaped_args = tuple(conn.escape(arg) for arg in args)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/cursors.py", line 93, in <genexpr>
escaped_args = tuple(conn.escape(arg) for arg in args)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/connections.py", line 698, in escape
return escape_item(obj, self.charset)
File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/converters.py", line 24, in escape_item
encoder = encoders[type(val)]
KeyError: <class 'pyramidapp.models.DataSource'>
If you start from the very end of the traceback and go a few lines up, you'll see that the error is raised in the PyMySQL lib when it tries to escape the value - supposedly it has some kind of a dict called encoders which contains a mapping of encoders. It looks like it fails when passed an instance if pyramidapp.models.DataSource class.
Surely, PyMySQL knows nothing about SQLAlchemy declarative classes and can only escape primitive types (ints, strings etc). So it's kinda unexpected it even sees the DataSource instance instead.
From your code samples it's not quite clear what causes the problem, but my best guess would be that you're either assigning a class instance to an attribute which is supposed to hold a primitive value, or maybe using a class instance in a query where it expects a primitive value. Something like this:
ds = DataSource(...)
other_ds.parent_id = ds # instead of .parent
or
session.query(DataSource).filter(DataSource.parent_id==ds)
Actually, here it is:
lDS = list(oDS.filter_by(name=name).filter_by(parent=parent))