Is there a way to mark tables as read-only when using Pyramid with SQLAlchemy?
Some brief background: I am working on an app with an existing database which is very complicated. My app adds a few new tables to this database, but updating (update, insert, delete) other tables is unsafe because the schema is not well understood (and isn't documented).
My basic aim is to fail-fast before any unsafe db updates so the offending code can be found and removed, so I and future developers don't cause any data issues with this app.
My naive approach: add a pyramid_tm commit veto hook that checks objects to be updated (new, dirty, and deleted in the db session) and vetos if there are any read-only tables.
Is there an existing mechanism to handle this? Or a better approach?
If you're using the "declarative" approach, you can try making the classes themselves "read-only":
class MyBase(object):
"""
This class is a superclass of SA-generated Base class,
which in turn is the superclass of all db-aware classes
so we can define common functions here
"""
def __setattr__(self, name, value):
"""
Raise an exception if attempting to assign to an atribute of a "read-only" object
Transient attributes need to be prefixed with "_t_"
"""
if (getattr(self, '__read_only__', False)
and name != "_sa_instance_state"
and not name.startswith("_t_")):
raise ValueError("Trying to assign to %s of a read-only object %s" % (name, self))
super(MyBase, self).__setattr__(name, value)
Base = declarative_base(cls=MyBase)
Similarly you can try to override __init__ method to prevent the objects from being instantiated.
However, solving this problem at the application level feels a bit fragile - I would just look at creating a special user in the database for your app to connect with, which would save limited permissions on the tables you want to keep intact.
Related
I have a Pylons project and a SQLAlchemy model that implements schema qualified tables:
class Hockey(Base):
__tablename__ = "hockey"
__table_args__ = {'schema':'winter'}
hockey_id = sa.Column(sa.types.Integer, sa.Sequence('score_id_seq', optional=True), primary_key=True)
baseball_id = sa.Column(sa.types.Integer, sa.ForeignKey('summer.baseball.baseball_id'))
This code works great with Postgresql but fails when using SQLite on table and foreign key names (due to SQLite's lack of schema support)
sqlalchemy.exc.OperationalError: (OperationalError) unknown database "winter" 'PRAGMA "winter".table_info("hockey")' ()
I'd like to continue using SQLite for dev and testing.
Is there a way of have this fail gracefully on SQLite?
I'd like to continue using SQLite for
dev and testing.
Is there a way of have this fail
gracefully on SQLite?
It's hard to know where to start with that kind of question. So . . .
Stop it. Just stop it.
There are some developers who don't have the luxury of developing on their target platform. Their life is a hard one--moving code (and sometimes compilers) from one environment to the other, debugging twice (sometimes having to debug remotely on the target platform), gradually coming to an awareness that the gnawing in their gut is actually the start of an ulcer.
Install PostgreSQL.
When you can use the same database environment for development, testing, and deployment, you should.
Not to mention the QA team. Why on earth are they testing stuff they're not going to ship? If you're deploying on PostgreSQL, assure the quality of your work on PostgreSQL.
Seriously.
I'm not sure if this works with foreign keys, but someone could try to use SQLAlchemy's Multi-Tenancy Schema Translation for Table objects. It worked for me but I have used custom primaryjoin and secondaryjoinexpressions in combination with composite primary keys.
The schema translation map can be passed directly to the engine creator:
...
if dialect == "sqlite":
url = lambda: "sqlite:///:memory:"
execution_options={"schema_translate_map": {"winter": None, "summer": None}}
else:
url = lambda: f"postgresql://{user}:{pass}#{host}:{port}/{name}"
execution_options=None
engine = create_engine(url(), execution_options=execution_options)
...
Here is the doc for create_engine. There is a another question on so which might be related in that regard.
But one might get colliding table names all schema names are mapped to None.
I'm just a beginner myself, and I haven't used Pylons, but...
I notice that you are combining the table and the associated class together. How about if you separate them?
import sqlalchemy as sa
meta = sa.MetaData('sqlite:///tutorial.sqlite')
schema = None
hockey_table = sa.Table('hockey', meta,
sa.Column('score_id', sa.types.Integer, sa.Sequence('score_id_seq', optional=True), primary_key=True),
sa.Column('baseball_id', sa.types.Integer, sa.ForeignKey('summer.baseball.baseball_id')),
schema = schema,
)
meta.create_all()
Then you could create a separate
class Hockey(Object):
...
and
mapper(Hockey, hockey_table)
Then just set schema above = None everywhere if you are using sqlite, and the value(s) you want otherwise.
You don't have a working example, so the example above isn't a working one either. However, as other people have pointed out, trying to maintain portability across databases is in the end a losing game. I'd add a +1 to the people suggesting you just use PostgreSQL everywhere.
HTH, Regards.
I know this is a 10+ year old question, but I ran into the same problem recently: Postgres in production and sqlite in development.
The solution was to register an event listener for when the engine calls the "connect" method.
#sqlalchemy.event.listens_for(engine, "connect")
def connect(dbapi_connection, connection_record):
dbapi_connection.execute('ATTACH "your_data_base_name.db" AS "schema_name"')
Using ATTACH statement only once will not work, because it affects only a single connection. This is why we need the event listener, to make the ATTACH statement over all connections.
I wish to use Django's F to easily get cross-relational information. I had a working form of what I wanted before, but I felt it was ugly and so I transitioned to making custom properties for my models as so:
#property
def stage(self):
"""Returns the competition's stage status by date"""
now = timezone.now()
if self.end_date and self.pub_date and self.close_date and self.start_date:
if self.start_date > now:
return "Launched"
elif self.start_date < now < self.end_date:
return "Commenced"
elif self.end_date < now < self.close_date:
return "Scoring"
else:
return "Ended"
return "Inactive"
In the past, I used a huge When/Case block and annotated this in. I prefer 'property' style of doing things because their values don't mess up when aggregating. (Eg. Team counts get overcounted when I do sums etc. and so I have to do them separately)
However, now I get errors when I try to access: (Team->CompetitionTeam->Competition)
F('competitionteam__competition__stage')
Is there a way to do this? The errors are: (When debugging and trying to annotate the above)
django.core.exceptions.FieldError: Cannot resolve keyword 'stage' into field.
Just running straight from my code, it doesn't cough an error immediately after annotating, but only when the QuerySet in question is accessed.
django.core.exceptions.FieldError: Unsupported lookup 'stage' for AutoField or join on the field not permitted, perhaps you meant range?
Note*: There is no error if I am not annotating and just accessing them from the Objects directly.
You can't use F objects to access model properties because F objects are used when the query is accessing the database and the database does not have access to your python code.
My preference is to use annotations, which you said you stopped doing due to overcounting when you have multiple annotations. This happens because Django's default implementation of annotations uses joins instead of subqueries. This is noted in the docs on combining multiple aggregations.
I believe the solution here is to do the annotation using a subquery. The stage property doesn't use any relational data, so you should be able to do that as you were before. For the other annotations you are doing (you mention Team counts and sums) it's hard to show exactly what you need to do without seeing your models and what you want to annotate, but here are the docs on subquery.
You can also use the package django-sql-utils which provides aggregates like SubqueryCount and SubquerySum which work just like Django's built in Count and Sum but are implemented with subqueries instead of joins.
We are working on a Top-Down-RPG-like Multiplayer game for learning purposes (and fun!) with some friends. We already have some Entities in the Game and Inputs are working, but the network implementation gives us headache :D
The Issues
When trying to convert with dict some values will still contain the pygame.Surface, which I dont want to transfer and it causes errors when trying to jsonfy them. Other objects I would like to transfer in a simplyfied way like Rectangle cannot be converted automatically.
Already functional
Client-Server connection
Transfering JSON objects in both directions
Async networking and synchronized putting into a Queue
Situation
A new player connects to the server and wants to get the current game state with all objects.
Data-Structure
We use a "Entity-Component" based architecture, so we separated the game logic very strictly into "systems", while the data is stored in the "components" of each Entity. The Entity is a very simple container and has nothing more than a ID and a list of components. Example Entity (shorten for better readability):
Entity
|-- Component (Moveable)
|-- Component (Graphic)
| |- complex datatypes like pygame.SURFACE
| `- (...)
`- Component (Inventory)
We tried different approaches, but all seems not to fit very well or feel "hacky".
pickle
Very Python near, so not easy to implement other clients in future. And I´ve read about some security risks when creating items from network in this dynamic way how pickle it offers. It does not even solve the Surface/Rectangle issue.
__dict__
Still contains the reference to the old objects, so a "cleanup" or "filter" for unwanted datatypes happens also in the origin. A deepcopy throws Exception.
...\Python\Python36\lib\copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle pygame.Surface objects
Show some code
The method of the "EnitityManager" Class which should generate the Snapshot of all Entities, including their components. This Snapshot should be converted to JSON without any errors - and if possible without much configuration in this core-class.
class EnitityManager:
def generate_world_snapshot(self):
""" Returns a dictionary with all Entities and their components to send
this to the client. This function will probably generate a lot of data,
but, its to send the whole current game state when a new player
connects or when a complete refresh is required """
# It should be possible to add more objects to the snapshot, so we
# create our own Snapshot-Datastructure
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = deepcopy(e.__dict__)
# Components are Objects, but dictionary is required for transfer
cmp_obj_list = result['entities'][e.id]['components']
# Empty the current list of components, its going to be filled with
# dictionaries of each cmp which are cleaned for the dump, because
# of the errors directly coverting the whole datastructure to JSON
result['entities'][e.id]['components'] = {}
for cmp in cmp_obj_list:
cmp_copy = deepcopy(cmp)
cmp_dict = cmp_copy.__dict__
# Only list, dict, int, str, float and None will stay, while
# other Types are being simply deleted including their key
# Lists and directories will be cleaned ob recursive as well
cmp_dict = self.clean_complex_recursive(cmp_dict)
result['entities'][e.id]['components'][type(cmp_copy).__name__] \
= cmp_dict
logging.debug("EntityMgr: Entity#3: %s" % result['entities'][3])
return result
Expectation and actual results
We can find a way to manually override elements which we dont want. But as the list of components will increase we have to put all the filter logic into this core class, which should not contain any components specializations.
Do we really have to put all the logic into the EntityManager for filtering the right objects? This does not feel good, as I would like to have all convertion to JSON done without any hardcoded configuration.
How to convert all this complex data in a most generic approach?
Thanks for reading so far and thank you very much for your help in advance!
Interesting articles which we were already working threw and maybe helpful for others with similar issues
https://gafferongames.com/post/what_every_programmer_needs_to_know_about_game_networking/
http://code.activestate.com/recipes/408859/
https://docs.python.org/3/library/pickle.html
UPDATE: Solution - thx 2 sloth
We used a combination of the following architecture, which works really great so far and is also good to maintain!
Entity Manager now calls the get_state() function of the entity.
class EntitiyManager:
def generate_world_snapshot(self):
""" Returns a dictionary with all Entities and their components to send
this to the client. This function will probably generate a lot of data,
but, its to send the whole current game state when a new player
connects or when a complete refresh is required """
# It should be possible to add more objects to the snapshot, so we
# create our own Snapshot-Datastructure
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = e.get_state()
return result
The Entity has only some basic attributes to add to the state and forwards the get_state() call to all the Components:
class Entity:
def get_state(self):
state = {'name': self.name, 'id': self.id, 'components': {}}
for cmp in self.components:
state['components'][type(cmp).__name__] = cmp.get_state()
return state
The components itself now inherit their get_state() method from their new superclass components, which simply cares about all simple datatypes:
class Component:
def __init__(self):
logging.debug('generic component created')
def get_state(self):
state = {}
for attr, value in self.__dict__.items():
if value is None or isinstance(value, (str, int, float, bool)):
state[attr] = value
elif isinstance(value, (list, dict)):
# logging.warn("Generating state: not supporting lists yet")
pass
return state
class GraphicComponent(Component):
# (...)
Now every developer has the opportunity to overlay this function to create a more detailed get_state() function for complex types directly in the Component Classes (like Graphic, Movement, Inventory, etc.) if it is required to safe the state in a more accurate way - which is a huge thing for maintaining the code in future, to have these code pieces in one Class.
Next step is to implement the static method for creating the items from the state in the same Class. This makes this working really smooth.
Thank you so much sloth for your help.
Do we really have to put all the logic into the EntityManager for filtering the right objects?
No, you should use polymorphism.
You need a way to represent your game state in a form that can be shared between different systems; so maybe give your components a method that will return all of their state, and a factory method that allows you create the component instances out of that very state.
(Python already has the __repr__ magic method, but you don't have to use it)
So instead of doing all the filtering in the entity manager, just let him call this new method on all components and let each component decide that the result will look like.
Something like this:
...
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = {'components': {}}
for cmp in e.components:
result['entities'][e.id]['components'][type(cmp).__name__] = cmp.get_state()
...
And a component could implement it like this:
class GraphicComponent:
def __init__(self, pos=...):
self.image = ...
self.rect = ...
self.whatever = ...
def get_state(self):
return { 'pos_x': self.rect.x, 'pos_y': self.rect.y, 'image': 'name_of_image.jpg' }
#staticmethod
def from_state(state):
return GraphicComponent(pos=(state.pos_x, state.pos_y), ...)
And a client's EntityManager that recieves the state from the server would iterate for the component list of each entity and call from_state to create the instances.
I found many examples for older versions of CherryPy but they each referenced importing modules not found in cherrypy 3.2.2. Looking in the documentation, I found a reference to the fact that there is built in functionality with storage_type (one of ‘ram’, ‘file’, ‘postgresql’).
For a start you could take a look at
https://github.com/3kwa/cherrys
how this guy writes his own session class and overwrites some methods. He does it for redis not MySQL. You would write methods for MySQL. A very similar class already exists in cherrypy in "cherrpy/lib/sessions.py":
class PostgresqlSession(Session)
which is very similar to what you want. I'd say, take the implementing approach from the "3kwa" but instead of his RedisSession-class copy the PostgresqlSession-class from "cherrpy/lib/sessions.py" and alter to match proper MySQL-Syntax.
A possible path could be:
Download the "cherrys.py" from above link and rename into "mysqlsession.py". Overwrite the "RedisSessions(Session)" with the "PostgresqlSession(Session)" from "cherrpy/lib/sessions.py" and rename to "MySQLSession(Session)". Be sure to add
locks = {}
def acquire_lock(self):
"""Acquire an exclusive lock on the currently-loaded session data."""
self.locked = True
self.locks.setdefault(self.id, threading.RLock()).acquire()
def release_lock(self):
"""Release the lock on the currently-loaded session data."""
self.locks[self.id].release()
self.locked = False
to your new "MySQLSession"-class (like it is done in RedisSession(Session). Alter the the PostgreSQL-Syntax to match MySQL-Syntax (that shouldn't be difficult). Put the "mysqlsession.py" somewhere below your project directory and import in the application with
import mysqlsession
and use
cherrypy.lib.sessions.MySQLSession = mysqlsession.MySQLSession
in the initialization of you app. In the config
tools.sessions.storage_type : 'mysql'
and the parameters (like host, port, etc.) like you would with class "PostgreSQL".
I can be wrong all along. But this is how I would try to solve this.
Imagine the following model:
A Table has many Rows
A Row has many Cells
What would be the preferable interface to deal with these classes in a "object oriented way"?
1 - Provide access to the properties rows / cells (Not necessarily exposing the underlying data structures, but creating for example a class RowCollection...)
my_table = new Table()
my_table.rows.add([1,2,3])
my_row = my_table.rows.get(0)
my_row.cells.get(0)
for(cell in my_row.cells) {}
...
2 - Or provide the methods directly in the Table and Row classes
my_table = new Table()
my_table.add_row([1,2,3])
my_row = my_table.get_row(0)
my_row.get_cell(0)
for(cell in my_row.get_cells) {}
...
3 - None of the above...
I think that the answer is largely subjective. If we go by your example, providing methods or properties of your class to return a value by a row/column reference might be appropriate. These could be implemented concurrently, eg:
myClass.Row[x].Column[y]
myClass.Column[y].Row[x]
myClass.Cell[x,y]
You might also decide that it is better to expose a list directly, if the data "rows" are finite:
myClass.SomeArrayOfValues[itemIndex]
I notice you using phrases like "tables" and "rows", so I might assume you wish to have your class represent a database or similar structure, but you might find that while it may be efficient to store the data that way, you might find that exposing the data in another form may make more sense to the user of your class.
In the end, how you choose to do this should really be designed to reflect the purpose of the data itself and the system you are modelling, and that can only be decided on a case-by-case basis.
Based on your comment regarding the usage, "The main use case is adding, sorting and iterating the values," I would probably not allow individual elements to be retrieved, but instead have the user provide a functor to act upon the stored elements. Expressed in C++.
class Table
{
public:
Table();
//Table(unsigned int numberOfRows, unsigned int numberOfCells);
void addRow();
void addCell();
//Throw exception if out of range or don't supply these functions if not needed by user.
void removeRow(unsigned int rowNumber);
void removeCell(unsigned int rowNumber, unsigned int cellNumber);
//Iterate over entire table
template<class Pred>
void forEach(Pred pred);
//Iterate over a specific row, throw exception if row is out of range.
template<class Pred>
void forEach(unsigned int row, Pred pred);
}
You will have to tailor the add/update/remove calls based on how you plan on inputting/updating data. This design is strongly oriented towards manipulating collections of elements. The positives of this design is that you are not committing your user to the specific underlying structure of how you are representing the Table. This is in keeping with the Law of Demeter.
If you need to access specific individual elements, you will want a different approach or make it an extension of what's already provided here.
Consider how many getters and setters you can get rid of. A robust OO design has objects exporting behavior to each other, not data. For example, the skeleton of a getter/setter model of a Person:
class Person:
def set_name(value):
def get_name:
def set_age(value):
def get_age:
def set_drink(value):
def get_drink:
def set_favorite_drink(value):
def get_favorite_drink:
And here's some (pseudo-)code that uses Person:
def order_drink(person)
if person.age >= 21:
puts "#{person.name} can have a drink"
person.drink = bar.order(person.favorite_drink)
else:
puts "#{person.name} cannot drink (legally)."
Here's how you can have a person with no getters or setters involved in ordering a drink:
class Person:
def order_drink_from(bar):
if self.age >= 21:
puts "#{self.name} can have a drink"
self.drink = bar.order(favorite_drink)
else:
puts "#{self.name} cannot drink (legally)"
used like this:
person.order_drink_from(bar)
I won't say that you'll never need getters in an OO program. But I will say this: setters, especially, ought to make you rethink the design. And every time you write either a getter or a setter, let a little voice in the back of your head ask you if there's a way to have the objects export behavior rather than data.
It depends on how you intend to access the Rows/Cells.
There is no one correct way of doing it - you need to decide how you want to access them and build your objects to expose them in the way you want to consume them.
It depends a lot on the data and what you plan to do with it.
Will users want individual cells? Individual rows/columns? Subsections of rows/columns?
Probably the cleanest way is to provide functor interfaces. Provide one or more functions that will run the functors on every element, or on a subset defined in the functor.
That may work less well if users need to access complex combinations of cells.