Is it possible to initiate a deep eagerload after the initial object is loaded?
E.g., the following is great:
session.query(Foo).options(joinedload('bars.bazs')).all()
But, what if I already have foo?
foo = session.query(Foo).first()
if foo.something:
do_nothing()
else:
# <- would now like to eagerload 'bars.bazs'
for bar in foo.bars:
for baz in bar.bazs:
# this is lazily loaded and slow
Any way to do this?
As far as I can see you are already doing what you asked for in your first example.
session.query(Foo).options(joinedload('bars.bazs')).all()
This will only load the Foo objects to start with. Only when you access the Foo.bars attribute will it load all Bar objects AND all Baz objects for all Bar objects.
If you want to eagerload all Foo, Bar and Baz objects with the initial query you would have to specify it like this:
session.query(Foo).options(joinedload('bars'), joinedload('bars.bazs')).all()
or shorter:
session.query(Foo).options(joinedload_all('bars.bazs')).all()
Documentation in the last part of this section.
To investigate how the objects get loaded from the DB when stepping through your code, you could pass echo=True to your create_engine() call.
Related
We are working on a Top-Down-RPG-like Multiplayer game for learning purposes (and fun!) with some friends. We already have some Entities in the Game and Inputs are working, but the network implementation gives us headache :D
The Issues
When trying to convert with dict some values will still contain the pygame.Surface, which I dont want to transfer and it causes errors when trying to jsonfy them. Other objects I would like to transfer in a simplyfied way like Rectangle cannot be converted automatically.
Already functional
Client-Server connection
Transfering JSON objects in both directions
Async networking and synchronized putting into a Queue
Situation
A new player connects to the server and wants to get the current game state with all objects.
Data-Structure
We use a "Entity-Component" based architecture, so we separated the game logic very strictly into "systems", while the data is stored in the "components" of each Entity. The Entity is a very simple container and has nothing more than a ID and a list of components. Example Entity (shorten for better readability):
Entity
|-- Component (Moveable)
|-- Component (Graphic)
| |- complex datatypes like pygame.SURFACE
| `- (...)
`- Component (Inventory)
We tried different approaches, but all seems not to fit very well or feel "hacky".
pickle
Very Python near, so not easy to implement other clients in future. And I´ve read about some security risks when creating items from network in this dynamic way how pickle it offers. It does not even solve the Surface/Rectangle issue.
__dict__
Still contains the reference to the old objects, so a "cleanup" or "filter" for unwanted datatypes happens also in the origin. A deepcopy throws Exception.
...\Python\Python36\lib\copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle pygame.Surface objects
Show some code
The method of the "EnitityManager" Class which should generate the Snapshot of all Entities, including their components. This Snapshot should be converted to JSON without any errors - and if possible without much configuration in this core-class.
class EnitityManager:
def generate_world_snapshot(self):
""" Returns a dictionary with all Entities and their components to send
this to the client. This function will probably generate a lot of data,
but, its to send the whole current game state when a new player
connects or when a complete refresh is required """
# It should be possible to add more objects to the snapshot, so we
# create our own Snapshot-Datastructure
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = deepcopy(e.__dict__)
# Components are Objects, but dictionary is required for transfer
cmp_obj_list = result['entities'][e.id]['components']
# Empty the current list of components, its going to be filled with
# dictionaries of each cmp which are cleaned for the dump, because
# of the errors directly coverting the whole datastructure to JSON
result['entities'][e.id]['components'] = {}
for cmp in cmp_obj_list:
cmp_copy = deepcopy(cmp)
cmp_dict = cmp_copy.__dict__
# Only list, dict, int, str, float and None will stay, while
# other Types are being simply deleted including their key
# Lists and directories will be cleaned ob recursive as well
cmp_dict = self.clean_complex_recursive(cmp_dict)
result['entities'][e.id]['components'][type(cmp_copy).__name__] \
= cmp_dict
logging.debug("EntityMgr: Entity#3: %s" % result['entities'][3])
return result
Expectation and actual results
We can find a way to manually override elements which we dont want. But as the list of components will increase we have to put all the filter logic into this core class, which should not contain any components specializations.
Do we really have to put all the logic into the EntityManager for filtering the right objects? This does not feel good, as I would like to have all convertion to JSON done without any hardcoded configuration.
How to convert all this complex data in a most generic approach?
Thanks for reading so far and thank you very much for your help in advance!
Interesting articles which we were already working threw and maybe helpful for others with similar issues
https://gafferongames.com/post/what_every_programmer_needs_to_know_about_game_networking/
http://code.activestate.com/recipes/408859/
https://docs.python.org/3/library/pickle.html
UPDATE: Solution - thx 2 sloth
We used a combination of the following architecture, which works really great so far and is also good to maintain!
Entity Manager now calls the get_state() function of the entity.
class EntitiyManager:
def generate_world_snapshot(self):
""" Returns a dictionary with all Entities and their components to send
this to the client. This function will probably generate a lot of data,
but, its to send the whole current game state when a new player
connects or when a complete refresh is required """
# It should be possible to add more objects to the snapshot, so we
# create our own Snapshot-Datastructure
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = e.get_state()
return result
The Entity has only some basic attributes to add to the state and forwards the get_state() call to all the Components:
class Entity:
def get_state(self):
state = {'name': self.name, 'id': self.id, 'components': {}}
for cmp in self.components:
state['components'][type(cmp).__name__] = cmp.get_state()
return state
The components itself now inherit their get_state() method from their new superclass components, which simply cares about all simple datatypes:
class Component:
def __init__(self):
logging.debug('generic component created')
def get_state(self):
state = {}
for attr, value in self.__dict__.items():
if value is None or isinstance(value, (str, int, float, bool)):
state[attr] = value
elif isinstance(value, (list, dict)):
# logging.warn("Generating state: not supporting lists yet")
pass
return state
class GraphicComponent(Component):
# (...)
Now every developer has the opportunity to overlay this function to create a more detailed get_state() function for complex types directly in the Component Classes (like Graphic, Movement, Inventory, etc.) if it is required to safe the state in a more accurate way - which is a huge thing for maintaining the code in future, to have these code pieces in one Class.
Next step is to implement the static method for creating the items from the state in the same Class. This makes this working really smooth.
Thank you so much sloth for your help.
Do we really have to put all the logic into the EntityManager for filtering the right objects?
No, you should use polymorphism.
You need a way to represent your game state in a form that can be shared between different systems; so maybe give your components a method that will return all of their state, and a factory method that allows you create the component instances out of that very state.
(Python already has the __repr__ magic method, but you don't have to use it)
So instead of doing all the filtering in the entity manager, just let him call this new method on all components and let each component decide that the result will look like.
Something like this:
...
result = {'entities': {}}
entities = self.get_all_entities()
for e in entities:
result['entities'][e.id] = {'components': {}}
for cmp in e.components:
result['entities'][e.id]['components'][type(cmp).__name__] = cmp.get_state()
...
And a component could implement it like this:
class GraphicComponent:
def __init__(self, pos=...):
self.image = ...
self.rect = ...
self.whatever = ...
def get_state(self):
return { 'pos_x': self.rect.x, 'pos_y': self.rect.y, 'image': 'name_of_image.jpg' }
#staticmethod
def from_state(state):
return GraphicComponent(pos=(state.pos_x, state.pos_y), ...)
And a client's EntityManager that recieves the state from the server would iterate for the component list of each entity and call from_state to create the instances.
I found two ways to instance class:
one is:
class_name create instance
instance class_method
the other is:
set instance [class_name new]
$instance class_method
Each way worked well, so is there any difference between two ways?
The only difference is that the new method on classes generates a unique name for you, and with the create method you get to specify what the name is. Both are provided because there's use cases for each of them. Use whichever makes sense for you. (Note that class objects themselves are always named because of how they're generally used, and so you can't normally create classes with new; it's hidden on oo::class instances.)
For the sake of completeness only, there's an additional way to make instances, createWithNamespace, which lets you also specify the name of the state namespace of the object. It's not exposed by default (you have to manually export it for general use) and is pretty specialist for people who aren't doing deep shenanigans. You probably don't want to use it.
At some point in the future new may be enhanced so that it also enables garbage collection for the object, whereas create will not (because you know the name out-of-band). Specifically, I've written a TIP for doing this for Tcl 9.0 but I don't have a working implementation yet, so do not hold your breath.
I have tried some cases,
class defined as this:
oo::class create class_name {
variable text
method add {t} { set text $t }
method print { } { puts $text }
}
First way:
class_name create foo
foo add "abc"
foo print
class_name create foo
It will return:
abc
Error: can't create object "foo": command already exists with that name
Second way:
set foo [class_name new]
$foo add "abc"
$foo print
set foo [class_name new]
$foo add "def"
$foo print
It will return:
abc
def
This is one difference I found, and it seems like secnod way is more convenient.
Still expecting master to provide authoritative answers or documents.
I don't know if this is a problem that others get, but I have code in python that goes like this:
def makemove(board,move,val):
new=board
new[move[0]][move[1]]=val
return new
My problem is that if I use this function by simply doing makemove(game,[0,1],-1) where game equals [[0,0,1],[0,1,0],[1,0,0]] the variable game becomes [[0, -1, 1], [0, 1, 0], [1, 0, 0]].
I have tried to look into functions setting global variables, but I have thus for not found a way to prevent makemove() from setting the variables that you put into it. Is there something obvious that I'm missing?
You need to clone board.
import
new = copy.deepcopy(board)
see https://stackoverflow.com/a/2612815/93910 for other ways of doing this.
Your code sets elements of a variable which is a "reference".
In other words, your variable new is really an array reference to board, i.e. it points to the same memory location. So when you change new, the original variable board gets changed.
Here's another answer on how to think about this: https://stackoverflow.com/a/9697367/93910
This is basically because assignment in Python doesn't mean what you think it does, and lists are mutable sequences.
Lists are Python objects. The name board is merely a label for or reference to that object.
So when you say new=board this means "let the name new reference the same object as the name board".
In Python every object has a unique identifier that you can retrieve using id(). Let's create a board, do the assignment and see what happens:
In [1]: board = [[0,0,1],[0,1,0],[1,0,0]]
In [2]: new = board
In [3]: id(new), id(board)
Out[3]: (34495504136, 34495504136)
new and board are referencing the same object.
Since a list is a mutable sequence you can change it without getting an error.
So if you want to play with any mutable sequence inside a function without modifying the original, you should use copy.deepcopy first to make a copy and modify that.
I have the following function:
(defn best-move [tracked-moves]
(def all-scores (ref tracked-moves))
#all-scores)
Its being called by a recursive function.
I want to be able to keep passing in tracked-moves, and for them to all exist within
#all-scores.
The way it is written right now, #all-scores will only hold onto the last tracked-moves that it is given. How can I get it to hold onto all of the data that it receives every time the best-move function is called? And to not just return the last of all the data it receives?
The problem is that you're using def incorrectly. Any use of def (and defn) will create a namespace-level var. It doesn't matter where you call def. As you've pointed out, you're continuously redefining all-scores. The short answer is to pull your definition of all-scores to the top level, so you're not constantly invoking it. Then, update the ref as described in the documentation. If you're not using transactions, and don't need to manage multiple refs, you might want to use atoms instead.
I had made a daemon that used a very primitive form of ipc (telnet and send a String that had certain words in a certain order). I snapped out of it and am now using JSON to pass messages to a Yesod server. However, there were some things I really liked about my design, and I'm not sure what my choices are now.
Here's what I was doing:
buildManager :: Phase -> IO ()
buildManager phase = do
let buildSeq = findSeq phase
jid = JobID $ pack "8"
config = MkConfig $ Just jid
flip C.catch exceptionHandler $
runReaderT (sequence_ $ buildSeq <*> stages) config
-- ^^ I would really like to keep the above line of code, or something like it.
return ()
each function in buildSeq looked like this
foo :: Stage -> ReaderT Config IO ()
data Config = MkConfig (Either JobID Product) BaseDir JobMap
JobMap is a TMVar Map that tracks information about current jobs.
so now, what I have are Handlers, that all look like this
foo :: Handler RepJson
foo represents a command for my daemon, each handler may have to process a different JSON object.
What I would like to do is send one JSON object that represents success, and another JSON object that espresses information about some exception.
I would like foos helper function to be able to return an Either, but I'm not sure how I get that, plus the ability to terminate evaluation of my list of actions, buildSeq.
Here's the only choice I see
1) make sure exceptionHandler is in Handler. Put JobMap in the App record. Using getYesod alter the appropriate value in JobMap indicating details about the exception,
which can then be accessed by foo
Is there a better way?
What are my other choices?
Edit: For clarity, I will explain the role ofHandler RepJson. The server needs some way to accept commands such as build stop report. The client needs some way of knowing the results of these commands. I have chosen JSON as the medium with which the server and client communicate with each other. I'm using the Handler type just to manage the JSON in/out and nothing more.
Philosophically speaking, in the Haskell/Yesod world you want to pass the values forward, rather than return them backwards. So instead of having the handlers return a value, have them call forwards to the next step in the process, which may be to generate an exception.
Remember that you can bundle any amount of future actions into a single object, so you can pass a continuation object to your handlers and foos that basically tells them, "After you are done, run this blob of code." That way they can be void and return nothing.