Trying to use event listeners(https://docs.sqlalchemy.org/en/14/core/events.html#sqlalchemy.events.ConnectionEvents
) on a async sqlalchemy engine and am getting this error:
{NotImplementedError}asynchronous events are not implemented at this time. Apply synchronous listeners to the AsyncEngine.sync_engine or AsyncConnection.sync_connection attributes.
If I'm understanding this correctly I cant use events on an async engine and I have to switch to a sync engine if I want event support?
engine: Engine = create_async_engine(
URL, echo=True, future=True
)
async_session = sessionmaker(
engine, class_=AsyncSession, expire_on_commit=False, future=True
)
event.listens_for(engine, "do_connect")(do_connect_listener)
event.listens_for(engine, "engine_connect")(engine_connect_listener)
got an answer from the SQLAlchemy discussion section on github. All credit to the OP(https://github.com/sqlalchemy/sqlalchemy/discussions/6594#discussioncomment-836437):
"Hi,
As suggested by the exception you may use the sync_engine/sync_connection when applying events. Here is an example:"
import asyncio
from sqlalchemy import event
from sqlalchemy.ext.asyncio import create_async_engine
engine = create_async_engine("sqlite+aiosqlite:///:memory:")
#event.listens_for(engine.sync_engine, "do_connect")
def do_connect(dialect, conn_rec, cargs, cparams):
print("some-function")
#event.listens_for(engine.sync_engine, "engine_connect")
def engine_connect(conn, branch):
print("engine_connect", conn.exec_driver_sql("select 1").scalar())
async def go():
async with engine.connect() as conn:
res = await conn.exec_driver_sql("select 2")
print("go", res.scalar())
asyncio.run(go())
Related
I'm trying to make bulk data downloads by serializing my entire database as JSON. The drf documentation on serializers has a section that says you can simply do:
from rest_framework.renderers import JSONRenderer
serializer = CommentSerializer(comment)
json = JSONRenderer().render(serializer.data)
Unfortunately, this doesn't work for HyperLinked relationships. When you try to do it with them, you get something like:
AssertionError: HyperlinkedIdentityField requires the request in the serializer context. Add context={'request': request} when instantiating the serializer.
So, I figured out I can add context attribute, like:
r = Request(request=HttpRequest())
context = dict(request=r)
serializer = CommentSerializer(comment, context=context)
json = JSONRenderer().render(serializer.data)
Which then returns the error:
django.core.exceptions.ImproperlyConfigured: Could not resolve URL for hyperlinked relationship using view name "opinioncluster-detail". You may have failed to include the related model in your API, or incorrectly configured the lookup_field attribute on this field.
I know this API works properly when it's called from the browser, but I can't get past this when I call it as above. Something that's automatic from the browser doesn't happen when you render it manually.
Any ideas?
First edit
Here's another strategy that seemed promising because it would add the path to my request object:
r = Request(request=RequestFactory().get(reverse('comment-list', kwargs={'version': 'v3'})))
context = dict(request=r)
serializer = CommentSerializer(comment, context=context)
json = JSONRenderer().render(serializer.data)
That returns the same problem as if I hadn't defined a path to the request:
django.core.exceptions.ImproperlyConfigured: Could not resolve URL for hyperlinked relationship using view name "opinioncluster-detail". You may have failed to include the related model in your API, or incorrectly configured the lookup_field attribute on this field.
Second Edit
As I said in my comments, I'm fairly certain my serializers and views aren't to blame, since they work perfectly fine via the browser. Nevertheless, here they are. If you're truly generous, the full serializers, filters, and codebase is online.
View:
class OpinionClusterViewSet(LoggingMixin, viewsets.ModelViewSet):
queryset = OpinionCluster.objects.all()
serializer_class = OpinionClusterSerializer
filter_class = OpinionClusterFilter
ordering_fields = (
'date_created', 'date_modified', 'date_filed', 'citation_count',
'date_blocked',
)
Serializer:
class OpinionClusterSerializer(DynamicFieldsModelSerializer,
serializers.HyperlinkedModelSerializer):
absolute_url = serializers.CharField(source='get_absolute_url',
read_only=True)
panel = serializers.HyperlinkedRelatedField(
many=True,
view_name='judge-detail',
read_only=True,
)
non_participating_judges = serializers.HyperlinkedRelatedField(
many=True,
view_name='judge-detail',
read_only=True,
)
docket = serializers.HyperlinkedRelatedField(
many=False,
view_name='docket-detail',
read_only=True,
)
sub_opinions = serializers.HyperlinkedRelatedField(
many=True,
view_name='opinion-detail',
read_only=True,
)
class Meta:
model = OpinionCluster
This management command creates a MockRequest to be used in a non-browser environment and should allow you to create your JSON:
from django.core.management.base import BaseCommand
from django.http import HttpRequest
from rest_framework.renderers import JSONRenderer
from rest_framework.request import Request
from comments.models import Comment
from comments.serializers import CommentSerializer
class MockRequest(HttpRequest):
def __init__(self):
super(MockRequest, self).__init__()
self.setup_host()
def setup_host(self):
# Required to give absolute urls in output
self.META['HTTP_HOST'] = 'localhost:8000'
class Command(BaseCommand):
help = 'Export JSON'
def handle(self, *args, **options):
request = MockRequest()
serializer_context = {
'request': Request(request),
}
comment = Comment.objects.first()
serializer = CommentSerializer(comment, context=serializer_context)
json = JSONRenderer().render(serializer.data)
print(json)
To remove the hard coded domain you could use the Sites framework to power the host name:
def setup_host(self):
from django.contrib.sites.models import Site
site = Site.objects.get_current()
self.META['HTTP_HOST'] = site.domain
After digging quite deeply into the code, I finally figured this out:
r = RequestFactory().request()
r.version = 'v3'
r.versioning_scheme = URLPathVersioning()
context = dict(request=r)
renderer = JSONRenderer()
json_str = renderer.render(
serializer(item, context=context).data,
accepted_media_type='application/json; indent=2',
)
This seems to work, but the HyperLinkRelated values in the JSON that is serialized have the server set to testserver. I could get around that by setting:
r.META['SERVER_NAME']
But I also need to set r.scheme to https, which doesn't seem to be possible (I get an error that r.scheme cannot be set).
I'm pretty close though, so this is going to have to serve as an answer for now.
I have a Scala Play 2.2.2 application and as part of my Specification tests I would like to insert some fixture data for testing preferably in json format. For the tests I use the usual in-memory H2 database. How can I accomplish this? I have searched all the documentation but there is no mention to this anywhere.
Note that I would prefer not to build my own flavor of fixture implementation via the Global. There should be a non-hacky way to this right?
AFAIK there is no built-in stuff to do this, ala Rails, and it's hard to imagine what the devs could do without making Play Scala much more opinionated about the way persistence should be handled (which I'd personally consider a negative.)
I also use H2 for testing and employ plain SQL fixtures in a resource file and load them before tests using a couple of (fairly simple) helpers:
package object helpers {
import java.io.File
import java.sql.CallableStatement
import org.specs2.execute.{Result, AsResult}
import org.specs2.mutable.Around
import org.specs2.specification.Scope
import play.api.db.DB
import play.api.test.FakeApplication
import play.api.test.Helpers._
/**
* Load a file containing SQL statements into the DB.
*/
private def loadSqlResource(resource: String)(implicit app: FakeApplication) = DB.withConnection { conn =>
val file = new File(getClass.getClassLoader.getResource(resource).toURI)
val path = file.getAbsolutePath
val statement: CallableStatement = conn.prepareCall(s"RUNSCRIPT FROM '$path'")
statement.execute()
conn.commit()
}
/**
* Run a spec after loading the given resource name as SQL fixtures.
*/
abstract class WithSqlFixtures(val resource: String, val app: FakeApplication = FakeApplication()) extends Around with Scope {
implicit def implicitApp = app
override def around[T: AsResult](t: => T): Result = {
running(app) {
loadSqlResource(resource)
AsResult.effectively(t)
}
}
}
}
Then, in your actual spec you can do something like so:
package models
import helpers.WithSqlFixtures
import play.api.test.PlaySpecification
class MyModelSpec extends PlaySpecification {
"My model" should {
"locate items correctly" in new WithSqlFixtures("model-fixtures.sql") {
MyModel.findAll().size must beGreaterThan(0)
}
}
}
Note: this specs2 stuff could probably be better.
Obviously if you really need JSON you'll have to add extra machinery to deserialise your models and persist them in the database (often in your app you'll be doing these things anyway, in which case that might be relatively trivial.)
You'll also need:
Some evolutions to establish your DB schema in conf/evolutions/default
The evolution plugin enabled, which will build your schema when the FakeApplication starts up
The appropriate H2 DB config
I'm creating an application in Scala using Play 2.2. I'm using play-slick 0.5.0.8 as my MySQL DB connector. I have the following application controller:
package controllers
import models._
import models.database._
import play.api._
import play.api.mvc._
import play.api.Play.current
import play.api.db.slick._
object Application extends Controller {
// WORKS:
def test = DBAction {
implicit session => Ok(views.html.test(Cameras.findById(1)))
}
// DOES NOT WORK:
def photo = Action {
val p = PhotoFetcher.fetchRandomDisplayPhoto(someParametersBlah))
Ok(views.html.photo(p))
}
}
As you can see, the test DBAction works, and it's able to fetch a photo from the DB just fine. Unfortunately, the photo Action does not work.
My PhotoFetcher.fetchRandomDisplayPhoto(blah) does a bunch of different things. Buried inside of it is a call to Cameras.findById(blah), which should return a Camera object (which works in the test DBAction). However, with this configuration I get the following error:
could not find implicit value for parameter s: slick.driver.MySQLDriver.simple.Session
I have tried making the photo Action into a DBAction, like so:
def photo = DBAction {
implicit session => {
val p = PhotoFetcher.fetchRandomDisplayPhoto(someParametersBlah))
Ok(views.html.photo(p))
}
}
But that just results in the same missing session error. It's like PhotoFetcher doesn't know about the implicit session.
The other thing I've tried is importing slick.session.Database.threadLocalSession in my PhotoFetcher, but that only results in the following error:
SQLException: No implicit session available; threadLocalSession can only be used within a withSession block
If it's any help, this is a simplified version of my Cameras object:
package models.database
import models.Format.Format
import scala.slick.driver.MySQLDriver.simple._
case class Camera(id: Long,
otherStuff: String)
trait CamerasComponent {
val Cameras: Cameras
class Cameras extends Table[Camera]("cameras") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def otherStuff = column[String]("otherStuff", O.NotNull)
def * = id ~ otherStuff <> (Camera.apply _, Camera.unapply _)
val byId = createFinderBy(_.id)
val byOtherStuff = createFinderBy(_.otherStuff)
}
}
object Cameras extends DAO {
def insert(camera: Camera)(implicit s: Session) { Cameras.insert(camera) }
def findById(id: Long)(implicit s: Session): Option[Camera] = Cameras.byId(id).firstOption
def findByOtherStuff(otherStuff: String)(implicit s: Session): Option[Camera] = Cameras.byOtherStuff(model).firstOption
}
So, it seems as if I've gotten crossed-up somewhere. Right now it's only possible for me to access my DAO objects directly from a Controller DBAction, and not from inside of some different class. Any help would be appreciated. Thanks!
Does your definition of PhotoFetcher.fetchRandomDisplayPhoto.fetchRandomDisplayPhoto take an implicit session?
// PhotoFetcher
def fetchRandomDisplayPhoto(args: Blah*)(implicit s: Session) = {
// ...
val maybeCam = Cameras.findById(blah) // <- sees the implicit session
// ...
}
Or are you relying on a threadLocalsession in PhotoFetcher? (no implicit session argument for fetchRandomDisplayPhoto)?
While Slick's threadLocalSession is handy for quickly trying out stuff, it can lead to confusion and loss of clarity later on. It's best to just use explicit (implicit s: Session) parameter lists for all methods that call your Slick models. This also plays
well with DBAction, letting the framework manage sessions.
The downside is you have to have (implicit s: Session) on all your methods - there
are workarounds like this:
https://github.com/freekh/play-slick/issues/20
Scala isn't verbose and is very amenable to refactoring - so I'd recommend thinking
about crossing that bridge when you come to it, and use DBAction for all actions
that do database stuff; give all methods that call your database models an
implicit session, and see how much that mileage that gives you.
resolved not an issue, was caused by the presence of:
def __del__(self):
print "deleted!"
in the model class. As soon as I removed it from the model class, I cannot experiment any problems with memory usage.
I have some models that I am using with an ad-hoc session/engine and this is working fine, but when I want to create these objects outside of a database/session context, it seems SQLAlchemy instrumentation keeps a reference on the objects and they are never deleted.
What I would like to do is to create a model object like a "normal" Python object and never add it to any session/database (but in other contexts I need to add it to a database). Is this wrong?
import gc
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, String
base = declarative_base()
class SomeObject(base):
__tablename__ = "SomeObject"
instrumented_name = Column('name', String(55), primary_key=True)
uninstrumented_name = None
def __del__(self):
print "deleted!"
obj1 = SomeObject()
obj1.uninstrumented_name = "foo"
obj1 = None
#obj1 is properly deleted
obj2 = SomeObject()
obj2.instrumented_name = "bar"
obj2 = None
gc.collect()
#obj2 never deleted
Edit I did some additional testing and it seems SQLAlchemy will cause memory leak if the objects are never commited into a session (e.g. rollbacked)
Is there a method that will force SQLAlchemy to release it's references on instrumented objects?
import gc
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, create_engine
Base = declarative_base()
class MemoryMonster(Base):
__tablename__ = "MemoryMonster"
_id = Column('id', Integer(), primary_key=True)
_name = Column('name', String(55))
def __init__(self):
self._name = "some monster name"
self._eat_some_ram = ' ' * 1048576
def __del__(self):
print "deleted!"
engine = create_engine("sqlite:///:memory:")
session_factory = sessionmaker(engine)
Session = scoped_session(session_factory)
Base.metadata.create_all(engine)
def create_and_commit():
session = Session()
for _ in range(100):
session.add(MemoryMonster())
session.commit()
Session.remove()
gc.collect()
def create_and_rollback():
session = Session()
for _ in range(100):
monster = MemoryMonster()
session.add(monster)
session.expunge(monster)
session.rollback()
Session.remove()
gc.collect()
def create_do_not_include_in_session():
session = Session()
for _ in range(100):
monster = MemoryMonster()
session.rollback()
Session.remove()
gc.collect()
# Scenario 1 - Objects included in the session and commited
# No memory leak
create_and_commit()
# Scenario 2 - Objects included in the session and rollbacked
# Memory leak
create_and_rollback()
# Scenario 3 - Objects are not not included in the session
# Memory leak
create_do_not_include_in_session()
Using __del__() can create memory leaks, because if an object becomes the subject of a cycle, it is then unreachable by cyclic GC. That is the case in this test because the SQLAlchemy object instrumentation creates a cycle from the object to itself in the case that the object is "dirty", that is, has pending attributes to be flushed to the database, so that you can add a dirty object to the Session and then lose all references to it, and the changes will still be flushed. This "reference marker" is removed as soon as the object is marked as clean (i.e. flushed).
For SQLAlchemy 0.8.1 I've improved this behavior: one is that this reference cycle is no longer created for "pending" or "detached" objects, that is, objects which aren't associated with a Session. Instead, the object is checked when it is attached to a Session for the .modified flag, and the reference marker is associated only at that point (and is removed when the object becomes clean, as was the case already). If the object is detached from the session, the marker is unconditionally removed, even if the object still has changes - the .modified flag stays true.
Additionally, I've added a warning when a class is first mapped and detected as having a __del__() method. It's very easy for a Python object to have cycles, and in the case of SQLAlchemy the pattern of placing a relationship() on a class with a backref will also have the effect of creating reference cycles, so even with the state management improvement here, using __del__() is a bad idea.
I want to know how to know, given an object, if it is an instance of an sqlalchemy mapped model.
Normally, I would use isinstance(obj, DeclarativeBase). However, in this scenario, I do not have the DeclarativeBase class used available (since it is in a dependency project).
I would like to know what is the best practice in this case.
class Person(DeclarativeBase):
__tablename__ = "Persons"
p = Person()
print isinstance(p, DeclarativeBase)
#prints True
#However in my scenario, I do not have the DeclarativeBase available
#since the DeclarativeBase will be constructed in the depending web app
#while my code will act as a library that will be imported into the web app
#what are my alternatives?
You can use class_mapper() and catch the exception.
Or you could use _is_mapped_class, but ideally you should not as it is not a public method.
from sqlalchemy.orm.util import class_mapper
def _is_sa_mapped(cls):
try:
class_mapper(cls)
return True
except:
return False
print _is_sa_mapped(MyClass)
# #note: use this at your own risk as might be removed/renamed in the future
from sqlalchemy.orm.util import _is_mapped_class
print bool(_is_mapped_class(MyClass))
for instances there is the object_mapper(), so:
from sqlalchemy.orm.base import object_mapper
def is_mapped(obj):
try:
object_mapper(obj)
except UnmappedInstanceError:
return False
return True
the complete mapper utilities are documented here: http://docs.sqlalchemy.org/en/rel_1_0/orm/mapping_api.html
Just a consideration: since specific errors are raised by SQLAlchemy (UnmappedClassError for calsses and UnmappedInstanceError for instances) why not catch them rather than a generic exception? ;)