I'm actually using SQLAlchemy with MySQL and Pyro to make a server program. Many clients connect to this server to make requests. The programs only provides the information from the database MySQL and sometimes make some calculations.
Is it better to create a session for each client or to use the same session for every clients?
What you want is a scoped_session.
The benefits are (compared to a single shared session between clients):
No locking needed
Transactions supported
Connection pool to database (implicit done by SQLAlchemy)
How to use it
You just create the scoped_session:
Session = scoped_session(some_factory)
and access it in your Pyro methods:
class MyPyroObject():
def remote_method(self):
Session.query(MyModel).filter...
Behind the scenes
The code above guarantees that the Session is created and closed as needed. The session object is created as soon as you access it the first time in a thread and will be removed/closed after the thread is finished (ref). As each Pyro client connection has its own thread on the default setting (don't change it!), you will have one session per client.
The best I can try is to create new Session in every client's request. I hope there is no penalty in the performance.
Related
I'm using the autobahn server in twisted to provide an RPC API. Some calls require queries to the database and multiple clients may be connected via websocket to the server.
I am using the SqlAlchemy ORM to access the database.
What are the pros and cons of the two following approaches for dealing with SqlAlchemy sessions.
Create and destroy a session for every RPC call
Create a single session when the server starts and use it in every RPC call
Which would you recommend and why? (I'm leaning towards 2)
The recommended way of doing SQL-based database access from Twisted (and Autobahn) with databases like PostgreSQL, Oracle or SQLite would be twisted.enterprise.adbapi.
twisted.enterprise.adbapi will run queries on a background thread pool, which is required, since most database drivers are blocking.
Sidenote: for PostgreSQL, there is a native-asynchronous, non-blocking
driver also: txpostgres.
Now, if you put an ORM like SQLAlchemy on top of the native SQL driver, I'm not sure how this will work together (if at all) with twisted.enterprise.adbapi.
So from the options you mention
Is a no go, since most drivers are blocking (and Autobahn's RPCs run on the main thread = Twisted reactor thread - and you MUST not block that).
With this, you need to put the database session(s) in background threads (again, to not block).
Also see here.
If you're using SQLAlchemy and Twisted together, consider using Alchimia rather than the built-in adbapi.
Hi am confused with sql servers session. What does it actually mean? Does it keep track of the client like httpSession? I have read some documents on query life cycle. None talks about the sesion. Most of the documents say that after the query is recived by the server it gets parsed and then maintains a syntax tree and then execution plan and then executes the query and then a dispatch palan and then dispatches the resultset to the client who issued the query on the server. In the whole story where does the session on sql server like mysql server fits in and what actually it does? or There is no session concept on Mysql server(any sql server)? am i in wrong imagination?
A session in this context usually just refers to a single client connection.
The client connects to the DB server and authenticates; this is the start of the session.
When the client disconnects (gracefully or not) the session ends.
This is relevant for things like temporary tables or transactions: Un-committed transactions will be rolled back by the DBMS and all temporary tables created through this connection (=session) are discarded when the client disconnects, i.e. when the session ends.
Note that a client does not necessarily actively end a session or connection. The client may crash, or the network connection may break, or the server may shut down &c. Any of this implicitly terminates the session.
Problems may arise when a (client) application uses a connection pool keeping connections (and sessions) open and handing them out transparently to different application components. When not handled correctly, errors may occur because a given session may already be 'spoiled' by a previous operation. If, for example, one routine on the client creates a temporary table named 'X' and fails to explicitly drop it afterwards, the next routine that 'inherits' this session may encounter an error when trying to create another temporary table of that name, because it already exists in this specific session; which couldn't be the case if the connection/session was freshly created.
"Session" is mainly a generic term. You connect to a server (MySQL, Oracle, FTP, IRC... whatever), you do your stuff and finally disconnect when you're done. That has been a session.
HTTP is a particular case. It's a stateless protocol: if you spend an hour reading a web site, you don't remain connected for a whole hour. You make a quick connection, fetch an item at a time (an HTML document, a style sheet, a picture...) and close the connection. (Internals are actually more complex but that's the general idea.) When you ask for a second page, the server doesn't know who you are: that makes it impossible to keep track of your whole browsing session at protocol level. Thus HTTP sessions were invented: they're a way to emulate physical sessions.
The MySQL session starts when you open a connection to the server. A connection ID is assigned which can be read via the SELECT CONNECTION_ID() statement. The session is terminated when the connection is closed or, in case of persistent connections, after a certain timeout or when the server shuts down.
Our mysql hoster has a limit of concurrent db connections. As it is rather pricey to expand that limit the following question came up:
Info:
I have a web app (been developed by an external coder- in case you might wonder about this question).
The web app is distributed and installed on many servers (every user installs it on their pc). These satellites are sending data into a mysql db. Right now the satellites are posting into the db directly.To improve security and error handling i would like to have the satellites posting to a XML-rpc (wordpress api) which then further posts into the db.
Question:
would such api reduce number of concurrent connections or not?
(right now as every satellite connects directly. It is like 1 user = 1 connection)
If 10 satellites are posting to one file, this file then processes the data and posts them into the db -> has this been one connection? (or as many connections as different data sets have been processed.)
What if the api throttles a little bit, so as there is only posting at a time. Would this lead to just one connection or not?
Any pointers are well appreciated!
Thank you in advance!
If you want to improve concurrent connections to the database (because the fact is, creating a connection to the database is "expensive"). You should look into using a ConnectionPool (example with Java).
How are concurrent database connections counted?
(source: iforce.co.nz)
A connectionless server
Uses a connectionless IPC API (e.g., connectionless datagram socket)
Sessions with concurrent clients can be interleaved.
A connection-oriented server
Uses a connection-oriented IPC API (e.g. stream-mode socket )
Sessions with concurrent clients can only be sequential unless the server is threaded.
(Client-server distributed computing paradigm, N.A.)
Design and Performance Issues
Database connections can become a bottleneck. This can be addressed
by using connection pools.
Compiled SQL statements can be re-used by using PreparedStatements
instead of statements. These statements can be parameterized.
Connections are usually not created directly by the servlet but either
created using a factory (DataSource) or obtained from a naming service
(JNDI).
It is important to release connections (close them or return them to the
connection pool). This should be done in a finally clause (so that it is done
in any case). Note that close() also throws an exception!
try
{
Console.WriteLine("Executing the try statement.");
throw new NullReferenceException();
}
catch (NullReferenceException e)
{
Console.WriteLine("{0} Caught exception #1.", e);
}
catch
{
Console.WriteLine("Caught exception #2.");
}
finally
{
Console.WriteLine("Executing finally block.");
}
There are various problems when developing interfaces between OO and
RDBMS. This is called the “paradigm mismatch”. The main problem is that
databases use reference by value while OO languages use reference by
address. So-called middleware/ object persistency framework software
tries to ease this.
Dietrich, R. (2012). Web Application Architecture, Server Side Scripting Servlets. Palmerston North: Massey University.
It depends on the way you implement the centralized service.
If the service after receiving a request immediatly posts the data to mysql, you may have many connections if there are simultaneous requests. But using connection pooling you can control precisely how many open connections you will have. In the limit, you can have just one connection open. This might cause contention if there are many concurrent requests as each request has to wait for the connection to be released.
If the service receives requests, store them in some place (other then the database), and processes them in chunks, you can also have just one connection. But this case is more complex to implement because you have to control the access (reading and writing) to the temporary data buffer.
In order to realize client-side notifications in an AJAX-driven application that I am developing with Grails (and GWT), I have implemented a service method that will block until it is being signaled. I am using a monitor object to wait for a signal. Once signaled, the thread will query the database for new objects and then return the entities to the browser.
It is working perfectly fine with the memory database but not as I expect when I use the MySQL database connector. What happens: whenever I do a findAllBy... call it will only find objects that were created before the request started.
The lifecycle of my service method
request from client
Hibernate session is being created by Grails
service querying database for new objects
if there are none: wait
incoming signal: query database for new objects (DOES NOT GET NEW OBJECTS when using MySQL, works fine with memory db)
The mysql query log shows all the queries as expected but the result of findAllBy... is just an empty array.
I disabled query and second level cache. Behaviour is the same no matter if data connection is pooled or not.
What am I doing wrong? Should I close the Hibernate session? Flush it? Use a transaction for my queries? Or somehow enforce the findAllBy... method to query the database?
Just a guess, but this sounds like a transaction isolation level problem where you are experiencing a phantom read. Does your service need need to be transactional? If not set transactional=false in the service.
I think that you need to flush the session on the save calls for the new objects that you are looking for, e.g.
DomainOfFrequentlyAddedStuff.save(flush:true)
Then they should be persisted to the db quickly so they will show up in your findAll() query.
I've been thinking, why does Apache start a new connection to the MySQL server for each page request? Why doesn't it just keep ONE connection open at all times and send all sql queries through that one connection (obviously with client id attached to each req)?
It cuts down on the handshake time overhead, and a couple of other advantages that I see.
It's like plugging in a computer every time you want to use it. Why go to the outlet each time when you can just leave it plugged in?
MySQL does not support multiple sessions over a single connection.
Oracle, for instance, allows this, and you can setup Apache to mutliplex several logical sessions over a single TCP connection.
This is limitation of MySQL, not Apache or script languages.
There are modules that can do session pooling:
Precreate a number of connections
Pick a free connection on demand
Create additional connections if not free connection is available.
the reason is: it's simpler.
to re-use connections, you have to invent and implement connection pooling. this adds another almost-layer of code that has to be developed, maintained, etc.
plus pooled connections invite a whole other class of bugs that you have to watch out for while developing your application. for example, if you define a user variable but the next user of that connection goes down a code path that branches based on the existence of that variable or not then that user runs the wrong code. other problems include: temporary tables, transaction deadlocks, session variables, etc. all of these become very hard to duplicate because it depends on the subsequent actions of two different users that appear to have no ties to each other.
besides, the connection overhead on a mysql connection is tiny. in my experience, connection pooling does increase the number of users a server can support by very much.
Because that's the purpose of the mod_dbd module.