In an ORM you can have nice syntax like this:
my $results = Model.objects.all()[10];
And in the Django ORM it even quite nicely handles foreign key relationships and many to many relationships all through the ORM.
However, in MySQL you can run a query like this:
SELECT t1.column1
, t2.column2
, t3.column3
FROM db1.table AS t1
, db2.table AS t2
, db3.table AS t3
WHERE t1.id = t2.t1_id
AND t1.id = t3.t1_id
LIMIT 0,10
I'm looking for an ORM that can support these types of query natively but can't really see anything that does.
Are there any existing ORMs that can do this? Or are there alternative strategies for tackling this problem?
Whenever I've used a framework like django to build a site, I've kept everything on the same database because I was aware of the limitation. Now I'm working with data that's spread across many different databases, for no apparent reason other than namespacing.
Might be worth looking at something at a lower level than the ORM? For example, something along the lines of C-JDBC provides a 'virtual' database driver that talks to a cluster of databases behind the scenes. (Tables can be distributed across servers)
(I realise you're using Python, so this specific example would only be of use if you could run Jython on the JVM as a platform integrating that way - however I'm guessing similar libraries probably exist closer suited to your specific requirements)
Related
For I/O throughput's sake, we have multiple tables distributed on multiple MySQL nodes. However, I don't want my clients to find these tables over multiple nodes, I wish to make them feel that there is only one MySQL node, though it might be virtual.
For example, we have table t1, t2 on node A; t3, t4 on node B; t5, t6 on node C, and I want the client could connect to a virtual node V that all the tables t1-t6 could be accessed. Actually, my clients only need read access.
Are there any mature solutions to meet this requirement?
After some research, I found that this problem could be solved by MySQL_Proxy.
I can use a LUA script to let the MySQL_Proxy redirect SQL commands to proper destinations and forward the results back.
I'm building a Django app with some pretty straightforward queries. When I run it locally the performance is pretty poor: DebugToolbar reports 4119.00 ms (6 queries) which is obviously pretty bad. The longest-running query is this one:
Django views.py code:
reviews = list(Review.objects.select_related('band', 'record', 'label').order_by('-date_posted')[:12])
Outputted SQL:
SELECT * FROM `reviews`
INNER JOIN `bands` ON (`reviews`.`band_id` = `bands`.`id`)
INNER JOIN `records` ON (`reviews`.`record_id` = `records`.`id`)
INNER JOIN `label` ON (`reviews`.`label_id` = `label`.`id`)
ORDER BY `reviews`.`date_posted` DESC
LIMIT 12
Now obviously there's a few joins there, but my tables are properly indexed. When I run that query in phpMyAdmin I get Query took 0.0241 sec, but in DebugToolbar I see 1838.00 ms. The other 5 queries are similarly sluggish, but run normally when queried directly.
I'm running Django on Windows and using MySQL (WAMP server). Can anyone think of any reason why Django's own queries could be so slow?
Make sure you are using the same database to do the comparisons. If one has a small amount of data, and the other is huge, the timings will change. This sounds like a silly thing to point out, but simpler mistakes have been made.
In Django, set DEBUG=False, so that Django won't store and trace the queries, which could add overhead. I wouldn't expect it to add as much as you're seeing, but you need to make the environment realistic.
This answer is really late but it might help if someone else is looking for a solution. I was facing the same issue on Windows but it got resolved when I disabled caching.
When using MySQL 5.1 Enterprise after years of using other database products like Sybase, Infomix, DB2; I run into things that MySQL just doesn't do. For example, it can only generate an EXPLAIN query plan for SELECT queries.
What other things I should watch out for?
You may take a look at long list here: MySQL Gotchas
Full outer joins. But you can still do a left outer join union right outer join.
One thing I ran into late in a project is that MySQL date types can't store milliseconds. Datetimes and timestamps only resolve to seconds! I can't remember the exact circumstances that this came up but I ended up needing to store an int that could be converted into a date (complete with milliseconds) in my code.
MySQL's JDBC drivers cache results by default, to the point that it will cause your program run out of memory (throw up OutOfMemory exceptions). You can turn it off, but you have to do it by passing some unusual parameters into the statement when you create it:
Statement sx = c.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,java.sql.ResultSet.CONCUR_READ_ONLY);
sx.setFetchSize(Integer.MIN_VALUE);
If you do that, the JDBC driver won't cache results in memory, and you can do huge queries BUT what if you're using an ORM system? You don't create the statements yourself, and therefore you can't turn of caching. Which basically means you're completely screwed if you're ORM system is trying to do something involving a lot of records.
If they had some sense, they would make that configurable by the JDBC URL. Oh well.
Allow for Roles or Groups
It doesn't cost a fortune. Try building a clustered website on multiple machines with multiple cores on an oracle database. ouch.
It still doesn't do CHECK constraints!
I've developed an application at working using MySQL 5, that uses Views to access the major pieces of data. It turns out that our production server uses MySQL 4, which does not have Views included.
Does anybody have a quick and dirty way to deal with this that doesn't involve rewriting all my code?
This certainly points out the importance of using the same technology in your development and production environments!
Workarounds involving triggers or stored procedures won't work, because these are also not supported on MySQL 4.x.
Your options at this point:
Rewrite application code to duplicate data in denormalized tables, designed to match your views.
Upgrade your production database to MySQL 5.0. If you're talking about a hosting provider, then contact that provider and ask if they have an option for MySQL 5.0, otherwise you need to relocate to a provider who does.
I'd recommend the latter path, it'll be far less work than writing code to manage duplicate data.
Note that MySQL 4.1 was released as production software over four years ago. Active support for this release ended in 2006. Extended support for MySQL 4.1 ends 2009-12-31. See http://www.mysql.com/about/legal/lifecycle/
The quick and very dirty way that comes to mind is to subclass DBI and re-write the SQL there. Depends on what you're using views for, of course, and if you mean MySQL 4.0 (does not have subqueries) or MySQL 4.1 (does have subqueries).
If you're on 4.1, you can turn:
CREATE VIEW foo AS
SELECT a, b, c FROM real_table WHERE fooable = 1;
SELECT * FROM foo;
into
SELECT v1.* FROM (
SELECT a, b, c FROM real_table WHERE fooable = 1
) v1;
At least, the latter syntax works in 5.0.x, I think it should in 4.1.x as well.
If you're on 4.0... well, it won't be as easy.
Ouch. Aside from a DeLorean and a flux capacitor or upgrading the server I don't know of any easy way to get around this issue. A lot of change seems necessary.
Unfortunately, without upgrading to MySQL 5, probably not.
Is it possible to do a cross table join in mysql spaning different tables? in different databases.
This seem to be easily possible in MSSQL, and greatly speeds up data transfer?
How about mysql, do you need to use a powerful IDE to achieve this? or do you have to write a program to do something like this?
UPDATE tblUser
SET tblUser.ReceiveInfo=old_database.dbo.contact.third_party_info_yn
FROM tblUser inner join old_database.dbo.contact
ON old_database.dbo.contact.id=tblUser.oldid
Sure, very easy. Prefix the table name with the database name, like you show. We do cross-database joins on a regular basis. For example:
SELECT COUNT(*) FROM newusers1.users
JOIN newusers2.users
It certainly won't speed up data transfer compared to having both tables in the same database, though.
Now, if your tables are sitting on different database servers, the answer is no. An example of this is if your database is too big and you need to shard your tables. Things get more than a little messy. But given that you seem to be happy with the MS SQL solution, that does not seem to apply here.
In MySQL you can do cross DB joins and, by way of the FEDERATED engine, even cross server joins.
MySQL doesn't actually care if the tables are in the same "database", it's just a logical collection of tables for convenient administration, permissions etc.
So you can do a join between them just as easily as if they were in the same one (see ChrisInEdmonton's answer)