I am writing a systems application that has to integrate into an existing older architecture. In order to do this, I have to access a bitmask field in a table; something like so:
SELECT * FROM directory WHERE (status & 64) | (status & 256);
Our existing system runs on MySQL -- and we have a statement similar to the above that works just fine.
However, in my new application which I have to integrate with the existing system, I am using embedded-HSQL in my unit-tests. And for the life of me I cannot figure out how to do bitwise operations in HSQL. Furthermore, even if I am able to figure it out, I am starting to worry that there is not a single statement compatible between both SQL engines.
Any tips on how to go about this? At the moment I'm thinking I'll have to just select everything where status != 0 (limiting the result-set, of course) and then use java to pick-out the specific ones I want that match the status's I'm targeting. Yikes.
These operations are done using functions in HSQLDB.
http://hsqldb.org/doc/2.0/guide/builtinfunctions-chapt.html#bfc_numeric_functions
See BITOR, BITXOR, BITAND, BITNOT, BITANDNOT functions.
Bitwise operators are not very common in SQL dialects. MySQL is an exception rather than the norm.
I never found a better solution to this than just making the effort to replace my embedded database from HSQL to MYSQL.
I also took the time to write better unit tests; ie: exercise my database logic against an actual database, but all other layers use mock database logic.
So that's my take away... run your DDL tests against the same database you are using in production and run all your other logic against DDL mocks.
For posterity, here's what I used to build my embedded MYSQL:
http://zhentao-li.blogspot.com/2013/06/using-embedded-mysql-database-for-unit.html?m=1
Related
I have a long and complex MySql statement that inserts records into a table. However it has gotten so complex that it takes forever on my 2k records table (I killed it after 22,000 seconds, it used to take 500s).
As a test I try running it in 100 records loops e.g. While ID<101 and ID<200.
So I figured I'd just set up some loop to step through 100 records at a time until it has inserted all the records.
I don't want/need to set up a procedure, I'm just wondering if there is some syntax to put into the statement itself vs creating a stored procedure which every example seems to include.
No, there is no such syntax in MySQL. If you need to write loops or other programming constructs, the assumption is that you will write code in some client application. You can choose whatever language you find most productive in. I'd use Python for quick single-use scripts, and probably Go for tools that I want to be used many times.
From its earliest days, SQL was intended to be a data-manipulation language, with integration in other programming languages to provide more general-purpose programming constructs.
MariaDB is a different database product that started as a fork of MySQL. They have introduced syntax to use compound statements outside of stored routines (https://mariadb.com/kb/en/using-compound-statements-outside-of-stored-programs/). But I can't see the appeal. If you're more comfortable using a conventional programming language, then do that.
we do not have a database schema versioning yet in our system, just wanted to know if there is way to check all the sql queries available in our system for their schema correctness (not syntactical correctness).
Requirement is to check that tables / columns mentioned in the query are available in the current database version, so that errors can be found out while starting the service.
We are using mysql and mybatis.
In your mapper file, when defining all of your SQL statements, you could create an equivalent one of the same name with Verify (or similar) appended to the name which checks the information_schema to determine present columns. You could then call all of these Verify queries during startup to see if they complete.
Another approach (which I think may ultimately be cleaner and potentially more thorough) would be setup a series of tests using TestNG or Spock or similar (I prefer Spock, although that uses Groovy, so may not be ideal for your situation) which run all of the mapper statements involved.
If you get an exception in the test because one or more columns aren't present, there will be an exception from the JDBC driver, and so the test will fail. In addition, if as part of the tests you add the expected results of the queries, you will also test the accuracy/behavior of the queries, so you will be able to notice if that later changes (and causes test failures) if the code changes.
Straight to the Qeustion ->.
The problem : To do async bulk inserts (not necessary bulk, if MySql can Handle it) using Node.js (coming form a .NET and PHP background)
Example :
Assume i have 40(adjustable) functions doing some work(async) and each adding a record in the Table after its single iteration, now it is very probable that at the same time more than one function makes an insertion call. Can MySql handle it that ways directly?, considering there is going to be an Auto-update field.
In C#(.NET) i would have used a dataTable to contain all the rows from each function and in the end bulk-insert the dataTable into the database Table. and launch many threads for each function.
What approach will you suggest in this case,
Shall the approach change in case i need to handle 10,000 or 4 million rows per table?
ALso The DB schema is not going to change, will MongoDB be a better choice for this?
I am new to Node, NoSql and in the noob learning phase at the moment. So if you can provide some explanation to your answer, it would be awesome.
Thanks.
EDIT :
Answer : Neither MySql or MongoDB support any sort of Bulk insert, under the hood it is just a foreach loop.
Both of them are capable of handling a large number of connections simultanously, the performance will largely depend on you requirement and production environment.
1) in MySql queries are executed sequentially per connection. If you are using one connection, your 40~ functions will result in 40 queries enqueued (via explicit queue in mysql library, your code or system queue based on syncronisation primitives), not necessarily in the same order you started 40 functions. MySQL won't have any race conditions problems with auto-update fields in that case
2) if you really want to execute 40 queries in parallel you need to open 40 connections to MySQL (which is not a good idea from performance point of view, but again, Mysql is designed to handle auto-increments correctly for multiple clients)
3) There is no special bulk insert command in the Mysql protocol on the wire level, any library exposing bulk insert api in fact just doing long 'insert ... values' query.
When using MySQL 5.1 Enterprise after years of using other database products like Sybase, Infomix, DB2; I run into things that MySQL just doesn't do. For example, it can only generate an EXPLAIN query plan for SELECT queries.
What other things I should watch out for?
You may take a look at long list here: MySQL Gotchas
Full outer joins. But you can still do a left outer join union right outer join.
One thing I ran into late in a project is that MySQL date types can't store milliseconds. Datetimes and timestamps only resolve to seconds! I can't remember the exact circumstances that this came up but I ended up needing to store an int that could be converted into a date (complete with milliseconds) in my code.
MySQL's JDBC drivers cache results by default, to the point that it will cause your program run out of memory (throw up OutOfMemory exceptions). You can turn it off, but you have to do it by passing some unusual parameters into the statement when you create it:
Statement sx = c.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,java.sql.ResultSet.CONCUR_READ_ONLY);
sx.setFetchSize(Integer.MIN_VALUE);
If you do that, the JDBC driver won't cache results in memory, and you can do huge queries BUT what if you're using an ORM system? You don't create the statements yourself, and therefore you can't turn of caching. Which basically means you're completely screwed if you're ORM system is trying to do something involving a lot of records.
If they had some sense, they would make that configurable by the JDBC URL. Oh well.
Allow for Roles or Groups
It doesn't cost a fortune. Try building a clustered website on multiple machines with multiple cores on an oracle database. ouch.
It still doesn't do CHECK constraints!
I've developed an application at working using MySQL 5, that uses Views to access the major pieces of data. It turns out that our production server uses MySQL 4, which does not have Views included.
Does anybody have a quick and dirty way to deal with this that doesn't involve rewriting all my code?
This certainly points out the importance of using the same technology in your development and production environments!
Workarounds involving triggers or stored procedures won't work, because these are also not supported on MySQL 4.x.
Your options at this point:
Rewrite application code to duplicate data in denormalized tables, designed to match your views.
Upgrade your production database to MySQL 5.0. If you're talking about a hosting provider, then contact that provider and ask if they have an option for MySQL 5.0, otherwise you need to relocate to a provider who does.
I'd recommend the latter path, it'll be far less work than writing code to manage duplicate data.
Note that MySQL 4.1 was released as production software over four years ago. Active support for this release ended in 2006. Extended support for MySQL 4.1 ends 2009-12-31. See http://www.mysql.com/about/legal/lifecycle/
The quick and very dirty way that comes to mind is to subclass DBI and re-write the SQL there. Depends on what you're using views for, of course, and if you mean MySQL 4.0 (does not have subqueries) or MySQL 4.1 (does have subqueries).
If you're on 4.1, you can turn:
CREATE VIEW foo AS
SELECT a, b, c FROM real_table WHERE fooable = 1;
SELECT * FROM foo;
into
SELECT v1.* FROM (
SELECT a, b, c FROM real_table WHERE fooable = 1
) v1;
At least, the latter syntax works in 5.0.x, I think it should in 4.1.x as well.
If you're on 4.0... well, it won't be as easy.
Ouch. Aside from a DeLorean and a flux capacitor or upgrading the server I don't know of any easy way to get around this issue. A lot of change seems necessary.
Unfortunately, without upgrading to MySQL 5, probably not.