I am in a process on converting a legacy system to web app using Ruby on Rails and MySQL.
There are few places that I'm stuck at while converting the data layer to MySQL procedures.
Giving a scenario below;
FUNCTION first_function
SELE Table1
REPL Table1.SmaCode WITH SMA(code,HcPc,FromDate)
ENDFUNC
FUNCTION SMA
... Lot of conditions ...
Lookup(param1,param2) * Parameters are based on the conditions above
.. Lot more conditions ....
ENDFUNC
FUNCTION Lookup
temp = Output of select on Check table
return temp
ENDFUNC
Here SMA is another function which has so many conditions and it also calls another function Lookup. In Lookup function it query a table named Checks, the parameter to Lookup is based on the SMA.
Please see the pastebin of the source code in disucssion, if you need more insight. http://pastebin.com/raw/Hvx3b8zN
How can I go and convert this kind of functions to MySQL procedures?
Edit:
I'm looking for insights on this from people who've already done these types of conversions, from procedure oriented languages to set based stored procedures to be exact.
The commentators are all right and I upticked them all. You have to actually write the code but it's not too hard once you get going.
The first thing I do is to examine my code and rewrite all the straightforward things like DELETE FOR .... into DELETE WHERE...
Then I look at my loops and think about how I can treat that data as a set. A lot of times, SCANs can be written as a regular query when you use appropriate JOIN conditions and WHERE conditions. There are a lot of query tools like CASE and subqueries that let you get a lot done with very little code. MySQL allows temporary tables and that can come in very useful. Lookups can often be done with subqueries.
On occasions, I have to use FETCH and WHILE loops but I avoid that as much as possible because it is slow and SQL is set based.
Just get started on the easy stuff and you'll get the hang of it :)
Related
So I'm kind of stumped.
I have a MySql project that involves a database table that is being manipulated and altered by scripts on a regular basis. This isn't so unusual, but I need to automate a script to run (after hours, when changes aren't happening) that would save the result of the following:
SHOW CREATE TABLE [table-name];
This command generates the ready-to-run script that would create the (empty) table in it's current state.
In SqlWorkbench and Navicat it displays the result of this SHOW command in a field in a result set, as if it was the result of a SELECT statement.
Ideally, I want to take into a variable in a procedure, and change the table name; adding a '-mm-dd-yyyy' to end of it, so I could show the day-to-day changes in the table schema on an active server.
However, I can't seem to be able to do that. Unlike a Select result set, I can't use it like that. I can't get it in a variable, or save it to a temporary, or physical table or anything. I even tried to return this as a value in a function, from which I got the error that a function cannot return a result set - which explains why it's displayed like one in the db clients.
I suspect that this is a security thing in MySql? If so, I can totally understand why and see the dangers exposed to a hacker, but this isn't a public-facing box at all, and I have full root/admin access to it. Hopefully somebody has already tackled this problem before.
This is on MySql 8, btw.
[Edit] After my first initial comments, I need to add; I'm not concerned about the data with this question whatsoever, but rather just these schema changes.
What I'd really -like- to do is this:
SELECT `Create Table` FROM ( SHOW CREATE TABLE carts )
But this seems to be mixing apples and oranges, as SHOW and SELECT aren't created equal, although they both seem to return the same sort of object
You cannot do it in the MySQL stored procedure language.
https://dev.mysql.com/doc/refman/8.0/en/show.html says:
Many MySQL APIs (such as PHP) enable you to treat the result returned from a SHOW statement as you would a result set from a SELECT; see Chapter 29, Connectors and APIs, or your API documentation for more information. In addition, you can work in SQL with results from queries on tables in the INFORMATION_SCHEMA database, which you cannot easily do with results from SHOW statements. See Chapter 26, INFORMATION_SCHEMA Tables.
What is absent from this paragraph is any mention of treating the results of SHOW commands like the results of SELECT queries in other contexts. There is no support for setting a variable to the result of a SHOW command, or using INTO, or running SHOW in a subquery.
So you can capture the result returned by a SHOW command in a client programming language (Java, Python, PHP, etc.), and I suggest you do this.
In theory, all the information used by SHOW CREATE TABLE is accessible in the INFORMATION_SCHEMA tables (mostly TABLES and COLUMNS), but formatting a complete CREATE TABLE statement is a non-trivial exercise, and I wouldn't attempt it. For one thing, there are new features in every release of MySQL, e.g. new data types and table options, etc. So even if you could come up with the right query to produce this output, in a couple of years it would be out of date and it would be a thankless code maintenance chore to update it.
The closest solution I can think of, in pure MySQL, is to regularly clone the table structure (no data), like so:
CREATE TABLE backup_20220618 LIKE my_table;
As far as I know, to get your hands on the full explicit CREATE TABLE statement, as a string, would require the use of an external tool like mysqldump which was designed specifically for that purpose.
I am trying to match two MySQL Queries (for now, the target is "Create VIEW") to analyze if the result of execution would result in the same effect to Database.
The source of the queries is not the same, making the syntax across the queries inconsistent.
To further simplify the question, let me add more details:
Let's say there is an already existing View in the database.
This View was created using a Create VIEW ... SQL statement.
There is a possibility that the Create VIEW ... statement get's updated, hence to reflect the changes in the database currently this statement is executed at the time of migration.
But, I want to avoid this situation, if the statement Create VIEW ... will result in the same structure as of the existing View in the database, I want to avoid executing it.
To generate the CREATE VIEW from database I am using SHOW CREATE VIEW... (comparing this with the query originally used to create the VIEW).
The primary restriction is I need to make this decision only at the time of migration and cannot presume any conclusions (say, using git diff or commit history...).
I have already done some search to look for a solution for this:
Found no direct solution for this problem (like a SQL engine to which I can feed both queries and know if the result would be the same).
Decided to Parse the queries and to achieve that ended up looking into ANTLR (also used by MYSQL WorkBench)
ANTLR's approach looks promising but, this will require an extensive rule-based parsing and creating a query match program from scratch.
I realized that just parsing queries is not enough, I have to create my own POJOs to store the atomic lexers from queries and then compare the queries based on some rules.
Even if I could find predefined POJOs, that would allow to quickly create a solution for this problem.
It's not like that I am having trouble executing my cursors which are enclosed in a stored procedure. But I want to find more efficient way to achieve the same.
Here it goes.
Stored procedure : RawFeed.sql (runs every 5 minutes)
Set #GetATM = Cursor For
Select DeviceCode,ReceivedOn
From RawStatusFeed
Where CRWR=2 AND Processed=0
Order By ReceivedOn Desc
Open #GetATM
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
While ##FETCH_STATUS = 0
Begin
Set #RawFeed=#ATM+' '+Convert(VarChar,#ReceivedOn,121)+' '+'002307'+' '+#ATM+' : Card Reader/Writer - FAULTY '
Exec usp_pushRawDataAndProcess 1,#RawFeed
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
End
Set #GetATM = Cursor For
Select DeviceCode,ReceivedOn
From RawStatusFeed
Where CRWR=0 AND Processed=0
Order By ReceivedOn Desc
Open #GetATM
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
While ##FETCH_STATUS = 0
Begin
Set #RawFeed=#ATM+' '+Convert(Varchar,#ReceivedOn,121)+' '+'002222'+' '+#ATM+' : Card Reader/Writer - OK '
Exec usp_pushRawDataAndProcess 1,#RawFeed
Fetch Next
From #GetATM Into #ATM,#ReceivedOn
End
Likewise I have 10 more SET statements which differ on WHERE condition parameter & string enclosed in #RawFeed variable.
For each row I get I execute another stored procedure on that particular row.
My question is
Is there any better way to achieve the same without using cursors?
Variable #RawFeed Contains following string which is input to usp_pushRawDataAndProcess stored procedure. now this will divide whole string and do some operation like INSERT,UPDATE,DELETE on some tables.
WE JUST CAN NOT PROCESS MORE THAN 1 STRING IN usp_pushRawDataAndProcess
NMAAO226 2012-09-22 16:10:06.123 002073 NMAAO226 : Journal Printer - OK
WMUAO485 2012-09-22 16:10:06.123 002222 WMUAO485 : Card Reader/Writer - OK
SQL Server, like other relational databases, is desgined to, and is pretty good at, working on sets of data.
Databases are not good at procedural code where all the opportunities for optimization are obscured from the query processing engine.
Using RawStatusFeed to store some proprietry request string and then processing a list of those one by one, is going to be ineffiencnt for database code. This might make the inserts very fast for the client, and this might be very important, but it comes at a cost.
If you break the request string down on insert, or better still, before insert via a specialised SP call, then you can store the required changes in some intermediate relational model, rather than a list of strings. Then, every so often, you can process all the changes at once with one call to a stored procedure. Admittedly, it would probably make sense for that stored procedure to contain several query statements. However, with the right indexes and statistics the query processing engine will able to make an efficient execution plan for this new stored procedure.
The exact details of how this should be achieved depend on the exact details of the RawStatusFeed table and the implementation of usp_pushRawDataAndProcess. Although this seems like a rewrite, I don't imagine the DeviceCode column is that complicated.
So, the short answer is certainly yes but, I'd need to know what usp_pushRawDataAndProcess does in detail.
The signature of the usp_pushRawDataAndProcess SP is acting as a bottle neck.
If you can't change usp_pushRawDataAndProcess and and won't create a set based alternative then you are stuck with the bottle neck.
So, rather than removing the bottle neck you could take another tack. Why not make more concurrent instances of the bottle neck to feed the data through.
If you are using SQL Server 2005 or above you could use some CLR to perform numerous instances of usp_pushRawDataAndProcess in parallel.
Here is a link to a project I used before to do something similar.
I had always disliked cursors because of their slow performance. However, I found I didn't fully understand the different types of cursors and that in certain instances, cursors are a viable solution.
When you have a business problem that can only be solved by processing one row at a time, then a cursor is appropriate.
So to improve performance with the cursor, change the type of cursor you are using. Something I didn't know was, if you don't specify which type of cursor you are declaring, you get the Dynamic Optimistic type by default, which is the one that is the slowest for performance because it's doing lots of work under the hood. However, by declaring your cursor as a different type, say a static cursor, it has very good performance.
See these articles for a fuller explanation:
The Truth About Cursors: Part I
The Truth About Cursors: Part II
The Truth About Cursors: Part III
I think the biggest con against cursors is performance, however, not laying out a task in a set based approach would probably rank second. Third would be readability and layout of the tasks as they usually don't have a lot of helpful comments.
The best alternative to a cursor I've found is to rework the logic to take a set based approach.
SQL Server is optimized to run the set based approach. You write the query to return a result set of data, like a join on tables for example, but the SQL Server execution engine determines which join to use: Merge Join, Nested Loop Join, or Hash Join. SQL Server determines the best possible joining algorithm based upon the participating columns, data volume, indexing structure, and the set of values in the participating columns. So it generally the best approach in performance over the procedural cursor approach.
Here is an article on Cursors and how to avoid them. It also discusses the alternatives to cursors.
Alernates for CURSOR in SQL server
1.While loop
2.Recursive CTE
Alernates for CURSOR in SQL server
1. Use temp table. create any column ID as identity column.
2. Use while loop to perform the operation.
My program restores a MySQL database from SQL file. If I wanted to display progress of SQL execution in my program, I would need to know the number of SQL statements in the file. How can I do this in MySQL? (The queries may consist of mysql specific multi-row insert statements)
I could use either MySQL command line tools or the Python API. You're welcome to post solutions for other DBMS too.
The simple (and easy) way: Add PRINT statements to your SQL script file, displaying progess messages.
The advantage (apart from the obvious 'it's hard to parse multi-statement constructs') is that you get precise control over the progress. For example, some statements might take much longer to run than others so you would need to weight them.
I wouldn't think of progress in terms of number of statements executed. What I do is print out feedback that specific tasks have been started and completed, such as 'Synchronising Table 'blah'', 'Updating Stored Procedure X' etc
The naive solution is to count the number of semicolons in the file (or any other character used as delimited in the file).
It usually works pretty well, except when the data you are inserting has many semicolons and then you have to start dealing with actual parsing of the SQLs, which is a headache.
What is the best thing to approach something like:
select * from (show tables like "T_DATA___") // Invalid
There are over 600 tables with the name T_DATAxy where x and y are letters
Something went seriously wrong with this design. Accessing 600 tables at once means accessing as much as 1800 files on disk. You should've partitioned this data instead.
As far as th question goes, Im afraid that you will need to use a stored procedure or external application, to build a multiple UNION query statement. Still, I seem to remember that there's a limit of 32 tables merged in a UNION.
You could get the list of tables whose data you want (show tables like __) and then use mysql dump, passing in that list.
http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html
If you are determined to get it from SQL queries, you could generate appropriate sql queries using macros and execute them all at once. e.g. get the list of tables, replace newline with "; (newline) select * from ", execute all queries. (The emacs mysql mode makes this super easy).
As the other commenter says, you won't be able to do it in a single query due to #-table limits.