I am trying to log the tables that a particular query might affect inside a DB wrapper function where all the DB calls will pass through.
I can't create a view or a transaction and execute the query and infer it from the resultset since it will increase the application performance time which I can't afford.
Instead I am trying to infer it from the query itself like table names will be the next word after 'FROM,JOIN,INSERT INTO' clauses. Is there any other places table names might be used in a MYSQL query?
Related
Getting ready to get rid of a MySQL database and switch to Oracle SQL. I am using Oracle SQL Developer. Need to get the records from a MySQL table and populate its corresponding table in SQL.
I was able to establish a Database connection in SQL Developer to the MySQL database. I checked the connection by doing a simple SELECT * from the table to make sure it returned all the records.
However, the new Oracle SQL table has quite a few changes - the names in the MySQL table all had a "tn" prefix, ie tnStore, tnConfigDate, etc. The SQL table gets rid of that prefix. That is issue #1.
There will also be several new columns in the new table. That data will be added later from elsewhere. And the data will not be in the same order as the MySQL table.
How do a write up a SELECT INTO statement in SQL Developer to populate the SQL table with the data from the MySQL table and correlate the corresponding columns while leaving new fields blank for now?
Here is a way by programming but not sure how to make it in single query:
I hope we need to use data dictionary tables in oracle all_tab_columns and I am not sure in Mysql ( like similar table)
Get the column name from Mysql table by taking out prefix "tn" and
compare the column name with SQL table. (possible use an cusrsor)
If matched build SQL statement for SELECT INTO statement and blank
for new fields possibly in a loop.
Once done for all columns , execute that statement
Consider migrating the existing MySQL tables as-is straight to Oracle using SQL Developer. Then move/refactor the data around to your new-tables with desired column definitions using INSERT as SELECTs.
Could be considerably faster, plus once the 'raw' data is there, you can do your work over and over again, until you get it just right.
Note you can also simply drag-and-drop to move a MySQL table from it's connection to an existing Oracle database connection to move the table over (DDL, Data, or Both).
I am developing high load web-application and trying to reduce the quantity of sql queries. Quite often I need to update one row and get the results. I think it would be nice to have possibility to run query and at the same time receive the values of updated fields without making 2 calls of mysql server. For example, I execute the following query:
update table set val=val+1 where id=1;
and function returns:
array("val"=>10)
Sure, I understand, that I can write my own function, which first makes update, than select and returns the result. But the problem is that in such case mysql server will have to seek data, update during first query, than comes the second query, which again requires to seek data and return it. And I am thinking about the way, where mysql seeks data, updates and returns updated data.
I have one sql function, let's say theFunction(item_id). It takes an item id and computes one value as its return. I read one table from the DB and I suppose to compute a new value to append for each row by this function given the item_id particular to taht row. Which desing block would do this form me with the following SQL (if not wrong).
select thsFunction(item_id);
I assume that the block gives me item_id of each row as a variable.
You can use another table input step, and have it accept fields from previous steps and execute for every row (both config options are at the bottom of the step's window).
Beware that this is a rather slow implementation. Each query is executed separately and as such each row requires a round trip to the database.
Alternatively, you can use the Row SQL Script. I believe it allows you to pass all SQL statements in a single trip to the database.
An SQL function is probably much more efficient to run in the database, for all rows at once, in stead of making a separate call into the database from PDI for each row to execute the function. So if performance is at all a relevant concern, I'd suggest a whole different strategy:
Write your rows to a table in the database. End your transformation here.
On the job level, first execute your transformation from above, then execute the function in an "Execute SQL script..." component, giving it an SQL command somewhat like "UPDATE my_temp_table set target_col = theFunction(item_id)".
Continue your job with the remaining steps in a new transformation, starting from that table as input.
This of course presupposes that you don't have too many other threads going on, but if your transofrmation is simple and linear -- or at least if it can be made single-linear at this particular step -- it may be possible to split it up into two parts before and after this SQL call.
I have a complicated SELECT query that filters on a time range, and I want this time range (start and end dates) to be specifiable using user-supplied parameters. So I can use a stored procedure to do this, and the return is a multiple-row result set. The problem I'm having is how to deal with this result set afterwards. I can't do something like:
SELECT * FROM (CALL stored_procedure(start_time, end_time))
even though the stored procedure is just a SELECT that takes parameters. Server-side prepared statement also don't work (and they're not persistent either).
Some suggest using temporary tables; the reason that's not an ideal solution is that 1) I don't want to specify the table schema and it seems that you have to, and 2) the lifetime of the temporary table would only be limited to a invocation of the query, it doesn't need to persist beyond that.
So to recap, I want something like a persistent prepared statement server-side, whose return is a result set that MySQL can manipulate as if it was a subquery. Any ideas? Thanks.
By the way, I'm using MySQL 5.0. I know it's a pretty old version, but this feature doesn't seem to exist in any more recent version. I'm not sure whether SELECT-ing from a stored procedure is possible in other SQL engines; switching is not an option at the moment, but I'd like to know whether it's possible anyway, in case we decide to switch in the future.
Selecting from functions is possible in other engines. For instance, Oracle allows you to write a function that returns a table of user defined type. You can define result sets in the function, fill them by using queries or even using a combination of selects and code. Eventually, the result set can be returned from the function, and you can continue to query on that by using:
select * from table(FunctionToBeCalls(parameters));
The only disadvantage, is that this result set is not indexed, so it might be slow if the function is used within a complex query.
In MySQL nothing like this is possible. There is no way to use a result set from a procedure directly in a select query. You can return single values from a function and you can use OUT or INOUT parameters to you procedure to return values from.
But entire result sets is not possible. Filling a temporary table within you procedure is the closest you will get.
So here is my situation: I have a vendor supplied DB we cannot modify and a custom db that imports data from the vendor app and acts on it. Once records are imported form the vendor app, they cannot appear on the list of records to be imported. Also we only want to display the 250 most recent records that have not been imported.
What I originally started with was select the list of ids that have been imported from the custom db, and then query the vendor db, using the list of ids in a .Where(x => !idList.Contains(x.Id)) clause on the remote query.
This worked up until we broke 2100 records imported into the custom db, as 2100 is the limit on the number of parameters that can be passed into SQL. After finding out this was the actual problem and not the 'invalid buffer'/'severe error' ADO.Net reported, my solution was to remove the first 2000 ids in the remote query, and then remove the remaining records in the local query.
Having to pull back a large number of irrelevant records, just to exclude them, so I can get the correct 250 records seems very inelegant. Is there a better way to do this, short of doing a cross db stored procedure?
Thanks in advance.
This might not be the best answer, depending on how many records you're dealing with, but you could force the SQL to execute and just deal with it as in-memory objects. Calling the ToList() method will execute the SQL and convert to an IEnumerable .
What I might suggest is to have started by querying the vendor database first ordering the results by some kind of criteria (perhaps a date field, oldest to most recent).
You could do a Skip().Take() to "skim" the results and then take each bulk set and insert them into the custom db where the ID doesn't already exist. That way you avoid the problem you have now.
If you have db-create access to the SQL Server that the vendor's db is running on (or if your custom db is on the same server), you could create a "has been imported" table in a different database on that same server, and then write a stored proc that does a cross-database join of that table against the vendor db, e.g.:
select top 250 from vendordb.to_be_imported
where not exists
(select 1 from customdb.has_been_imported where idWasImported = idToBeImported)
order by whatever;
You might even be able to do this in Linq 2 SQL -- I've never tried adding objects from different databases into a single DataContext...