Issue with Querying Large Table - mysql

Started having the following errors as the size of my database grow. It's at about 4GB now for this table with millions of rows.
Laravel cant handle large tables?
$count = DB::table('table1')->distinct('data')->count(["data"]);
$count2 = DB::table('table2')->distinct('data')->count(["data"]);
SQLSTATE[HY000]: General error: 2014 Cannot execute queries while other unbuffered queries are active. Consider using PDOStatement::fetchAll(). Alternatively, if your code is only ever going to run against mysql, you may enable query buffering by setting the PDO::MYSQL_ATTR_USE_BUFFERED_QUERY attribute. (SQL: select count(distinct data) as aggregate from data)

I think the table is very large and will take some time to finish. You are running another huge query directly after which also needs a long time to fully execute. I think you need to rerun a second query when first finishes.
Please try this:
$count = DB::table('table1')->distinct('data')->count(["data"]);
if($count){
$count2 = DB::table('table2')->distinct('data')->count(["data"]);
}
But this will need a result from first the query.
Try using:
if($count>-1)
or
if(DB::table('table1')->distinct('data')->count(["data"]) >-1){
$count2 = DB::table('table2')->distinct('data')->count(["data"]);
}
For the MySQL statement this might be faster:
DB::table('table1')->select(DB::raw("SELECT COUNT (DISTINCT data)"))->count();
And best solution will be using execute as Laravel allow to use it too
Please tell me if you encounter any problems.

Related

How do you time multiple SQL queries in MySQL workbench?

A program I inherited runs 800 single queries once every minute or so. I was able to grab all these queries and I want to test to see how long it takes to run them in a sequence to see if its an issue that I need to address or if it is ok as is.
The SQL queries are all simple SELECT queries with a few where clauses:
SELECT DISTINCT roles.desc FROM descRoles roles, listUsers users, listUsers mapping WHERE mapping.roleId = roles.roleId AND mapping.idx = users.idx AND users.UserName = 'fakeNameHere';
If there's a typo in my select query please ignore it they run fine. I want to know if there is something I can put before and after all 800 queries to time how long it takes to run all of them. Also, if I could turn off the result tabs on them since after about 40 I get a message that my maximum result tabs are reached, that also seems necessary.
Workbench is not the tool for timing queries. What you want is mysqlslap https://dev.mysql.com/doc/refman/8.0/en/mysqlslap.html

EclipseLink: ConnectionPools and native queries

We are using Spring (Eclipselink) on MariaDB. Our SQL via ORM result in a long lasting DB query. Therefore I need to refine it into a nativequery - which is no big deal by itself. Nevertheless the Resultset is limited by a LIMIT and I need a total counter for all found records. For querying the total counter I found for MariaSQL the following solution.
My question:
Is it save to query the two SQL command separately or should I send them once with a UNION combined?
The question arises due to the fact that between my query and the SELECT FOUND_ROWS() the another query might interfere (with a request from the same Microservice) and dilute the result.
If both queries are done in the same transaction, the MVCC of INNODB should guarantee, that the results will not be influenced by other transactions.
see: https://dev.mysql.com/doc/refman/8.0/en/innodb-multi-versioning.html

MySQL JOIN Keeps Timing Out

I am currently trying to run a JOIN between two tables in a local MySQL database and it's not working. Below is the query, I am even limiting the query to 10 rows just to run a test. After running this query for 15-20 minutes, it tells me "Error Code" 2013. Lost connection to MySQL server during query". My computer is not going to sleep, and I'm not doing anything to interrupt the connection.
SELECT rd_allid.CreateDate, rd_allid.SrceId, adobe.Date, adobe.Id
FROM rd_allid JOIN adobe
ON rd_allid.SrceId = adobe.Id
LIMIT 10
The rd_allid table has 17 million rows of data and the adobe table has 10 million. I know this is a lot, but I have a strong computer. My processor is an i7 6700 3.4GHz and I have 32GB of ram. I'm also running this on a solid state drive.
Any ideas why I cannot run this query?
"Why I cannot run this query?"
There's not enough information to determine definitively what is happening. We can only make guesses and speculations. And offer some suggestions.
I suspect MySQL is attempting to materialize the entire resultset before the LIMIT 10 clause is applied. For this query, there's no optimization for the LIMIT clause.
And we might guess that there is not a suitable index for the JOIN operation, which is causing MySQL to perform a nested loops join.
We also suspect that MySQL is encountering some resource limitation which is causing the session to be terminated. Possibly filling up all space in /tmp (that usually throws an error, something like "invalid/corrupted myisam table '#tmpNNN'", something of that ilk. Or it could be some other resource constraint. Without doing an analysis, we're just guessing.
It's possible MySQL wrote something to the error log (hostname.err). I'd check there.
But whatever condition MySQL is running into (the answer to the question "Why I cannot run this query")
I'm seriously questioning the purpose of the query. Why is that query being run? Why is returning that particular resultset important?
There are several possible queries we could execute. Some of those will run a long time, and some will be much more performant.
One of the best ways to investigate query performance is to use MySQL EXPLAIN. That will show us the query execution plan, revealing the operations that MySQL will perform, and in what order, and indexes will be used.
We can make some suggestions as to some possible indexes to add, based on the query shown e.g. on adobe (id, date).
And we can make some suggestions about modifications to the query (e.g. adding a WHERE clause, using a LEFT JOIN, incorporate inline views, etc. But we don't have enough of a specification to recommend a suitable alternative.
You can try something like:
SELECT rd_allidT.CreateDate, rd_allidT.SrceId, adobe.Date, adobe.Id
FROM
(SELECT CreateDate, SrceId FROM rd_allid ORDER BY SrceId LIMIT 1000) rd_allidT
INNER JOIN
(SELECT Id FROM adobe ORDER BY Id LIMIT 1000) adobeT ON adobeT.id = rd_allidT.SrceId;
This may help you get a faster response times.
Also if you are not interested in all the relation you can also put some WHERE clauses that will be executed before the INNER JOIN making the query faster also.

Fetching large number of records from MySQL through Java

There is a MySQL table, Users on a Server. It has 28 rows and 1 million records (It may increase as well). I want to fetch all rows from this table, do some manipulation on them and then want to add them to MongoDB. I know that it will take lots of time to retrieve these records through simple 'Select * from Users' operation. I have been doing this in Java, JDBC.
So, the options I got from my research is:
Option 1. Do batch processing : My plan was to get the total number of rows from the table, ie. select count(*) from users. Then, set a fetch size of say 1000 (setFetchSize(1000)). After that I was stuck. I did not know if I can write something like this:
Connection conn = DriverManager.getConnection(connectionUrl, userName,passWord);
Statement stmt =conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,java.sql.ResultSet.CONCUR_READ_ONLY);
String query="select * from users";
ResultSet resultSet=stmt.executeQuery(query);
My doubt was whether resultSet will have 1000 entries once I execute the query and should I repeatedly do the operation till all records are retrieved.
I dropped the plan because, I understand that for MySQL, ResultSet is fully populated at once and batching might not work. This stackoverflow discussion and MySQL documentation helped out.
Option 2. Do pagination: My idea is that I will set a Limit which will tell starting index for fetching and offset for fetching. May be, set the offset as 1000 and iterate over the index.
I read a suggested article link, but did not find any loop holes in approaching this problem using Limit.
Anybody who is kind enough and patient enough to read this long post, could you please share your valuable opinions on my thought process and correct me if there is something wrong or missing.
Answering my own question based on the research I did:
Batching is not really effective for select queries, especially if you want to use the resultset of each query operation.
Pagination - Good if you want to improve the memory efficiency, not for improving speed of execution. Speed comes down as you fire multiple queries with Limit, as every time JDBC has to connect to MySQL.

Stop mysql query if it too long

How do I automatically interrupting long queries in the mysql, if it possible?
I'm understand, that I need optimize queries instead of. But now I have access on database server only.
While you are optimizing your queries you can interrupt them by killing the corresponding thread.
As far as I know, you can not stop a query once it has been executed.
I'm understand, that I need optimize
queries instead of.
That's the way you should go.
Other Possibility:
You may want to have a look at SQL Transactions if applicable in your case.
There is no way in mysql to limit how long query can run and then timeout. Have "long query killer" script which catches kills and reports bad queries by eg. looking at SHOW FULL PROCESSLIST time property. You can do it via stored procedure and call it on scheduled time or via application that is connecting to mysql also on scheduled basis.
Try kill long query process time:
$result = mysql_query("SHOW FULL PROCESSLIST");
while ($row=mysql_fetch_array($result)) {
$process_id = $row["Id"];
if ($row["Time"] > 200 ) {
$sql="KILL $process_id";
mysql_query($sql);
}
}