I have a MySQL table that grows very rapidly. I would like to have an ongoing process (e.g. running every 30 minutes) that selects all the rows that are "new" since the last time the process was run. The table has an AUTO_INCREMENT column called id, and I thought I could use it to make sure I'm getting all the records. Here was the approach:
Initialize
LAST_ID_SELECTED = 0
Run repeatedly, forever:
startID = 1 + LAST_ID_SELECTED
nextAutoIncrementID = getQuery("SELECT AUTO_INCREMENT FROM information_schema.tables WHERE TABLE_NAME = 'myTable'")
lastID = nextAutoIncrementID - 1
data = SELECT * FROM myTable WHERE id BETWEEN startID AND lastID
LAST_ID_SELECTED = lastID
However, I don't think this will actually work: I'm worried that I can get my (startID, lastID) range and make my select query before all the inserts for the ids <= lastID have been completed. So I won't actually get all the rows in the range (startID,lastID).
What's the right way to do this?
Related
I have this query
SELECT * FROM outbox where Status=0 ;
then I need to update the selected records so Status should be equal 1
i.e (UPDATE outbox(selected records from SELECT query) SET Status =1 )
any help ?
This is a much harder problem than it sounds. Yes, in the simplistic case where you are only thinking of one user and a few records, it seems easy. But, databases are designed to be ACID-compliant, with multiple users and multiple concurrent transactions that can all be affecting the data at the same time. And there is no single statement in MySQL that does what you want (other databases support an OUTPUT clause, RETURNING or something similar).
One structure that will work in MySQL is to place the items in a temporary table, then do the update, then return them. The following shows the semantics using transactions:
start transaction;
create temporary table TempOutboxStatus0 as
select *
from outbox
where status = 0;
update outbox o
set status = 1
where status = 0;
select *
from TempOutboxStatus0;
commit;
For the update, I actually prefer:
where exists (select 1 from TempOutboxStatus0 t where t.outboxid = o.outboxid);
because its intention is clearer -- and the code is safer in case the conditions subtly change.
Note: you may want to use explicit table locks. Such considerations depend on the storage engine you are using.
BEGIN
Start transaction;
SELECT *
FROM
outbox
where
Status = 0 and
Is_Expired = 0 and
Service_ID=p_service_id
order by
Next_Try_Date asc FOR Update;
update outbox
set
Status=1
where
Status = 0 and
Is_Expired = 0 and
Service_ID=p_service_id;
commit;
END
is this possible .. it seems it works with me
You can do something like that, the outbox is your table:
update outbox set Status = 1 where Status = 0
you can do it like below
$sql=mysql_query("SELECT * FROM outbox where `Status`=0");
while($result=mysql_fetch_array($sql))
{
$update="UPDATE `outbox` SET `Status` =1 where
'your column name'='your previous fetched value');
}
I have a query running on a large table that worked when I did limit 100. When I remove the limit I get:
[Err] 126 - Incorrect key file for table '/tmp/#sql_5e2d_6.MYI'; try
to repair it
I checked with server admin basically the /tmp file fills up quickly.
Is there a way to set up the query to flush the table as it goes along? Or run say 100 records, stop, re-run? The query is pretty simple:
select distinct a,
min(b) N_b
from K
group by a;
At the end of the day what I am trying to do is delete from a large table duplicate records, keeping the record with the lowest value in b. This was the initial select statement.
You can use something like,
*** Edit according to the language you use.
$sql = true;
int i = 0;
While($sql){
$sql = "select distinct a,
min(b) N_b
from K
group by a LIMIT i, i+99";
//Do whatever you want.
i=i+100;
}
I have a table with the columns :
pos1 pos2 pos3 pos4 pos5 id
pos1-5 -> varchar , id -> int
I'm using the query:
Query = "select distinct * from (SELECT * FROM database.turkey2 t1 WHERE pos2='"+comboBox2.Text+"' and NOT EXISTS (SELECT 1 FROM database.turkey2used t2 WHERE t1.id = t2.id)) as t3 order by rand() limit "+amount+" ;";
Meaning the user will choose pos2(will make it static) while the rest is random , the amount will be set by the user as well.
What I'm trying to do is to add a condition in this query that force at least 2 diffrent pos(positions) that must be chosen in random.
meaning , I don't want to get 2 or more rows with the same valuse at pos1,2,3,4 and only pos5 will be diffrent(igonore the id - and the rule dosen't apply only to pose 5 of course).
I solved the problem by runnning another query on the already select amount - but that's not good because if the user asked for 200 combinitions then after the "fix" he may lose some rows.
some other info - it's a medium size db (about 10m rows).
Please keep the solution simple because I'm rather new to mysql and c#.
Thanks.
Are there any solutions on MySQL script to filter the results with specific interval number.
For example, if I have 100,000 records in database and I'd like to get only the record number 1000, 2000, 3000, etc. (step by 1000).
I could do this on server side script by getting the entire results (e.g. 100,000) and use syntax like:
for($i=0, $i <= 100,000, $i = $i+1000) $filterResult[] = $record[$i];
However, as you may see, it would pull stress to the system as 100,000 records will need to generated first.
Are there any solutions that could complete from database script? Please note that, primary key may not start with 1 - 100,000 as the results based on some condition in where clause.
Your help would be really appreciated.
You can do:
SELECT *
FROM tbl
WHERE id % 1000 = 0
But it seems like you don't want to rely on the primary key value, but rather the row ranking of a result set.
In that case, you can do:
SELECT *
FROM (
SELECT *, #rn:=#rn+1 AS rank
FROM tbl
CROSS JOIN (SELECT #rn:=0) var_init
WHERE column1 = value AND
column2 = value
) a
WHERE a.rank % 1000 = 0
Where column1 = value AND column2 = value is just a placeholder for whatever filtration you're doing in your query.
I have a mysql table of data and I need to only return the rows that do not have a status of "Deactive" and do not have a Total of 0 (but there can be rows where status is deactive and total is not 0 and vice versa). Before I needed this requirement I was just doing the standard query to select all rows:
SELECT * FROM tableName WHERE uid = id;
However now when I change the query to add the constraints above:
SELECT * FROM tableName WHERE uid = id AND (status != "Deactive" OR Total != 0);
This bottom query is taking much much longer to complete. Why is this happening and is there a better way I can do this query?
The first query is looking up based on an index (I'm assuming by 'uid'). The second query is filtering on other values. Run this, and it will help you figure out how you can best optimize it:
EXPLAIN EXTENDED SELECT * FROM tableName WHERE uid = id AND status != "Deactive" OR Total != 0;
It's dirty, but this would probably be a quick way to speed it up:
SELECT
*
FROM
(
SELECT
*
FROM
tableName
WHERE
uid = id
) as temp
WHERE
temp.status != "Deactive"
OR temp.Total != 0;
This would force the query to just get the rows with a matching uid first, and then filter them down, instead of having to do a big table scan. Like I said before though, EXPLAIN EXTENDED is your friend
Do you need to return all the data in each row? ie picking only the columns you need will make the query run faster.
You also say you need to 'return the rows that do not have a status of "Deactive" and do not have a Total of 0'. Your query should then read:
SELECT * (or column names) FROM tableName WHERE uid = id AND status != "Deactive" AND Total != 0;