I need to insert rows of data in a specific order. Sometimes I forget to insert the row on time and I have to insert it later. Other rows have taken up its place though and till now I manually (programmatically of course) change the index of different number of rows - it could be a couple of rows or hundreds of rows. This is not very efficient and I was looking for another way to go. My thought was to order by date and create a "day's index" number to reorder only the day's records but I was wandering... is there any mysql way to reorder the rows? That is to inform mysql about the required row position and then let it update the primary keys?
I think you need to look at your table design. This is actually a non-problem for most applications because it would have been addressed at the start.
Now, you need to add a DateTime column to your table, and initialise it with some sensible values for the data that's already there. As new rows are added, set the DateTime column in each new row to the actual DateTime. If you have to add a row late, set the DateTime to the time the record should have been added.
When you query your table, use ORDER BY myDateTime (or whatever you decide to call it). Your rows should appear in the correct order.
For small tables (less than a few thousand rows) an index might not help much. For larger tables you should index your DateTime column. You'd have to run some tests to see what works best.
What you think is actually the solution. Create a Date column if not already, and then Create Index on that field, also use Order by in your Query. There is no way other than manual, and even if there is it is not recommended to play with MYSQL way of storing rows, because row storage is done by DB Engine and it is not ideal to play with them, as they store row in best optimal way, so why mess their efficiency for such a small thing.
Related
rails app, I have a table, the data already has hundreds of millions of records, I'm going to split the table to multiple tables, this can speed up the read and write.
I found this gem octopus, but he is a master/slave, I just want to split the big table.
or what can I do when the table too big?
Theoretically, a properly designed table with just the right indexes will be able to handle very large tables quite easily. As the table grows the slow down in queries and insertion of new records is supposed to be negligible. But in practice we find that it doesn't always work that way! However the solution definitely isn't to split the table into two. The solution is to partition.
Partitioning takes this notion a step further, by enabling you to
distribute portions of individual tables across a file system
according to rules which you can set largely as needed. In effect,
different portions of a table are stored as separate tables in
different locations. The user-selected rule by which the division of
data is accomplished is known as a partitioning function, which in
MySQL can be the modulus, simple matching against a set of ranges or
value lists, an internal hashing function, or a linear hashing
function.
If you merely split a table your code is going to become inifinitely more complicated, each time you do an insert or a retrieval you need to figure out which split you should run that query on. When you use partitions, mysql takes care of that detail for you an as far as the application is concerned it's still one table.
Do you have an ID on each row? If the answer is yes, you could do something like:
CREATE TABLE table2 AS (SELECT * FROM table1 WHERE id >= (SELECT COUNT(*) FROM table1)/2);
The above statement creates a new table with half of the records from table1.
I don't know if you've already tried, but an index should help in speed for a big table.
CREATE INDEX index_name ON table1 (id)
Note: if you created the table using unique constraint or primary key, there's already an index.
Does anyone answer me why MySQL table raw id not serially?
MySQL will try to give you the results as quick as it can, based on your query.
If you didn't tell MySQL to sort on any field, MySQL will probably pick the order in which it's sorted on the disk.
Records in mysql aren't always stored in order on disk. An example where they might go out of order is if you delete a record in the middle of the table. The next record might (for space saving reasons) be inserted in the position where you deleted a record earlier.
Don't worry about this. This is normal. If you create an application that uses MySQL, make sure you include ORDER BY id at the end of your query to get your predictable order.
If you don't use an ORDER BY clause to make your desired order explicit, the query results are up to the implementation. That means MySQL gets to choose the order if you don't.
In the case of MyISAM tables, the default order is the order rows are stored in the table, which can get mixed up over time as rows are added and deleted. New rows may fit into gaps left by deleted rows, even if that makes them stored "out of order".
In the case of InnoDB tables, the default order is by the index used to read the rows. This is often the primary key, but it might not be. It depends on your table definition and the SQL query you use to read the rows.
Just use ORDER BY if you want the rows ordered by a specific column.
I have a large table with more than 200,000 rows that I only need to check the last few thousand rows for duplicates (not all) before I insert a new row into. Currently I'm running this query for each row I want to add:
SELECT ID from table where date='' and time=''
And based on the response from that query I write the row if the response is empty.
The issue I have with doing this is that it takes a very long time, and as the database grows this only increases how long it takes.
I tried using LIMIT and OFFSET by saying SELECT ID from table where date='' and time='' limit 200000,18446744073709551615 which I thought would only search through rows after 200,000 to the end of the database however running this query doesn't seem to be any faster.
My question is this: Is there a more efficient way to "skip ahead" in the database and only search a portion of the rows instead of all of the rows?
You should be using INSERT IGNORE, and using a UNIQUE constraint on the table based on the columns that should be unique.
When using INSERT IGNORE, MySQL will automatically detect if the row is unique, and ignore the entry into the database. See this question for more information.
Additionally, searching a multi-million row database should be fast as long as you have the correct indexes on the table. You should not need to search a sub-set of data (Without keys, the database will be forced to do a row-scan, which could cause the delays you're talking about).
See this post for some additional ideas.
See also Avoiding Full Table Scans.
I need to optimize a query fetching all the deals in a certain country before with access by users before a certain datetime a certain time.
My plan is to implement the following index
add_index(:deals, [:country_id, :last_user_access_datetime])
I am doubting the relevance and efficientness of this index as the column last_user_access_datetime can have ANY value of date ex: 13/09/2015 3:06pm and it will change very often (updated each time a user access it).
That makes an infinite number of values to be indexed if I use this index?
Should I do it or avoid using 'infinite vlaues possible column such as a totally free datetime column inside an index ?
If you have a query like this:
select t.
from table t
where t.country_id = :country_id and t.last_user_access_datetime >= :some_datetime;
Then the best index is the one you propose.
If you have a heavy load on the machine in terms of accesses (and think many accesses per second), then maintaining the index can become a burden on the machine. Of course, you are updating the last access date time value anyway, so you are already incurring overhead.
The number of possible values does not have an effect on the value. A database cannot store an "infinite" number of values (at least on any hardware currently available), so I'm not sure what your concern is.
The index will be used. Time for UPDATE and INSERT statements just take that much longer, because the index is updated each time also. For tables with much more UPDATE/INSERT than SELECTs, it may not be fruitful to index the column. Or, you may want to make an index that looks more like the types of queries that are hitting the table. Include the IDs and timestamps that are in the SELECT clause. Include the IDs and timestamps that are in the WHERE clause. etc.
Also, if a table has a lot of DELETEs, a lot of indices can slow down operations a lot.
I have 4 databases with tables having lots of data. My requirement is to show count of all the records in these tables on mouse hovering the corresponding div in UI(It is an asp.net website). Please note the count may change in every minute or in hour. (Means new records can be added or deleted from the table [using another application]). Now the issue is like, it is taking lot of time to get the count (since it has lots of data). So each mouse over, it is having a call to corresponding database and taking the count. Is there any better approach to implement this?
I am thinking of implementing something like as below.
http://www.worldometers.info/world-population/
But to change the figures like that in each second I need to have a call to the database. Right? (To get the latest count) Is there any better approach to show data like this statistics?
By the Way, I am using MySQL.
Thanks
You need to give more details - what table engines you are using, how does your count query look like, etc.
But assuming that you are using InnoDB, and you are trying to run count(*) or count(primary_id_column), you have to remember that InnoDB has clustered primary keys, that are stored with the data pages of the row itself, not in separate index pages, so the count will do full scan on the rows.
One thing you can try is to create additional, separate, not primary index on any unique column (like row's id etc,) and make sure (use explain query statement) that your count uses this index.
If this does not work for you, I would suggest to create separate table (for example with columns: table_name, row_count) to store counters in it and create triggers on insert and on delete on other tables (you need to count records in) to increment or decrement these values. From my experience (we monitor number of records in daily and hourly manners, on tables with hundreds of milions of records and heavy write load: ~150 inserts/sec) this is the best solution I have came up so far.