So i have a mysql table that contains records with ip and error type for that IP.
This table has millions of records every 15 minutes so I need to be able to truncate it before each load. All I need to keep is a running total of how many of each error type per IP throughout the month.
I researched and saw that I need to do a summary table of this but I cannot seem to figure out how to keep a running total of the data after i delete the raw records from the original table.
Let me know if I need to include anything, i'm sure this is an easy request but I'm still newish to mysql. Thank you in advance for all the help!
use this:
SELECT COUNT(t.ErrorType) as 'ErrorTypesPerIp', t.Ip
FROM TableName as t
-- WHERE Month >= 'last_month_date'
GROUP BY t.Ip
I am working on an application that awards the top person of each category for being first. The way you become first in a category is by having the most number of votes in the past 30 (or so) days. So even if you had a total of 2,000 votes but got only 2 votes within the past 30 days, someone with 10 votes but got all 10 within the past 30 days would be ranked above you. I am just trying to seek advise on the best way to create this type of system with a MySQL database and how to structure the database.
I am pretty unsure of the best way to go about this, any advice would be greatly appreciated!
The first desicion you have to make is, whether you want to keep a record for every vote cast: This has the potential for a huge table, but it lets you keep a lot of information, so you trade storage and performance against information. This must be answered by business logic, not implementation.
Assuming you DO want to keep every vote, keep it with a timestamp and the only thing you have to do is to join the user person table with the vote table, use a WHERE clause to select only the last N days and a COUNT() aggregate to count your votes.
If you do NOT want to keep every vote, you should have an accumulation table with person, day and votecount - an analogous query with SUM() instead of COUNT() will do what you want.
I have a query which i have been working on for hours and can't seem to get it working.
the query is to Generate list of customers that have spent > €100 in the last 365 days. I am creating a video rental database..
This is how far i have gotten but cant seem to link the data together with date_rented data table.
SELECT CUST_ID, CUSTOMER_SPEND
FROM ACCOUNT_TEST
WHERE CUSTOMER_SPEND > 100;
the tables I am working with are cust_id, customer_spend, date_rented and account_test
Instead of trying to answer the question that was (sort of) asked, maybe it makes sense to step back, look at the (apparently) desired result, and show how that could be achieved. For the moment, I'm going to only look at one table out of what should be a number. This table will hold the details of an individual customer rental:
customer_id
date_rented
cost
More fields are certainly needed, but those seem to cover what we care about for this query. From this, we want a list of customers who've spent at least 100 (of whatever unit cost is in), along with the amount spent by each. The only slightly tricky part is that we can't use an aggregate like sum(customer_paid) in a where clause, so we put that in a having clause instead.
select customer_id, sum(cost) as customer_paid
from rental_details
where to_days(now()) - to_days(date_rented) <= 365
group by customer_id
having customer_paid > 100
As a quick warning, that might need minor tweaking to work with MySQL -- most of what I've written recently has been for SQL Server.
I have a project, where I have posts for example.
The task is next: I must show to user his last posts visit.
This is my solution: every time user visits new (for him) topic, I create a new record in table visits.
Table visits has next structure: id, user_id, post_id, last_visit.
Now my tables visits has ~14,000,000 records and its still growing every day..
May be my solution isnt optimal and exists another way how to store users visits?
Its important to save every visit as standalone record, because I also have feature to select and use users visits. And I cant purge this table, because data could be needed later month, year. How I could optimize this situation?
Nope, you don't really have much choice other than to store your visit data in a table with columns for (at a bare minimum) user id, post id, and timestamp if you need to track the last time that each user visited each post.
I question whether you need an id field in that table, rather than using a composite key on (user_id, post_id), but I'd expect that to have a minor effect, provided that you already have a unique index on (user_id, post_id). (If you don't have an index on that pair of fields, adding one should improve query performance considerably and making it a unique index or composite key will protect against accidentally inserting duplicate records.)
If performance is still an issue despite proper indexing, you should be able to improve it a bit by segmenting the table into a collection of smaller tables, but segment it by user_id or post_id (rather than by date as previous answers have suggested). If you break it up by user or post id, then you will still be able to determine whether a given user has previously viewed a given post and, if so, on what date with only a single query. If you segment it by date, then that information will be spread across all tables and, in the worst-case scenario of a user who has never previously viewed a post (which I expect to be fairly common), you'll need to separately query each and every table before having a definitive answer.
As for whether to segment it by user id or by post id, that depends on whether you will more often be looking for all posts viewed by a user (segment by user_id to get them all in one query) or all users who have viewed a post (segment by post_id).
If it doesn't need to be long lasting, you could store it in session instead. If it does, you could either break the records apart by table, like say 1 per month, or you could only store the last 5-10 pages visited, and delete old ones as new ones come in. You could also change it to pages visited today, this week, etc.
If you do need all 14 million records, I would create another historical table to archive the visits that are not the most relevant for the day-to-day site operation.
At the end of the month (or week, or quarter, etc...) have some scheduled logic to archive records beyond a certain cutoff point to the historical table and reduce the number of records in the "live" table. This should help increase the query speed on the "live" table since you would have less records in it.
If you do need to query all of the data, you can use both tables and have all of the data available to you.
you could delete the ones you don't need - if you only want to show the last 10 visited posts then
DELETE FROM visits WHERE user_id = ? AND id NOT IN (SELECT id from visits where user_id = ? ORDER BY last_visit DESC LIMIT 0, 10);
(i think that's the best way to do that query, any mysql guru can tell me otherwise? you can ORDER BY in DELETE but the LIMIT only takes 1 parameter, so you can't do LIMIT 10, 100 there)
after inserting/updating each new row, or every few days if you like
Having a structure like (id, user_id, post_id, last_visit) for your vists table, makes it appear as though you are saving all posts, not just last post per Topic. Don't you need a topic ID in there somewhere so that you can determine what there last post PER TOPIC was, and so you know which row to replace when they post in the same topic more than once?
Store post_ids to $_SESSION and then using MYSQL IN with one SELECT query you will be able to show his visited posts. But all those ids will be destroyed after member close his browser, but anyways, this is much more faster and optimal than using database.
edit: sorry, I didn't notice you that you must store that records in database and use it after months. Then I have no idea how to optimize it, but with 14 mln. records you should definitely use indexes.
So I'm having serious speed problems using a left join to count ticket comments from another table. I've tried using a sub-select in the count field and had precisely the same performance.
With the count, the query takes about 1 second on maybe 30 tickets, and 5 seconds for 19000 tickets (I have both a production and a development server so times are skewed a tad). I'm trying to optimize this as four variations of the query need to be run each time per page refresh.
Without the count, I see execution time fall from 1 second to 0.03 seconds, so certainly this count is being run across all tickets and not just the ones which are selected.
Here's a trimmed down version of the query in question:
SELECT tickets.ticket_id,
ticket_severity,
ticket_short_description,
ticket_full_description,
count(*) as commentCount,
FROM tickets (LEFT JOIN tickets_comment on ticket_id = tickets_comment.ticket_id)
WHERE ticket_status='Open'
and ticket_owner_id='133783475'
GROUP BY
everything,
under,
the,
sun
Now, not all tickets have comments, so I can't just do a right or standard join. When doing that the speed is fairly good (1/10th the current), but any tickets without comments aren't included.
I see three fixes for this, and would like any and all advice you have.
Create a new column comment_count and use a trigger/update query on new comment
Work with the UI and grab comments on the fly (not really wanted)
Hope stackoverflow folks have a more elegant solution :þ
Ideas?
A co-worker has come to the rescue. The query was just using join improperly.
What must be done here is create a second table with a query like:
select count(*) from tickets_comment group by ticket_id where (clause matches other)
which will create a table with counts for each ticket id. Then join that table with the ticket table where the ticket ids match. It's not as wicked fast as creating a new column, but it's at least 1/10th the speed it was, so I'm pleased as punch.
Last step is converting nulls (on tickets where there were no comments) into zeros
Is by far the fastest solution and you'll see it done in Rails all the time because it really is that fast.
count(*) is really only used when you aren't selecting any other attributes. Try count(ticket_id) and see if that helps. I can't run explain so I can't test it myself but if your analysis is correct it should help.
Try running explain on the query to make sure the correct indexes are being used. If there are no indexes being used, create another one