Hello Stackoverflow community,
This is my first post here, so I apologize for my bad English! :)
If you don't want to read everything, the Questions are marked like this.
The title is a little misleading, but I didn't know how to explain it in a better way, but the detailed explanation should let you understand:
I have a big log table (about 500000 rows at this time), where a game server logs many actions that occur in the game. I want to extract some specific log rows the most efficient way.
I can't change the logging system of the game server, if I could I would change it to many more log tables, to create more compact logs. (because executing queries on that table takes it's time..)
Now my problem is, that I want to get the last log row of a specific type and from a specific player id to get the players last action, and I don't know how to do that in an efficient way.
Example:
SELECT * FROM log WHERE logType = "PLAYER" AND playerID = [playerID] ORDER BY time DESC LIMIT 1
Then the output on the website would be:
You last action was [the human readable action and additional information].
MySQL profiling now tells me that sorting of the results takes the most amount of time.
Now my question is: Is it possible to get only the last row of a specific player id and type?
I guess it could be done with the ID, cause it has auto_increment. So is it possible to get the row with the highest ID, a specific type and a specific player id?
The table structure:
ID(int) | logType(varchar) | time(datetime) | playerID(int) | positionX(int) | positionY(int) | actionID(int) | action(varchar) | hints(varchar) | ip(varchar) | itemNumber(int)
Explaination:
ID: the unique id of the logged action
logType: the type of the logged action (Example: "PLAYER" or "ITEM")
time: the time the action occured
playerID: the id of the player (or other id's related to that type)
positionX: X position in the game
positionY: Y position in the game
actionID: an id in relation to the action (Example: If the log action is "KILLED_BY_PLAYER", then the player id of the other player who killed the player)
action: the action that is logged (Example: KILLED_BY_PLAYER)
hints: Some useful hints like the name of the player
ip: The IP of the player
itemNumber: The number of the Item, if an Item is involved, else NULL
Thanks for your help. :)
to get the highest id:
select max(id) from table;
the others you could put a where clause in the select statement.
You should be able to use your query where you sort on time with LIMIT 1. If your sort on the column time is slow, you must make sure the indexes are optimized (run explain on your query). You will need indexes on both logtype, player_id and time. Even with 500 000 rows this shouldn't be a problem...
Related
I have a table in Access that I use as a progress tracker/to do list, with one field containing the date (short text) and what I did that day (long text). An example would be like this
date | progress
----------------
6/20 | did item1
| tomorrow do item2 and item3
6/21 | long text I continue in the next line for visibilty
| continued
| to do tomorrow
6/22 | item6 completed
etc. I enter these things manually. the past 3 weeks or so that I have been updating this table, it opens in the same order every time -- the order in which I created the records. Recently, the table opened in a completely random order, and continues to open in that new order.
I know now that it would have been good to create an autonumber field and order it by that, or have the date field's default value =Now(). I ahve far to many fields to make a new ID field and manually number each record in the order I created them.
Is it at all possible to force the table to order the records in either the order I created the records, or at least the previous configuration (which was just ordered by the time created)?
Also, is there a better way to be doing this...? I want to just have a record for other people who will work with this in the future. In addition, I am new to SQL, and the Access SQl has weird/unique ways of doing things, so for some queries I know I may need in the future I keep a table with the query name and some documentation for what exactly it does and some notes about the syntax (the SQL editor does not allow for -- comments). Is there a better way to do this, too?
Thanks for any help!
one field containing the date (short text)
Dates should never be stored as anything else than date values. No exceptions.
So change the data type of the field to Date, sort on this, and your troubles are gone.
To order Null dates last:
Order By Abs([DateField] Is Null), [DateField]
Could someone please help me on Rails query regarding this
I have mysql table like this. There is a recreation club. ticket_id is unique. There is entry and exit for each ticket (member).
I want to identify on particular time, how many members are inside. suppose if i want to know the inside members by the time '2015-12-17 16:48:00'.
id ticket_id event_type timestamp
1739 869 entry 2015-12-17 15:09:51
1740 869 entry 2015-12-17 17:46:55
1730 864 exit 2015-12-17 16:48:27
Assumption 1 : Every ticket_id can have multiple entries and exits. The data you have provide has 2 entries for ticket_id 869. Hence, the ActiveQuery results are subject to uniq. Even when a person has multiple entries (or exits) before required time, they are counted only once.
Assumption 2 : In some cases, between 2 entries, an exit may not be captured, as demonstrated by the sample data. Some discrepancy in the result might creep in because of this.
You can use the following query to obtain all people who are inside at one particular point of time, say reqd_timestamp :
#entered = Member.where("timestamp <= ? AND event_type = ?",
reqd_timestamp, 'entry')
.pluck(:ticket_id).uniq
#exited = Member.where("timestamp <= ? AND event_type = ?",
reqd_timestamp, 'exit')
.pluck(:ticket_id).uniq
#inside = (#entered - #exit).count
Please note, ticket_id 869 will be counted as being inside as of 2015-12-17 16:48:00 because he has an entry with id 1739 prior to the timestamp. He actually may have exited before stipulated time and subsequently making another entry as id 1740. But no data being available, that particular information cannot be captured.
Try this
#members = Member.where("date(timestamp) =? AND event_type=?", '2015-12-17 16:48:00', 'entry').count
I have 2 tables posts<id, user_id, text, votes_counter, created> and votes<id, post_id, user_id, vote>. Here the table vote can be either 1 (upvote) or -1(downvote). Now if I need to fetch the total votes(upvotes - downvotes) on a post, I can do it in 2 ways.
Use count(*) to count the number of upvotes and downvotes on that post from votes table and then do the maths.
Set up a counter column votes_counter and increment or decrement it everytime a user upvotes or downvotes. Then simply extract that votes_counter.
My question is which one is better and under what condition. By saying condition, I mean factors like scalability, peaktime et cetera.
To what I know, if I use method 1, for a table with millions of rows, count(*) could be a heavy operation. To avoid that situation, if I use a counter then during peak time, the votes_counter column might get deadlocked, too many users trying to update the counter!
Is there a third way better than both and as simple to implement?
The two approaches represent a common tradeoff between complexity of implementation and speed.
The first approach is very simple to implement, because it does not require you to do any additional coding.
The second approach is potentially a lot faster, especially when you need to count a small percentage of items in a large table
The first approach can be sped up by well designed indexes. Rather than searching through the whole table, your RDBMS could retrieve a few records from the index, and do the counts using them
The second approach can become very complex very quickly:
You need to consider what happens to the counts when a user gets deleted
You should consider what happens when the table of votes is manipulated by tools outside your program. For example, merging records from two databases may prove a lot more complex when the current counts are stored along with the individual ones.
I would start with the first approach, and see how it performs. Then I would try optimizing it with indexing. Finally, I would consider going with the second approach, possibly writing triggers to update counts automatically.
As this sounds a lot like StackExchange, I'll refer you to this answer on the meta about the database schema used on the site. The votes table looks like this:
Votes table:
Id
PostId
VoteTypeId, one of the following values:
1 - AcceptedByOriginator
2 - UpMod
3 - DownMod
4 - Offensive
5 - Favorite (if VoteTypeId = 5, UserId will be populated)
6 - Close
7 - Reopen
8 - BountyStart (if VoteTypeId = 8, UserId will be populated)
9 - BountyClose
10 - Deletion
11 - Undeletion
12 - Spam
15 - ModeratorReview
16 - ApproveEditSuggestion
UserId (only present if VoteTypeId is 5 or 8)
CreationDate
BountyAmount (only present if VoteTypeId is 8 or 9)
And so based on that it sounds like the way it would be run is:
SELECT VoteTypeId FROM Votes WHERE VoteTypeId = 2 OR VoteTypeId = 3
And then based on the value, do the maths:
int score = 0;
for each vote in voteQueryResults
if(vote == 2) score++;
if(vote == 3) score--;
Even with millions of results, this is probably going to be a very fast operation as it's so simple.
I actually have a table with 30 columns. In one day this table can get around 3000 new records!
The columns datas look like :
IMG Name Phone etc..
http://www.site.com/images/image.jpg John Smith 123456789 etc..
http://www.site.com/images/image.jpg Smith John 987654321 etc..
I'm looking a way to optimize the size of the table but also the response time of the sql queries. I was thinking of doing something like :
Column1
http://www.site.com/images/image.jpg|John Smith|123456789|etc..
And then via php i would store each value into an array..
Would it be faster ?
Edit
So to take an example of the structure, let's say i have two tables :
package
package_content
Here is the structure of the table package :
id | user_id | package_name | date
Here is the structure of the table package_content :
id | package_id | content_name | content_description | content_price | content_color | etc.. > 30columns
The thing is for each package i can get up to 16rows of content. For example :
id | user_id | package_name | date
260 11 Package 260 2013-7-30 10:05:00
id | package_id | content_name | content_description | content_price | content_color | etc.. > 30columns
1 260 Content 1 Content 1 desc 58 white etc..
2 260 Content 2 Content 2 desc 75 black etc..
3 260 Content 3 Content 3 desc 32 blue etc..
etc...
Then with php i make like that
select * from package
while not EOF {
show package name, date etc..
select * from package_content where package_content.package_id = package.id and package.id = package_id
while not EOF{
show package_content name, desc, price, color etc...
}
}
Would it be faster? Definitely not. If you needed to search by Name or Phone or etc... you'd have to pull those values out of Column1 every time. You'd never be able to optimize those queries, ever.
If you want to make the table smaller it's best to look at splitting some columns off into another table. If you'd like to pursue that option, post the entire structure. But note that the number of columns doesn't affect speed that much. I mean it can, but it's way down on the list of things that will slow you down.
Finally, 3,000 rows per day is about 1 million rows per year. If the database is tolerably well designed, MySQL can handle this easily.
Addendum: partial table structures plus sample query and pseudocode added to question.
The pseudocode shows the package table being queried all at once, then matching package_content rows being queried one at a time. This is a very slow way to go about things; better to use a JOIN:
SELECT
package.id,
user_id,
package_name,
date,
package_content.*
FROM package
INNER JOIN package_content on package.id = package_content.id
WHERE whatever
ORDER BY whatever
That will speed things up right away.
If you're displaying on a web page, be sure to limit results with a WHERE clause - nobody will want to see 1,000 or 3,000 or 1,000,000 packages on a single web page :)
Finally, as I mentioned before, the number of columns isn't a huge worry for query optimization, but...
Having a really wide result row means more data has to go across the wire from MySQL to PHP, and
It isn't likely you'll be able to display 30+ columns of information on a web page without it looking terrible, especially if you're reading lots of rows.
With that in mind, you'll be better of picking specific package_content columns in your query instead of picking them all with a SELECT *.
Don't combine any columns, this is no use and might even be slower in the end.
You should use indexes on a column where you query at. I do have a website with about 30 columns where atm are around 600.000 results. If you use EXPLAIN before a query, you should see if it uses any indexes. If you got a JOIN with 2 values and a WHERE at the same table. You should make a combined index with the 3 columns, in order from JOIN -> WHERE. If you join on the same table, you should see this as a seperate index.
For example:
SELECT p.name, p.id, c.name, c2.name
FROM product p
JOIN category c ON p.cat_id=c.id
JOIN category c2 ON c.parent_id=c2.id AND name='Niels'
WHERE p.filterX='blaat'
You should have an combined index at category
parent_id,name
AND
id (probably the AI)
A index on product
cat_id
filterX
With this easy solution you can optimize queries from NOT DOABLE to 0.10 seconds, or even faster.
If you use MySQL 5.6 you should step over to INNODB because MySQL is better with optimizing JOINS and sub queries. Also MySQL will try to run them into MEMORY which will make it a lot faster aswel. Please keep in mind that backupping INNODB tables might need some extra attention.
You might also think about making MEMORY tables for super fast querieing (you do still need indexes).
You can also optimize by making integers size 4 (4 bytes, not 11 characters). And not always using VARCHAR 255.
It is possible to schedule a snapshot of a MySQL table?
The situation:
I have a table that collects votes (up and down) from a website. It registers the number of votes (that can be either a vote up, “+1”, or a vote down, “-1”), and records the score (for example, if one ID received 5 votes, 3 voting "up" (+1) and 2 voting "down" (-1), then the table would record that number of votes was "5", and that the vote score was "2". It is recorded in a table that has the following column headers/fields:
ID | Score | nvotes
…where ID is the reference of the item being voted on; 'score' is the actual score that is calculated (from the number of “+1” and “-1” votes), and 'nvotes' is the number of votes that have been received
This is great for looking at the website at any point in time and seeing what the score is, and how many people have voted.
However, I now want to be able to chart the trend for the ID – to look back over time and see how the score has gone up and down over time.
Is there a facility in MySQL to be able to take a snapshot at the end of each day, recording where that particular ID was at the end of that day in terms of their score and the number of votes received, and store this in another table so that I can create charts and analysis over time?
Or, failing that, can anyone think of a better, more intelligent way of trying to acheive what I need?