I apologize in advance that I don't know the terminology of the tools I'm trying to use.
I have a table of events with a startdate field (among others) and a related repeats table with a reference to the event id. The repeats table stores the days of the week on which the event repeats and whether it's monthly, weekly, etc. What I'm hoping to do is duplicate the repeating events within the SQL query so my final result will have the the same event in different places when ordered on start date, so I can limit the results for proper pagination.
I'm looking at creating virtual tables and cloning tables documentation, but I'm having trouble applying the examples to my situation.
Update:
Hopefully I can elaborate on this.
The basics of what I have now is SELECT * FROM 'events' WHERE 'start_date' >= TODAY() ORDER BY 'start_date LIMIT 20 which gets me every event from today on, but I'm paginating the results so only 20 are displayed at a time.
What I would like to do is create a temporary 'virtual' table with the events which have an associated repeat entry, on which I will change the start_date based on the repeat information. So if it's a weekly repeat, this second table would be filled with identical events except that each start_date would be 7 days from the last. Then I could do a join on these two tables, limit those results to the 20 pagination limit I want, and have a query result with the events in the correct place and easy to perform pagination on.
I understand that creating a function in mySQL might be on the right track as I imagine I would have to loop through some information for adding to dates. I only know the level of SQL one picks up by writing in PHP, so functions are a bit out of my scope, though it doesn't seem that it'll be too hard to pick up with a little reading. I'm more confused about how I would create a fake table, add entries to it in a loop and then use a join on it to merge it with the first query.
I'm also beginning to wonder about the overhead for doing this in mySQL and, should I be successful in getting this to work, how I might cache these results, though it's only an afterthought right now.
Thanks to those who are trying to help me, I'm having trouble getting this question into words for some reason.
Related
I have a table that I use for statistical purposes.
Its columns are id and 1,2,3,..,31 and pivot.
This table gives the number of views on each day for the last 31 days.
1 gives the number of views for yesterday.
14 gives the number of views for 14 days ago.
etc ...
(pivot is just used to calculate the number of views)
I would use a cron job every day to update this table, but how would I go about "shifting" all the values to the side ( value column 15 would become value column 16; new value for column 1; delete value for column 31)
Define a table with only two columns — "date" and "views"
INSERT a new row in the table with the view count for that day when the CRON job runs
Modify your application query to read through this new table over a custom date range, which could be 31 days or anything else either — please have a look at this link to get an idea:
MySQL Query - Records between Today and Last 30 Days
Not really sure how pivot is being used here. However, I'm almost certain that if you're using it to store the sum of the views, it could as well be computed by using SUM() or GROUP BY without having to need a separate column in the table
As far as data archival / removal is concerned, your daily CRON job could be modified to include a DELETE query (as the last step) which cleans up records older than a certain date. Again, you could use the link above to get your "target" date
.
I apologise that this might sound like a little too long a solution to what you've asked for. However, I feel, this approach should help you organise and maintain the table in question in a better way.
Ok, so what is the best practice when it comes down to paginating in mysql. Let me make it more clear, let's say that a given time I have 2000 records and there are more being inserted. And I am displaying 25 at a time, I know I have to use limit to paginate through the records. But what am I supposed to do for the total count of my records? Do I count the records every time users click to request the next 25 records. Please, don't tell me the answer straight up but rather point me in the right direction. Thanks!
The simplest solution would be to just continue working with the result set normally as new records are inserted. Presumably, each page you display will use a query looking something like the following:
SELECT *
FROM yourTable
ORDER BY someCol
LIMIT 25
OFFSET 100
As the user pages back and forth, if new data were to come in it is possible that a page could change from what it was previously. From a logical point of view, this isn't so bad. For example, if you had an alphabetical list of products and a new product appeared, then the user would receive this information in a fairly nice way.
As for counting, your code can allow moving to the next page so long as data is there to support a new page being added. Having new records added might mean more pages required to cover the entire table, but it should not affect your logic used to determine when to stop allowing pages.
If your table has a date or timestamp column representing when a record was added, then you might actually be able to restrict the entire result set to a snapshot in time. In this case, you could prevent new data from entering over a given session.
3 sugggestions
1. Only refreshing the data grid, while clicking the next button via ajax (or) storing the count in session for the search parameters opted .
2. Using memcache which is advanced, can be shared across all the users. Generate a unique key based on the filter parameters and keep the count. So you won't hit the data base. When a new record, gets added then you need to clear the existing memcache key. This requires a memache to be running.
3. Create a indexing and if you hit the db for getting the count alone. There won't be much any impact on performance.
I have an AngularJS front-end app that sends requests to a Rails API back-end. When a user search for items, so far the query is limited to 20 elements and is always ordered by popularity (a field stored in database). After the results are retrieved from DB, there is a complex process that calculates the item prices iterating one by one (remember only 20 elements). After that, the results are served to the user. Note: As told, item prices cannot be calculated directly in the query because an item can have different prices according to dates and also discounts can be applied.
This is how it´s working so far.
Now, I would like to introduce in the search results page an innocent order by: Price functionality. So, the array of items should come ordered by price.
As, I can´t get the prices directly with just one query, I see two choices:
To keep it as it is right now, I mean, making the query ordered by popularity and order the results after the prices are calculated. But I see a problem, If I get 20 elements each time ordered by popularity, then I can calculate prices and order by price these 20 elements, so I assume I´m not ordering correctly by price. This case, I would need to query without limit, to get all items, calculate prices, order them by price and return to user. I think I would also to develop a home-made pagination functionality.
Develop some kind of stored procedure in the database to provide the results with the complex prices calculations. I don´t know if I can order them easily. I´m worried because I don´t know stored procedures in MySQL and not sure if it´s possible to do what I need.
But, from the performance point of view, I guess the second choice should be better, right? I´d appreciate comments or any other options?
UPDATE:
According to comments, I detail how to calculate prices functionality.
A user can rent an item for many days (a week i.e.). So, there is a check-in and check-out dates.
Also, prices changes according to seasons. This means days can have different prices in a selected week. So, in order to calculate the total price, you have to get the daily price matching each selected day in the week and add it to the total price.
Once, the base price is calculated, there can be discounts. Same as prices, discounts can be applied only for some days, so first, it must be checked if there is any discount for the selected week. If so, the discount is applied to base price to get the total final price.
Please, let me know if you need the code.
This should be done on the backend (MySQL).
Like you mentioned, if you want to sort by price on the front-end, you'll have to replicate the entire database into Angular. Which is probably a bad idea.
This is the approach I would take in MySQL:
Set up a table with item_id, date, and price (or perform the joins necessary to get this table).
Apply discounts for each date. This step yields an interim table with updated prices.
Build your final query. Your SELECT clause should SUM(price). You should GROUP BY item_id. And you should ORDER BY SUM(price) DESC.
Depending on the size of your database, this query may require a lot of fine-tuning in order to return results quickly. But it definitely can (and should) be done in the backend.
Good luck!
EDIT: With a really big set of items, running this query through MySQL may become too slow, regardless of how much time you spend tuning performance. If MySQL doesn't cut it, you may need to rely on an auxiliary database like Elasticsearch.
But before turning to Elasticsearch/Hadoop/etc, you should think carefully about how "complex" your pricing algorithm really is. In all likelihood you can optimize the MySQL query to the point where it performs just fine.
This is a classical search listing problem. If you want to perform sort based on price, obviously you need the required inputs. it is worth mentioning that stored procedure won't magically improve the performance. The best what you can do is run recurring CRON job every hour/day ( bonus tip : use upset and perform mass operation) which updates the price. Now perform join and do the query and order it with "price". Also make sure to put indexes on required fields. After grabbing the result apply discount and show the result. If query is complex the pagination won't work. You have to do all sorts of operation manually. If you are using will_paginate , it has some function which also support Array instead of active record. Hope this helps.
So you have the following factors then:
User Inputs Beginning Date, End Date and sees list of items.
The Beginning - End Date period (selected period) is consistent for all items in the query.
To sort by price you must, for each item in the table (not just those retrieved)
Get Item Price for each day of the selected period.
Get available discounts for each day of the selected period.
Calculate Total Price for selected period.
Because user input is involved you will not be able to definitively calculate this and store the final product. The best you could do is calculate the gross and net prices per day and cache that, then performing the final calculation based on user input (or sending that data to browser for calculation there)
You could also use a first order approximation and just go with the current gross value, and then tweak from there. That assumes that the relationship between item prices remains more or less consistent.
I'm trying to help a taxi company. The problem is that they have a credit card machine that takes payments, and has its own database entries, and there is a completely separate database that has a list of entries such as pick up time, and drop off time.
I need to match the database of trip entries to the credit card purchases, and the only way to do this is by matching which vehicle is running the transaction, and looking for a time CLOSE TO the DROP OFF time and see if it's a match. It's not ideal.
So, I am trying to compare two times in yy-mm-dd hh:mm:ss format. They need to be plus or minus 5 minutes of each other. How do I do this in MYSQL?
I thought SubTime and Addtime would work, and it seemed to, but then I got wierd results.
SELECT * FROM completedtrans WHERE DrivID = 128 AND TransTime BETWEEN SUBTIME('2013-06-20 16:53:06', '0 00:05:00') AND ADDTIME('2013-06-20 16:53:06', '0 00:05:00')
Here's an example of one of my searches. Can anyone tell me what's wrong with it? It's supposed to search 5 minutes before and after that particular given time. I can't simply write the time, because the query is automatically generated through php code.
I'm working on an app that is partly an employee time clock. It's not too complex but I want to make sure I head in the right direction the first time. I currently have this table structure:
id - int
employee_id - int (fk)
timestamp - mysql timestamp
event_code - int (1 for clock in, 0 for clock out)
I've got everything working where if their last event was a "clock in" they only see the "clock out" button and visa-versa.
My problem is that we will need to run a report that shows how many hours an employee has worked in a month and also total hours during the current fiscal year (Since June 1 of the current year).
Seems like I could store clock in and outs in the same record and maybe even calculate minutes worked between the two events and store that in a column called "worked". Then I would just need to get the sum of all that column for that employee to know how much time total.
Should I keep the structure I have, move to all on one row per pair of clock in and out events, or is there a better way that I'm totally missing?
I know human error is also a big issue for time clocks since people often forget to clock in or out and I'm not sure which structure can handle that easier.
Is MySQL Timestamp a good option or should I use UNIX Timestamp?
Thanks for any advise/direction.
Rich
I would go with two tables:
One table should be simple log of what events occurred, like your existing design.
The second table contains the calculated working hours. There are columns for the logged in and logged out times and perhaps also a third column with the time difference between them precalculated.
The point is that the calculation of how many hours an employee has worked is complicated, as you mention. Employees may complain that they worked longer hours than your program reports. In this case you want to have access to the original log of all events with no information loss so that you can see and debug exactly what happened. But this raw format is slow and difficult to work with in SQL so for reporting purposes you also want the second table so that you can quickly generate reports with weekly, monthly or yearly sums.
Is MySQL Timestamp a good option or should I use UNIX Timestamp?
Timestamp is good because there are lots of MySQL functions that work well with timestamp. You might also want to consider using datetime which is very similar to timestamp.
Related
Should I use field 'datetime' or 'timestamp'?