I started building a search engine monitor. I'm pulling data from the google rest api into a mysql database with the following fields: date, search-keyword, domain, url, position.
Now I got into trouble querying and outputting the data for charting. The results go up and down, new results from google come into the list which haven't been there on the first day. However for charting I have to assign the first days at least blank values to output a chart.
What I do right now: First I select every domain showing up in the period. Lets say the for the keyword searchengine I get the domains wikipedia.org, ixquick.com, yahoo.com, searchenginewatch.com When I make another request for ever domain to query an array of rankings grouped by day. leading to the ...
Problem: Is where any query (mysql/nosql) which returns for each day an average and if where is no row a default value e.g. blank?
Result should look like:
dates={01/01/2014,02,03,04,05,06,07,08,...,31}
wikipedie={1,1,1,1,1,1,1,1,...,1}
yahoo = {"","",7,5,3,3,3,...,3}
You can create a date table, select the date range you'd like, and outer join your data to it, filling in 0s for values that do not exist for a given term/date.
Edit:
Some more details.
1) Create a table that has a row for every date +- 10 years (or whatever is appropriate). You can make this one column if you'd like, or many columns (date, month, year, etc.). The second approach makes this extensible if you want to summarize by various rollups in the future.
2) Outer join your table to the date table and use a NVL statement to coerce any null averages to 0.
3) Profit!
If your results are grouped by date, how can MySQL know there's (for example) 31 days in that month?
On the other hand, you can somehow fill the holes in PHP by loop through the array and fill a zero if the value does not exist.
Related
I have a table that I use for statistical purposes.
Its columns are id and 1,2,3,..,31 and pivot.
This table gives the number of views on each day for the last 31 days.
1 gives the number of views for yesterday.
14 gives the number of views for 14 days ago.
etc ...
(pivot is just used to calculate the number of views)
I would use a cron job every day to update this table, but how would I go about "shifting" all the values to the side ( value column 15 would become value column 16; new value for column 1; delete value for column 31)
Define a table with only two columns — "date" and "views"
INSERT a new row in the table with the view count for that day when the CRON job runs
Modify your application query to read through this new table over a custom date range, which could be 31 days or anything else either — please have a look at this link to get an idea:
MySQL Query - Records between Today and Last 30 Days
Not really sure how pivot is being used here. However, I'm almost certain that if you're using it to store the sum of the views, it could as well be computed by using SUM() or GROUP BY without having to need a separate column in the table
As far as data archival / removal is concerned, your daily CRON job could be modified to include a DELETE query (as the last step) which cleans up records older than a certain date. Again, you could use the link above to get your "target" date
.
I apologise that this might sound like a little too long a solution to what you've asked for. However, I feel, this approach should help you organise and maintain the table in question in a better way.
I am writing a script which will counts the number of ticket as per as different conditions and store the count in a summary table. I am unable to understand how to structure my table since there will be more than 1 variables.
The script will run every week and with each execution a new week will be added. Once the new month starts and scripts execute, a new month will be added and once new year starts a new year will be added. So in a nutshell I have columns which are also variables. Not sure how to handle it and structure my db.
Have columns Date Service Count.
Like so.
Date Service Count
16 May 2016 Service1 35
Then when you go to display them in the report pivot them as you want. Probably better to make date an INT and either an auto number PK or composite PK of Date and Service.
Wrong approach. Do not splay an array across columns. Instead have rows for the data. This lets you trivially "add" another month.
For displaying, well that is an application problem, no a MySQL problem. Sure, you can write specialized queries to "pivot" the data from rows to columns, but it is messy.
I have a database for our local real estate listings, there are no dates or timestamp columns.
I would like to be able to get out just the rows that were added in the past day or two.
Can anyone point me in the right direction to get this data out?
EDIT:
Each new row does get a new id number which is incrementally higher, so I can ORDER the results by newest.
Would it be possible to save my query count in a file, or in another database, each day, then calculate the difference and use that as my number of new listings?
"Would it be possible to save my query count in a file, or in another database, each day, then calculate the difference and use that as my number of new listings?"
I understand that you can't change the table structure to add a date...so instead, I suggest to have a cron job at midnight that will create a record with a date and the higher ID at this moment. This way, you will be able to finde a range of ID for a specific date...
If you relayed on query count, you will get problem when you will start to delete some rows...
PostPosted: 09 May 2014 22:26
Post subject: Determine table based on prompt
Hello,
I have three fact tables. First table holds current data, FACT_CUSTOMER_CURRENT. Other two tables hold historical snapshots. For example, one of these table holds last 60 days' records- FACT_CUSTOMER_DAILY. The other table holds data for the last day of the months.-FACT_CUSTOMER_MONTHLY
I want to add a date prompt. If the user selects yesterday as a prompt value, report should bring value from first table which holds current data (FACT_CUSTOMER_CURRENT). If user enters 28.02.2014, the report should retrieve data from FACT_CUSTOMER_MONTHLY. I tried to use context and aggregate awareness, but I could not be successful.
Can you help me?
Kind regards
There's no direct, easy way to do what you want.
Aggregate Awareness is useful for selecting a table based on the selection of objects in a query, but it does not support dynamic selection of tables based on values in a prompt.
If yesterday's data will only exist in fact_customer_current, then you can use this method: In your report, create a UNION query. One query includes objects from fact_customer_current, and the other from fact_customer_monthly. They both have an identical prompt on the appropriate date field. When a user enters yesterday's date, the first UNION query will return data but the second one won't. Likewise for date before yesterday, the first UNION will return no data but the second one will. This solution requires that the tables are correctly indexed such that a query on a date that isn't in the table will return quickly.
Let's say I record my car's latitude & longitude every minute or so, adding a row to a table each time.
I want to have a DB grid with 4 columns
latitude
longitude
distance since last measurment
curent street address, if known
Number 4, I can try to retrieve from Google Maps and I either get a text or blank, so let's ignore that.
How do I get #3? Should I calculate it in my application or in MySql (using a stored procedure or just a complicated SELECT)?
Whichever answer, can someone provide some sample code, or a link to an example? Google is not my friend today :-(
You can do it either way:
Calculate it in your query, and display it in a calculated column (such as when showing line totals where you're displaying ITEM_PRICE, QUANTITY, ITEM_PRICE * QUANTITY AS LINE_COST.
Calculate it in your application, using a calculated field (double-click the TTable component to open the Fields Editor, add field, set the type to ftCalculated, and write code in the OnCalcEvent for that calculated field.
The first choice is usually preferable, because the DB is usually more efficient at doing those sorts of thing on sets of data than when your code does it by row instead. (Set operations are obviously more efficient than row operations.)
Ken’s answer is goes far as it goes, because the most difficult task of solving your problem is getting data from two rows at the same time in SQL. If you were using Oracle or SQL Server, you could use the analytical/windowing LAG function. With MySQL you have to do it yourself. Here are 2 articles on how to simulate LAG in MySQL
http://www.onlamp.com/pub/a/mysql/2007/03/29/emulating-analytic-aka-ranking-functions-with-mysql.html?page=1
http://explainextended.com/2009/03/12/analytic-functions-optimizing-lag-lead-first_value-last_value/
Plus there are StackOverflow questions on how to get data from the previous row.
How do I lag columns in MySQL?
Calculate delta(difference of current and previous row) in sql
I cannot offer any opinion on any of them because I seldom work with MySQL.