This is what I am doing:
Running query that Grabs all albums for an owner
Looping through the 1. query, and run another query to grab cover_pic from photo where pid= cover_pid and show it.
In the same loop through the 1. query, I am now also running another query that selects photos limit 5 and show them.
This is: 1 + 1 + 5 = 7 queries Per album, and there's 33 albums = 231 Queries, which takes TIME to load the site, and destroys the user experience!
I have after debugging found out it's not the 1. query that are taking its time to load, but the 2. and 3. (2. runs 33 queries and 3. runs 165 queries!)
How can I in any way reduce this?
I guess I need to rethink the structure, but I cant see any better solution to load the 2. and 3.
I have thought about cookies, sessions to save so next time it is faster, but I think that the first load will the user leave because of the loading time, and therefore next time doesnt matter..
Please use the Graph API Field Expansion tool to achieve this.
eg - graph.facebook.com/[USER_ID]?fields=name,albums.fields(cover_photo,photos.limit(5))
You can visually construct the query using the Graph Explorer
Related
Person -> Item - > Work -> Component.
This are the main tables in the database.
I have to search for Item by a criteria. It will give a list. I will join the Person to get his "parent". After this maybe is a record in Work table maybe not, but if is, I will join also for Work I will join the list of Components if can be found.
The original code it was with nested tables. It will crash the browser, because taking to much memory with that design and is extremely slow around 150 records.
I rewrote it with divs the nested table. The performance got a huge boost, but start to be slow again because of buttons and design. ( It wasn't able to show 200 record before even after 10 min waiting!, now display 5k rows in 23 seconds)
Some of my benchmark logs:
SQL execution time: 0.18448090553284 seconds. Found 5624 rows.
For each result processing took: 0.29220700263977 seconds.
Writing the HTML code took:0.4107129573822 seconds.
Rows in HTML: 26551 headers + data.
Total Cells in HTML: **302491** headers and data.
Time until DOMready: 23691 milliseconds (in JavaScript)
0.18 + 0.29 + 0.41 = 0.88 So is around 1 second!
But when the browser actually want to show it to you ( paint ) it will take like 20 seconds!!!
Please don't suggest the paging! - the customer (final user) want to see all data in 1 web page for whatever his reason. No comment here.
Running on an i7 processor and 8/16 GB ram as requirement is accepted.
Most of the data rows has a collapse/expand button.
Most of the data row has the CRUD buttons: Add, Edit, Delete, View in details
All 4 kind of data table has headers and they don't match the length of the other kind of table header size, neither in column numbers.
When I want to just list the data in a blank page ( without design) and use 1 single table it is like 2 seconds or 3, not 20-30.
The original nested table solution has buttons with functionality in data row.
I would like to use it, and not implement it again.
My idea is to go back to the original nested table design ( to not re implement a lot of functionality of buttons) Then display only the top level table collapsed, with expand buttons. Then call an AJAX to get the second level data, when ready call the 3rd level then the 4th level.
The user is using intranet or the same PC as the server, so this maybe can be acceptable? -and doesn't have a blocking user interface for long time.
How would you handle this case, when there is not an option to show a next page button with 20 records / page.
Ok, so what is the best practice when it comes down to paginating in mysql. Let me make it more clear, let's say that a given time I have 2000 records and there are more being inserted. And I am displaying 25 at a time, I know I have to use limit to paginate through the records. But what am I supposed to do for the total count of my records? Do I count the records every time users click to request the next 25 records. Please, don't tell me the answer straight up but rather point me in the right direction. Thanks!
The simplest solution would be to just continue working with the result set normally as new records are inserted. Presumably, each page you display will use a query looking something like the following:
SELECT *
FROM yourTable
ORDER BY someCol
LIMIT 25
OFFSET 100
As the user pages back and forth, if new data were to come in it is possible that a page could change from what it was previously. From a logical point of view, this isn't so bad. For example, if you had an alphabetical list of products and a new product appeared, then the user would receive this information in a fairly nice way.
As for counting, your code can allow moving to the next page so long as data is there to support a new page being added. Having new records added might mean more pages required to cover the entire table, but it should not affect your logic used to determine when to stop allowing pages.
If your table has a date or timestamp column representing when a record was added, then you might actually be able to restrict the entire result set to a snapshot in time. In this case, you could prevent new data from entering over a given session.
3 sugggestions
1. Only refreshing the data grid, while clicking the next button via ajax (or) storing the count in session for the search parameters opted .
2. Using memcache which is advanced, can be shared across all the users. Generate a unique key based on the filter parameters and keep the count. So you won't hit the data base. When a new record, gets added then you need to clear the existing memcache key. This requires a memache to be running.
3. Create a indexing and if you hit the db for getting the count alone. There won't be much any impact on performance.
I wrote a VBA script that runs in an Access database. The script looks up values on various tables and assigns an attribute to a main table based on the combination of values.
The script works as intended, however, I am working with millions of records so it takes an unacceptably long time.
I would like to break the process up into smaller parts and run the script concurrently on separate threads.
Before I start attempting to build a solution, I would like to know:
Based on your experience, would this increase performance? Or would the process take just as long?
I am looking at using Powershell or VBScript to accomplish this. Any obstacles to look out for?
Please Note: Due to the client this will run on, I have to use Access for the backend and if I use Powershell it will have to be version 1.0.
I know these are very vague questions but any feedback based on prior experience is appreciated. Thanks
Just wanted to post back with my final solution on this...
I tried the following ways to assign an attribute to a main table based on a combination of values from other tables for a 60,000 record sample size:
Solution 1: Used a combination of SQL queries and FSO Dictionary objects to assign attribute
Result: 60+ minutes to update 60,000 records
Solution 2: Ran script from Solution 1 concurrently from 3 separate instances of Excel
Result: CPU was maxed out (Instance 1 - 50% of CPU, Instances 2 and 3 - 25% each); stopped the code after an hour since it wasn't a viable solution
Solution 3: Tried using SQL UPDATE queries to update main table with the attribute
Result: This failed because apparently Access does not allow for a join on an UPDATE sub-query (or I just stink at writing SQL)
Solution 4 (Best Result): Selected all records from main table that matched the criteria for each attribute,
output the records into csv and assigned the attribute to all records in the csv file.
This created a separate file for each attribute, all in the same format. I then
imported and appended all of the records from the csv files into a new main table.
Result: 2.5 minutes to update 60,000 records
Special thanks to Pynner and Remou who suggested writing the data out to csv.
I never would have thought that this would be the quickest way to update the records with the attribute. I probably would have scrapped the project thinking it was impossible to accomplish with Access and VBA had you not made this suggestion. Thank you so much for sharing your wisdom!
When I run this line of code
Movie.increment_counter :views, #moive.id
So the column views will be incremented twice (+2, not +1). In terminal I see this ran query to database:
UPDATE `movies` SET `views` = COALESCE(`views`, 0) + 1 WHERE `movies`.`id` = 8
If I attempt to run this query direct to MySQL, co the value of views is incremented correctly once (+1).
Any tips, what am I missing or I haven't set up?
Are you tracking page views by any chance? I ran into this as well- for every page load I would see the page view counter increment by three rather than 1, but only in production.
It turned out to be Google's Adsense code loading the page remotely. I noticed that they hit the page twice for every time one of my users would hit the page, effectively resulting in 3 page views. I suspect they do this to verify that content on the page meets their guidelines and to help match ads appropriately to page content. Check your httpd logs for Mediapartners-Google. I bet that's what's going on.
General advice: use Google Analytics (or similar service) for tracking page views. In my case I still needed to track this in a DB because I implement autocomplete based on "popularity" of certain page actions, but you might not need this.
I'm trying to select the top ten most similar properties for a given property in a realty site and I wondering if you guys could help me out. The variables I'm working with would be price(int), area(int), bathrooms(int), bedrooms(int), suites(int), parking(int). At the moment I'm thinking of ordering by ABS(a-b) but wouldn't that be slow if I had to calculate that every time a property is viewed? (I'm not sure I could cache this since the database is constantly being updated) Is there another option?
Thanks for your help!
One solution could be to create a new table containing the result ready. Like this:-
property_id similar_properties_ids
--------------------------------------
1 2,5,8
2 3,10
...
...
And a cron running at regular intervals doing the calculation for all the properties and filling up the similar_properties_ids.
So, at runtime, you don't have the calculation overhead but the downside is that you get results which are a little old (updated during the last cron run).