I'm busy looking at building a matchmaking (dating-) site. The tables and queries are set up in a way i'm not happy with. It's still under construction so i'm happy to change anything in order to get it to work properly. It was very speedy until i decided to fill the database with loads of records to see how it performs.
Beware, this is a rather long post with lots (i hope the right...) information.
I have been googling' and reading in books for days now but A) the query performs very slowly (no wonder, it's no good i think) and B) i'm not getting any further with it.
So i hope somebody can help me to tell me what i'm doing wrong, what i should be doing and how to speed the query up. It's taking as long as 10-20 seconds to generate a result set and thats not good, anybody knows..
The information:
I have a table called 'profiles' with about 500.000 records in it. (at this time, could be more in the future)
Profiles table:
example contents:
Every answer to a question gets inserted to the profiles table, as a row. There are questions who are multiple choice so every answer selected by the member will be inserted as a new row in the profiles table and there are questions which can only have one answer thus one row in the profiles table.
Also, there is the table 'status' in which i keep record of members blocking or favoriting each other:
example contents:
Once the member fills in his/her criteria, php dynamically builds the query which needs to fetch records from the profiles table:
You can imagine how big this select query will get if all 90 questions are in the sql statement above..
Explain tells me this:
Basically i want to query the profiles table, fetching members matching the given criteria. The criteria are:
the criteria that the member who searches entered as his/hers will retrieve matching members as a result
the current member (the one who searches) should not be in the result(s)
members who are present in the status table with a status of 'block' should not be in the result(s)
I'm aware (now) that the 'where in' clauses are not very fast, the indexes could be wrong and the maybe my whole table lay-out needs to be different, but i hope somebody can either point me or help me in the right direction. After a day of trying and googling' i'm at my wits end.
If you need more information, just shout! Hopefully somebody can help me.
Related
I have a quite complex view with two queries (a view in a view), one select users with related data and another one select orders with related data. Both of them have some filters, but now I have an issue and I am looking for proper and just decent solution, with good performance because I have a lot of data and relationships in the queries.
Assume I have:
Query 1 - Select user data, some left joins to other tables, conditions depends on provided parameters.
Query 2 - Select orders depends on users from Query 1, many joins, conditions depends on parameters.
I display data from two queries in one view, users, their data, orders, and some orders data and now I want to implement pager, but it has to work and display proper number of users depends on filters form Query 1 and Query 2. So there is an issue that I can't really limit from any query cuz another one has filters as well so maybe those users maybe aren't really selected to display depends on other query filters.
So I guess there are two ways, one is to put those queries in loop and collect data until I get proper number of results depends on query.
Another way is to merge those two queries into one, but there an issue that I get many rows per user, so I can't set any page limit and get results only for specific number of users, like for example 30. Because results will be like user 1 => order 1, user 1 => order 2, so is there any way to get specific number of unique results depends on user id or something.
Let me know if you have any questions.
Sample data will make more sense. I am unable to understand the whole requirement here in your question. will you be able to create some sample data and share with us ? if you are dealing with a lot of data, avoid loops as that will just make performance worse.
I have learned more, and was compelled to find the solutions, mind changed.
PS dear experts, your help is appreciated in the time-saving nature of having forums and discussion in the first place, and also is mined for usefulness, not street cred. get helping or get off. The time I wasted reading posts where the answer was "I don't understand what you are trying to do..." and then questioning the OP with animus or incredulity, or suggesting some unrelated answer further confusing issues, is seemingly the problem with the world these days, if you want to help, help
Original Post:
Ok so as per the comments, thanks to ANYONE who volunteers any help with this problem.
I have a table and relationship design problem.
I have a table with a pk auto and I want to have a related table with a related column incrementally numbered that updates every time new data is entered in the form that is bound to it. It needs to update the related rows in the autonumbered table's key.
---the answer was to join the table on the autonumbered field.
The autonumber of the first table (main recordsource) is just an ID. I think I need two Primary Keys as I need to update the related record with that number in the entry form and move to NextRec but update the pk in the main table and move to NewRec , how do I join (see jpg image)?
I want enter results and update that existing record but update the adjoining pk so that a new record is created in the main table.
Tourney
more in depth:
From yours
My desired form
note that the calculation table feeds the upcoming games table, where a query shows the players recent results. I would like to split the upcoming results to show the players' recent game history, the second tab I would like to enter either one result or many results at a time. I hope I am clearer. You can see why I have had a challenge. See my form though. The recordsource is the upcominggames table and the data entry form is for input (ENTER GAME DATA)
-----the answer to this was an update query (a separate form) and then requery the statistics form to show the new matchups that were entered.
As from the comments, this is what I'm thinking of when I read your description:
Since the UpcomingGames will be entered first and exactly one GameResult can be entered per UpcomingGame, this will be a 1:1 relationship. As the name Upcoming says: The Upcoming data has to be entered before the Result can make sense. Unless an UpcomingGame can be cancelled, there will indeed be a Result for the Game, so there is no need to separate the information into 2 tables. I'd say, a user interface could look like this:
As you can see, the T_NUM column is an autovalue. Before entering any data, I initialized that column using a query like this (and deleted that record afterwards):
INSERT INTO Games ( T_NUM )
VALUES (1004);
This way, the numbering started with number 1005.
You won't be able to to avoid gaps in the numbering, as long as the users can remove existing records or cancel the insertion of a new record. If you want at least to avoid the latter, you will need some VBA code in the form.
I'm writing an application to allow users to create a Poll. They ask a question and set n number of predefined answers to the question. Other users can vote on the answers provided for that question.
Current structure
I have designed the database like this:
Storing the vote count
Current thinking is, I create a new column called vote_count on the link table and every time that answer gets voted, it updates the record.
This works. But is it right? I'm new to database systems, so I can't imagine I'm doing much right. What are some more efficient ways to achieve this?
As far as it goes yes that's OK. However these tables will be incomplete. When your second quiz is created, you'll have to extend the QUESTIONS table. If this second quiz's Q1 also has a yes/no answer, you're going to have to extend the LINK/VOTES table.
You also have to think about how it's going to be queried and design indexes to support those queries.
Cheers -
This may be a little difficult to answer given that I'm still learning to write queries and I'm not able to view the database at the moment, but I'll give it a shot.
The database I'm trying to acquire information from contains a large table (TransactionLineItems) that essentially functions as a store transaction log. This table currently contains about 5 million rows and several columns describing products which are included in each transaction (TLI_ReceiptAlias, TLI_ScanCode, TLI_Quantity and TLI_UnitPrice). This table has a foreign key which is paired with a primary key in another table (Transactions), and this table contains transaction numbers (TRN_ReceiptNumber). When I join these two tables, the query returns one row for every item we've ever sold, and each row has a receipt number. 16 rows might have the same receipt number, meaning that all of these items were sold in a single transaction. Below that might be 12 more rows, each sharing another receipt number. All transactions are broken down into multiple rows like this.
I'm attempting to build a query which returns all rows sharing a single receipt number where at least one row with that receipt number meets certain criteria in another column. For example, three separate types of gift cards all have values in the TLI_ScanCode column that begin with "740000." I want the query to return rows with values beginning with these six digits in the TLI_ScanCode column, but I would also like to return all rows which share a receipt number with any of the rows which meet the given scan code criteria. Essentially, I need the query to return all rows for every receipt number which is also paired in at least one row with a gift card-related scan code.
I attempted to use a subquery to return a column of all receipt numbers paired with gift card scan codes, using "WHERE A.TRN_ReceiptAlias IN (subquery..." to return only those rows with a receipt number which matched one of the receipt numbers returned by the subquery. This appeared to run without issue for five minutes before the server ground to a halt for another twenty while it processed the query. The query appeared to complete successfully, but given that I was working with IT to restore normal store operations during this time I failed to obtain the results of the query (apart from the associated shame and embarrassment).
I'd like to know if there is a way to write a query to obtain this information without causing the server to hang. I'm assuming that either: a) it wasn't very smart to use a subquery in this manner on such a large table, or b) I don't know enough about SQL to obtain the information I need. I'm assuming the answer is both A and B, but I'd very much like to learn how to do this the right way. Any help would be greatly appreciated. Thanks!
SELECT *
FROM a as a1
JOIN b
ON b.id = a.id
JOIN a as a2
ON a2.id = b.id
WHERE b.some_criteria = 'something';
Include an index on (b.id,b.some_criteria)
You aren't the first person, nor will you be the last to bring down your system with an inefficient query.
The most important lesson is that "Decision Support" and "Analytics" really don't co-exist with a transaction system. You really want to pull the data into a datamart or datawarehouse or some other database that isn't your transaction database, so that you don't take the business offline.
In terms of understanding why your initial query was so inefficient, you want to familiarize yourself with the EXPLAIN EXTENDED syntax that returns you plan information that should help you debug your query and work on making it perform acceptably. If you update your question with the actual explain plan output for it, that would be helpful in determining what the issue is.
Just from the outline you provided, it does sound like a self join would make sense rather than the subquery.
Apologies if this is redundant, and it probably is, I gave it a look but couldn't find a question here that fell in with what I wanted to know.
Basically we have a table with about ~50000 rows, and it's expected to grow much bigger than that. We need to be able to allow admin users to add in custom data to an item based on its category, and users can just pick which fields defined by the administrators they want to add info to.
Initially I had gone with an item_categories_fields table which pairs up entries from item_fields to item_categories, so admins can add custom fields and reuse them across categories for consistency. item_fields has a relationship to item_field_values which links values with fields, which is how we handled things in .NET. The project is using CAKEPHP though, and we're just learning as we go, so it can get a bit annoying at times.
I'm however thinking of maybe just adding an item_custom_fields table that is essentially the item_id and a text field that stores XMLish formatted data. This is just for the values of the custom fields.
No problems if I want to fetch the item by its id as the required data is stored in the items table, but what if I wanted to do a search based on a custom field? Would a
SELECT * FROM item_custom_fields
WHERE custom_data LIKE '%<material>Plastic</material>%'
(user input related issues aside) be practical if I wanted to fetch items made of plastic in this case? Like how slow would that be?
Thanks.
Edit: I was afraid of that as realistically this thing will be around 400k rows for that one table at launch, thanks guys.
Any LIKE query that starts with % will not use any indexes you have on the column, so the query will scan the whole table to find the result.
The response time for that depends highly on your machine and the size of the table, but it definitely won't be efficient in any shape or form.
Your previous/existing solution (if well indexed) should be quite a bit faster.