what is the best approach to find duplicates in my Db table - mysql

In my app the user can select multiple filter options. I store this in a DB table.
For example
User 1 can select filters A^B
User 2 can select filters AORC^D
and so forth.
The way it is stored in Db is
user filter_selected
user1 A^B
user2 AORC^D
Now the criteria is no user can have the same filters selected. So if user 3 comes and select A^B or B^A it should throw a error.
I am trying to come up with a smart logic to validate this in javascript.
One approach is go through all the users in the DB (can be many) and sort alphabetically and check if its the same. So in our example A^B and B^A will be the same AB^. This way I can check. Any other better approach may be using mysql command itself ?

you can sort your filter rule based on character and then insert it to do
for example, B^A will convert to AB^ and when you want to check you can sort your filter and then search it
if you want to have an original filter you don't care about the size of your database and more you care about speed you can save original as another column too.if you are care about size of database you can just save the original filter and when you want to search select the rows that have the same length as your filter and then you need to sort alphabetically or you can save index of every filter chars for example when you change A^B to AB^ you can save this filter AB^|021 but this will need to some more space too like original column and I don't suggest this method. also if your filters are always in small length you can don't fetch all record and compare to all. you can just create all possible way of the filter(for example AB^ A^B B^A BA^ ^AB ^BA) but you must be careful because in this method you are creating n! string and this is not good at all, just for too small length string its ok and that's when you have too many records in your database this method can be good

Related

Use a another table's column's value as the name of a table in a join statement

First, I'll ask the question:
Is it possible to make a value's type 'dynamic' by creating different tables with different value types and use a simple join query to fetch the correct table based on the type specified in a column?
Here is the context:
I'm building a sort of page builder where I want the "modules" to be dynamic in the sense that I can create new modules without having to modify the database in any way.
I envision 4 entities:
A field, which represents a single value of a specific type, for example a text field or a checkbox;
An entry type, which represents a group of fields to be used for an entry. For example, Article = Title + Content + Image.
An entry, which is the content defined by the corresponding entry type.
A value, which contains the data for one field for one entry, defined by the field's value type.
For the 'value' entity, I was planning on using multiple tables for each "type of value". "value_text", "value_integer", "value_boolean", etc.
What I wanted to do (and I think it's an anti-pattern in relational database) was to grab a value in the fields table to select the proper "value" table.
Here is an image of the structure I envisioned:
(imgur) UML Diagram of DB
So I'm looking for a way to make a single query to fetch the correct value for each field of a specific entry (by ID).
This means I have to access the assigned entry_type, fetch all the fields related to that entry_type, then fetch the value of each of those fields from the value tables based on the column 'value_type' of the fields table.
If this is possible, how can I achieve this? If it's not possible, what would be the best way to tackle this problem:
Make a first query to retrieve the fields, then a query for each field to grab the correct value? This makes a lot of queries to get the values of a single entry...
Modify the structure of the database to a more efficient set of relations, taking into account everything is dynamic...
Get rid of the idea of using mutiple "value types" and stick to using a more common everything-is-serialized-text approach.
Also, as this is one of my first questions on StackOverflow, please let me know if I asked incorrectly and how I should formulate this in the future.
Thanks for all of you who even take the time to read this :)

Creating a global variable in Talend to use as a filter in another component

I have job in Talend that is designed to bring together some data from different databases: one is a MySQL database and the other a MSSQL database.
What I want to do is match a selection of loan numbers from the MySQL database (about 82,000 loan numbers) to the corresponding information we have housed in the MSSQL database.
However, the tables in MSSQL to which I am joining the data from MySQL are much larger (~ 2 million rows), are quite wide, and thus cost much more time to query. Ideally I could perform an inner join between the two tables based on the loan number, but since they are in different databases this is not possible. The inner join that is performed inside a tMap occurs after the Lookup input has already returned its data set, which is quite large (especially since this particular MSSQL query will execute a user-defined function for each loan number).
Is there any way to create a global variable out of the output from the MySQL query (namely, the loan numbers selected by the MySQL query) and use that global variable as an IN clause in the MSSQL query?
This should be possible. I'm not working in MySQL but I have something roughly equivalent here that I think you should be able to adapt to your needs.
I've never actually answered a Stackoverflow question and while I was typing this the page started telling me I need at least 10 reputation to post more than 2 pictures/links here and I think I need 4 pics, so I'm just going to write it out in words here and post the whole thing complete with illustrations on my blog in case you need more info (quite likely, I should think!)
As you can see, I've got some data coming out of the table and getting filtered by tFilterRow_1 to only show the rows I'm interested in.
The next step is to limit it to just the field I want to use in the variable. I've used tMap_3 rather than a tFilterColumns because the field I'm using is a string and I wanted to be able to concatenate single quotes around it but if you're using an integer you might not need to do that. And of course if you have a lot of repetition you might also want to get a tUniqueRows in there as well to save a lot of unnecessary repetition
The next step is the one that does the magic. I've got a list like this:
'A1'
'A2'
'B1'
'B2'
etc, and I want to turn it into 'A1','A2','B1','B2' so I can slot it into my where clause. For this, I've used tAggregateRow_1, selecting "list" as the aggregate function to use.
Next up, we want to take this list and put it into a context variable (I've already created the context variable in the metadata - you know how to do that, right?). Use another tMap component, feeding into a tContextLoad widget. tContextLoad always has two columns in its schema, so map the output of the tAggregateRows to the "value" column and enter the name of the variable in the "key". In this example, my context variable is called MyList
Now your list is loaded as a text string and stored in the context variable ready for retrieval. So open up a new input and embed the variable in the sql code like this
"SELECT distinct MY_COLUMN
from MY_SECOND_TABLE where the_selected_row in ("+
context.MyList+")"
It should be as easy as that, and when I whipped it up it worked first time, but let me know if you have any trouble and I'll see what I can do.

How to add a calculated column using conditions in a query in MS Access 2010

I have two tables and from that I am generating a query. I have columns in my query with field type yes/no. These are basically columns to determine the race of a person. The user enters information in yes/no for various races. I want another calculated column in the query which checks for all other race columns and calculates values in it.
I have to check for a few conditions in for the values in columns
For example:
1) If Hispanic is chosen, the new column should say hispanic(no matter what other options are selected. This is like a trump card)
2) If more than one is selected, then new column entry should say "multi"
3) If none of the options are selected, it should say "unknown"
4) If exactly one of them is selected, then that race should be displayed
Can anyone help me with this? I am new to Access
I can't code it for you but I can point you in the right direction. What you want to do is take all the tests you explained above and put them in a coded format :
iif ( condition, value_if_true, value_if_false )
Since you have a lot of possible outputs i'd use something like a Case Statement where you can test for all the possibilities.
Follow this link if you need any info on how to code both type of statements (iif and case).
Once you have tried something like this, you can comeback with a specific question if you encountered a problem in the process.
Good luck with your database.

How do I speed up this Select Distinct query in mysql

I am implementing an autocomplete field in a website, and my endpoint queries a mysql to get the data. This particular field can only be retrieved from a database 'list' which has a about 600,000 entries, where only 4000 of them are unique. Once a day or so, a new unique list item will be added.
I'm using a call like this, where query is whatever the user has started typing in the autocomplete box:
SELECT DISTINCT item FROM list WHERE item like '%query%' LIMIT 10;
This query takes about .8 milliseconds, and the majority of the time is due to the SELECT DISTINCT, I believe. Is there any way to increase the performance of this task?
I'm fine to create new tables/views, but I'm not sure what will help.
When you use the % sign at the beginning in a query with LIKE no index will be used, and every single row of your table will be scanned, no matter how your indexes look like.
Think about an address book with letter tabs, if I'm looking for 'Mads%', I can point my finger to the letter M and read from there, but If I look for '%dse%', I've to read every entry.
If you can you should drop the percent sign at the beginning of the string, in this case the index on item will be used (be sure to add it, you can also index a part of the field in case it's a long field).
Otherwise, you should use full text search instead.

MySQL: Storing a value from a field that may show up variable times on a form

I'm creating a form in CodeIgniter that has a paired value- Companies, and CEOs. The idea is, while there is only one Company field with one CEO field below initially on the form, there is a button for me to create more pairs of Company-CEO fields. I have an idea for how to make that appear on the frontend, but I'm not quite sure which is the best way to store it on the backend.
I think in order to combine each Company and CEO field into one value for storing in the MySQL database, I could simply use CONCAT. However, I'm not sure if I should even bother creating such a value. In any case, how would I handle the fact that there are a variable number of Companies (and thus CEOs) for each record? Would I have to store Company and CEO within another value? How could such an array or tuple be handled in MySQL?
Point 1: Do not combine the values. It will give you headaches in code splitting and combining. Make two columns, one for company, one for CEO.
Point 2: In general, when using a database, any time you have a list it gets its own table.
The various JS libraries out there let you handle this with a grid.