I'm not even sure it's possible to do this but I want to order a query based on the maximum value of one of three columns.
Example table structure:
guid, column1, column2, column3
Columns 1-3 have numerical values and I want to order the select statement based on the maximum value of 1, 2 OR 3.
For example:
record column1 column2 column3
---------------------------------
1 5 0 2
2 2 0 6
3 0 1 2
Would be ordered record 2, 1, 3 because 6 is the maximum value of the three fields across the three records, record 1 is the second and record 3 is the third.
Does this make any sense?
Thanks.
It may be possible to do in a select query (possibly using something like case when though I'm not sure that's allowed in the order by clause itself, YMMV depending on the DBMS) but it's rarely a good idea to use per-row calculations if you want your database to scale well as tables get bigger ("not have the performance of a one-legged pig in a horse race", as one of our DBAs eloquently puts it).
In situations like this, I set up an additional (indexed) column to hold the maximum and ensure that the data integrity is maintained by using an insert/update trigger to force that new column to the maximum of the other three.
Because most database tables are read far more often than written, this amortises the cost of the calculation across all the reads. The cost is borne only when the data is updated and the queries become blindingly fast since you're ordering on a single, indexed, column:
select f1, f2, f3 from t order by fmax desc;
As mentioned here, what you want is an equivalent of the GREATEST function.
In the absence of that, and assuming you've defined a UDF LargerOf to return the largest of two numbers, use
SELECT *
FROM Table
ORDER BY LargerOf(LargerOf(column1, column2), column3)
create table myTable
(column1 int, column2 int, column3 int)
go
insert into myTable
values (5, 0 , 2)
go
insert into myTable
values (2, 0 , 6)
go
insert into myTable
values (0, 1 , 2)
go
select *
from mytable
order by case when column1 > column2 and column1 > column3 then column1
when column2 > column3 then column2
else column3 end desc
Related
I want to perform a MySQL query on a large table (500+ million rows). A particular column has thousands of possible element values I want to return all the rows where this column is just 100 of these values.
Lets say this column1 can have these values: A1, A2, A3, .., A12309. I want to do a query like so:
SELECT column2 WHERE column1='A1' OR column1='A2'... OR column1='A100'
Having 100 different OR operators is highly inefficient.
Is there a better way to structure this query?
Is MySQL suited for this or is there a better DBMS / data analysis tool?
SELECT column2 FROM tbl
WHERE column1 IN ('A1', 'A2', ... 'A100')
This will work reasonably well for 100 values.
This should help, too:
INDEX(column1, column2)
Which version of MySQL? (Some versions have a cutoff -- above some number of values in the IN leads to a less efficient execution plan, especially for 500M rows.)
I need to create more than 64 indexes on a table but am running into the "Too many keys specified; max 64 keys allowed" error. Is there some workaround that would allow me to increase this limit beyond 1000 for MariaDb / TokuDB? or is there a reason why this limit is necessary?
(I've seen this question asked/answered for MySQL - with the answers being either to pass --with-max-indexes=256 to ./configure, or modify MAX_KEY in one of the header files at compile time. Unfortunately, these answers don't appear to work for MariaDB)
Ps. Since a typical response is "if you need this many indexes you're doing something wrong", I'll explain why I want to do this, and would appreciate any advice on modifying the design if that's the best "workaround".
My data is stored in 2 tables:
table1 stores 5 columns: (unique key x_position int, column1 string, column2 float, column3 int, column4 tinyint) - it can be as large as 100 million rows
table2 conceptually can be represented as 4 columns: (foreign key x_position int, sample_id string, value1 tinyint, value2 float) - since there could be up to 5000 unique sample_id values, and a different value1 would exist for each (x_position, sample_id) pair, the max number of rows would be 100 million x 5000 = 500 billion rows
The queries I need to do are like:
select column1, column2, column3... sample_id,
group_concat(value1)
from table1, table2
where column1 = string1
and column2 < float2
and ( (sample_id = string1 and value1=1)
or (sample_id = string2 and value1=0)
or (sample_id = string3 and value1=1)
)
and value2 < float1
group by sample_id;
Instead, I was thinking it would be more efficient to pivot table2 so that it's columns are: (foreign key x_position, sample_id1_value1 tinyint, sample_id1_value2 float, sample_id2_value1 tinyint, sample_id2_value2 float, ...)
and then create composite indexes on small subsets of the (sample_id1_value1, sample_id1_value2, .. ) columns based on domain-specific details of which of these columns will be queried together. This table would have 100 million rows x 10,000 columns (split across several tables to avoid the column limit) which seems better than 500 billion rows. Also it would eliminate the need for "or" and "group by" clauses in the queries, allowing queries to be rewritten like:
select column1, column2, column3... sample_id,
sample_id1_value1,
sample_id1_value2
from table1, table2
where column1 = string1
and column2 < float2
and sample_id1_value1=1
and sample_id2_value1=0
and sample_id3_value1=1
and sample_id1_value2 < float1
and sample_id2_value2 < float1
and sample_id3_value2 < float1;
Unfortunately the "Too many keys" error is getting in the way of this.
I've two tables one is the main table having data and I want to insert data from another existing table having about 13 million records. I'm using the query to insert from another table i.e.
insert into table1 ( column1, col2 ...) select col1, col2... from table2;
But, unfortunately the query fails as lock wait timeout comes Error 1205.
What is the best way to do it in least time without timeout.
If you have a primary key on table2, then you can use that for ordering and inserting in batches:
insert into table1 ( column1, col2 ...)
select col1, col2...
from table2
order by <primary key>
limit 0, 100000
Then repeat this for additional values. (Of course, the 100,000 is arbitrary. A larger value might work. A smaller value might be necessary.)
Another possibility is to remove all indexes and insert triggers from table1, try the insert without them, and then add them back after the new data is in the table.
I have two tables with identical (6) columns, and varying entries. I would like to obtain a "master" table with the same columns, including every unique entry from table A and B. There exists no primary or foreign keys on either table, and the "uniqueness" of an entry is determined by each of the 6 fields being identical to another entry. In other words, if entry x has column 1 equal to entry y's column 1, and all remaining columns are equal as well, these two entries are considered non-unique, whether they exist in the same table or separate tables. I've researched and found similar solutions, but none that quite solve this issue. Any thoughts?
A UNION is definitely what's needed here, but there are a few extraneous items in the query from #PhilCross:
The GROUP BY isn't needed to flatten the results because the UNION does this naturally when selecting on all columns.
Similarly, the HAVING isn't needed either.
The column aliasing in the UNION SELECT query will be ignored by MySQL because the first SELECT list determines to column names for the result. All you need for a UNION is (a) the same number of columns in all SELECT statements and (b) compatible data types for the corresponding columns - either the same or implicitly castable.
The parentheses aren't needed either, but you should include them if it makes the query more readable for you.
So all you really need is the following:
SELECT column1 AS column1, column2 AS column2, column3 AS column3
FROM table1
UNION SELECT column1, column2, column3
FROM table2
You can just use a union statement:
(
SELECT column1 AS column1, column2 AS column2, column3 AS column3
FROM table1
) UNION (
SELECT column1 AS column1, column2 AS column2, column3 AS column3
FROM table2
)
GROUP BY column1, column2, column3
HAVING COUNT(column1, column2, column3)>0
I've been trying to figure this out, but can't seem to come up with a simple solution.
Say for instance I have a table that has similar data throughout 3 columns (i.e. different types of activities spanning 3 columns) but I want to have those three columns inserted into a separate table (Table2) so I can keep the like data together and perform a JOIN to match it with its respective data in Table1.
I'm not talking about performing a CONCAT or CONCAT_WS, but moving those three columns from Table1 into one column in Table2, each item with its own row.
Is there a way to do this through a query without having to manually insert each entry into Table2?
Thank you in advance!
It might be as simple as:
insert into table2
(field)
select column1 from table1
union
select column2 from table1
union
select column3 from table1
But, before you do this, decide what you want to do if two columns in table1 have the same value.