Related
I have the following problem that I'm trying to find the best solution for. Let's say I have a view such as the following:
CREATE VIEW myView AS (
SELECT
country_code,
other_column,
COUNT(1) as cnt
FROM mytable
JOIN otherDatabase.otherTable ON (id)
GROUP BY 1,2 ORDER BY 1 LIMIT 1
)
What would be the fastest way to resolve the field names and types of the view? For example, on the above I am looking to get something along the lines of:
{
country_code: VARCHAR,
other_column: BOOL,
cnt: INT
}
The first approach is just to run the query (with a limit, if necessary) and then get the types of the result-set from the driver. The downside of this is what if the query takes 50 minutes to resolve?
The second approach I thought of is to 'follow' the columns to get their types and then do some parsing to resolve any expressions/literals/etc. This would involve a lot of code but would be orders of magnitude faster than the above. However, the potential downside of this is we may have access to the view but not have access to a table (possibly in another database on the server) that contains the column type, so it's possible we might not be able to resolve all field names.
What would be the best way to resolve the types of a view? Note I have tagged this as MySQL, but I'm also wondering if there's a more generic way to resolve types or if it's something that is non-standard and more needs to be done on a per-database basis?
Update: I believe the correct answer is just to run a DESCRIBE myView, and that would give me the column names and types without running the query?
In the current version of MySQL at least, INFORMATION_SCHEMA.COLUMNS holds metadata for views as well as base tables:
mysql> create table mytable (id serial primary key, x int);
Query OK, 0 rows affected (0.01 sec)
mysql> create view v as select * from mytable;
mysql> select column_name, data_type from information_schema.columns where table_name='v';
+-------------+-----------+
| COLUMN_NAME | DATA_TYPE |
+-------------+-----------+
| id | bigint |
| x | int |
+-------------+-----------+
A related issue...
SHOW CREATE TABLE myView;
or
SHOW CREATE VIEW myView;
will fully qualify all the columns.
(When writing a JOIN, it is wise to always qualify the column names.)
I have read in the PostgreSQL docs that without an ORDER statement, SELECT will return records in an unspecified order.
Recently on an interview, I was asked how to SELECT records in the order that they inserted without an PK or created_at or other field that can be used for order. The senior dev who interviewed me was insistent that without an ORDER statement the records will be returned in the order that they were inserted.
Is this true for PostgreSQL? Is it true for MySQL? Or any other RDBMS?
I can answer for MySQL. I don't know for PostgreSQL.
The default order is not the order of insertion, generally.
In the case of InnoDB, the default order depends on the order of the index read for the query. You can get this information from the EXPLAIN plan.
For MyISAM, it returns orders in the order they are read from the table. This might be the order of insertion, but MyISAM will reuse gaps after you delete records, so newer rows may be stored earlier.
None of this is guaranteed; it's just a side effect of the current implementation. MySQL could change the implementation in the next version, making the default order of result sets different, without violating any documented behavior.
So if you need the results in a specific order, you should use ORDER BY on your queries.
Following BK's answer, and by way of example...
DROP TABLE IF EXISTS my_table;
CREATE TABLE my_table(id INT NOT NULL) ENGINE = MYISAM;
INSERT INTO my_table VALUES (1),(9),(5),(8),(7),(3),(2),(6);
DELETE FROM my_table WHERE id = 8;
INSERT INTO my_table VALUES (4),(8);
SELECT * FROM my_table;
+----+
| id |
+----+
| 1 |
| 9 |
| 5 |
| 4 | -- is this what
| 7 |
| 3 |
| 2 |
| 6 |
| 8 | -- we expect?
+----+
In the case of PostgreSQL, that is quite wrong.
If there are no deletes or updates, rows will be stored in the table in the order you insert them. And even though a sequential scan will usually return the rows in that order, that is not guaranteed: the synchronized sequential scan feature of PostgreSQL can have a sequential scan "piggy back" on an already executing one, so that rows are read starting somewhere in the middle of the table.
However, this ordering of the rows breaks down completely if you update or delete even a single row: the old version of the row will become obsolete, and (in the case of an UPDATE) the new version can end up somewhere entirely different in the table. The space for the old row version is eventually reclaimed by autovacuum and can be reused for a newly inserted row.
Without an ORDER BY clause, the database is free to return rows in any order. There is no guarantee that rows will be returned in the order they were inserted.
With MySQL (InnoDB), we observe that rows are typically returned in the order by an index used in the execution plan, or by the cluster key of a table.
It is not difficult to craft an example...
CREATE TABLE foo
( id INT NOT NULL
, val VARCHAR(10) NOT NULL DEFAULT ''
, UNIQUE KEY (id,val)
) ENGINE=InnoDB;
INSERT INTO foo (id, val) VALUES (7,'seven') ;
INSERT INTO foo (id, val) VALUES (4,'four') ;
SELECT id, val FROM foo ;
MySQL is free to return rows in any order, but in this case, we would typically observe that MySQL will access rows through the InnoDB cluster key.
id val
---- -----
4 four
7 seven
Not at all clear what point the interviewer was trying to make. If the interviewer is trying to sell the idea, given a requirement to return rows from a table in the order the rows were inserted, a query without an ORDER BY clause is ever the right solution, I'm not buying it.
We can craft examples where rows are returned in the order they were inserted, but that is a byproduct of the implementation, ... not guaranteed behavior, and we should never rely on that behavior to satisfy a specification.
when running
SELECT maxlen FROM `information_schema`.`CHARACTER_SETS`;
mysql 5.7 and mysql 8 produce different results:
on mysql 5.7 the results row names are lower cased,
on mysql 8 the results row names are upper cased.
NB : in the CHARACTER_SETS table, the comumn name is MAXLEN (upper cased).
Since I can't find a resource documenting it, my question is :
what are the changes in mysql 8 result rowset case ?
MySQL 8.0 did change the implementation of some views in the INFORMATION_SCHEMA:
https://mysqlserverteam.com/mysql-8-0-improvements-to-information_schema/ says:
Now that the metadata of all database tables is stored in transactional data dictionary tables, it enables us to design an INFORMATION_SCHEMA table as a database VIEW over the data dictionary tables. This eliminates costs such as the creation of temporary tables for each INFORMATION_SCHEMA query during execution on-the-fly, and also scanning file-system directories to find FRM files. It is also now possible to utilize the full power of the MySQL optimizer to prepare better query execution plans using indexes on data dictionary tables.
So it's being done for good reasons, but I understand that it has upset some of your queries when you fetch results in associative arrays based on column name.
You can see the definition of the view declares the column name explicitly in uppercase:
mysql 8.0.14> SHOW CREATE VIEW CHARACTER_SETS\G
*************************** 1. row ***************************
View: CHARACTER_SETS
Create View: CREATE ALGORITHM=UNDEFINED DEFINER=`mysql.infoschema`#`localhost` SQL SECURITY DEFINER VIEW `CHARACTER_SETS` AS
select
`cs`.`name` AS `CHARACTER_SET_NAME`,
`col`.`name` AS `DEFAULT_COLLATE_NAME`,
`cs`.`comment` AS `DESCRIPTION`,
`cs`.`mb_max_length` AS `MAXLEN` -- delimited column explicitly uppercase
from (`mysql`.`character_sets` `cs`
join `mysql`.`collations` `col` on((`cs`.`default_collation_id` = `col`.`id`)))
character_set_client: utf8
collation_connection: utf8_general_ci
You can work around the change in a couple of ways:
You can declare your own column aliases in the case you want when you query a view:
mysql 8.0.14> SELECT MAXLEN AS `maxlen`
FROM `information_schema`.`CHARACTER_SETS` LIMIT 2;
+--------+
| maxlen |
+--------+
| 2 |
| 1 |
+--------+
You could start a habit of querying columns in uppercase prior to 8.0. Here's a test showing results in my 5.7 sandbox:
mysql 5.7.24> SELECT MAXLEN
FROM `information_schema`.`CHARACTER_SETS` LIMIT 2;
+--------+
| MAXLEN |
+--------+
| 2 |
| 1 |
+--------+
Or you could fetch results into a non-associative array, and reference columns by column number, instead of by name.
There is no change in case sensitivity. If you check mysql documentation on identifier case sensitivity, both v5.7 and v8.0 say that field names are case insensitive:
Column, index, stored routine, event, and resource group names are not case-sensitive on any platform, nor are column aliases.
To me this seems more like a display difference.
First, I will describe a simplified version of the problem domain.
There is table strings:
CREATE TABLE strings (
value CHAR(3) COLLATE utf8_unicode_ci NOT NULL,
INDEX(value)
) ENGINE=InnoDB;
As you can see, it have a non-unique index of CHAR(3) column.
The table is populated using the following script:
CREATE TABLE a_variants (
letter CHAR(1) COLLATE utf8_unicode_ci NOT NULL
) ENGINE=MEMORY;
INSERT INTO a_variants VALUES -- 60 variants of letter 'A'
('A'),('a'),('À'),('Á'),('Â'),('Ã'),('Ä'),('Å'),('à'),('á'),('â'),('ã'),
('ä'),('å'),('Ā'),('ā'),('Ă'),('ă'),('Ą'),('ą'),('Ǎ'),('ǎ'),('Ǟ'),('ǟ'),
('Ǡ'),('ǡ'),('Ǻ'),('ǻ'),('Ȁ'),('ȁ'),('Ȃ'),('ȃ'),('Ȧ'),('ȧ'),('Ḁ'),('ḁ'),
('Ạ'),('ạ'),('Ả'),('ả'),('Ấ'),('ấ'),('Ầ'),('ầ'),('Ẩ'),('ẩ'),('Ẫ'),('ẫ'),
('Ậ'),('ậ'),('Ắ'),('ắ'),('Ằ'),('ằ'),('Ẳ'),('ẳ'),('Ẵ'),('ẵ'),('Ặ'),('ặ');
INSERT INTO strings
SELECT CONCAT(a.letter, b.letter, c.letter) -- 60^3 variants of string 'AAA'
FROM a_variants a, a_variants b, a_variants c
UNION ALL SELECT 'BBB'; -- one variant of string 'BBB'
So, it contains 216000 indistinguishable (in terms of the utf8_unicode_ci collation) variants of string "AAA" and one variant of string "BBB":
SELECT value, COUNT(*) FROM strings GROUP BY value;
+-------+----------+
| value | COUNT(*) |
+-------+----------+
| AAA | 216000 |
| BBB | 1 |
+-------+----------+
As value is indexed, I expect the following two queries to have similar performance:
SELECT SQL_NO_CACHE COUNT(*) FROM strings WHERE value = 'AAA';
SELECT SQL_NO_CACHE COUNT(*) FROM strings WHERE value = 'BBB';
But in practice the first one is more than 300x times slower than the second! See:
+----------+------------+---------------------------------------------------------------+
| Query_ID | Duration | Query |
+----------+------------+---------------------------------------------------------------+
| 1 | 0.11749275 | SELECT SQL_NO_CACHE COUNT(*) FROM strings WHERE value = 'AAA' |
| 2 | 0.00033325 | SELECT SQL_NO_CACHE COUNT(*) FROM strings WHERE value = 'BBB' |
| 3 | 0.11718050 | SELECT SQL_NO_CACHE COUNT(*) FROM strings WHERE value = 'AAA' |
+----------+------------+---------------------------------------------------------------+
-- I ran the 'AAA' query twice here just to be sure.
If I change size of the indexed column or change its type to VARCHAR, the problem with performance still manifests itself. Meanwhile, in analogous situations, but when the non-unique index is not CHAR/VARCHAR (e.g. INT), queries are as fast as expected.
So, the question is why performance of MySQL queries are so bad when using a CHAR/VARCHAR index?
I have strong feeling that MySQL perform full linear scan of all the values matched by the index key. But why it do so when it can just return the count of the matched rows? Am I missing something and that is really needed? Or is that a sad shortcoming of MySQL optimizer?
Clearly, the issue is that the query is doing an index scan. The alternative approach would be to do two index lookups, for the first and last values that are the same, and then use meta information in the index for the calculation. Based on your observations, MySQL does both.
The rest of this answer is speculation.
The reason the performance is "only" 300 times slower, rather than 200,000 times slower, is because of overhead in reading the index. Actually scanning the entries is quite fast compared to other operations that are needed.
There is a fundamental difference between numbers and strings when it comes to comparisons. The engine can just look at the bit representations of two numbers and recognize whether they are the same or different. Unfortunately, for strings, you need to take encoding/collation into account. I think that is why it needs to look at the values.
It is possible that if you had 216,000 copies of exactly the same string, then MySQL would be able to do the count using metadata in the index. In other words, the indexer is smart enough to use metadata for exact equality comparisons. But, it is not smart enough to take encoding into account.
One of things you may want to check on is the logical I/O of each query. I'm sure you'll see quite a difference. To count the number of 'BBB's in the table, probably only 3 or 4 LIOs are needed (depending on things like bucket size). To count the number of 'AAA's, essentially the entire table must be scanned, index or not. With 216k rows, that can add up to significantly more LIOs -- not to mention physical I/Os. Logical I/Os are faster than physical I/Os, but any I/O is a performance killer.
As for text vs numbers, it is always easier and faster for software (any software, not just database engines) to compare numbers than text.
I have a table in a MySQL database. I am given a value that occurs as a cell value in that table but I do not know which cell is it i.e. the row and column of that cell. What is the most efficient way to find the column to which that value belongs? Thanks in advance.
Example:
Column_1 | Column_2 | Column_3
1 | 2 | 3
4 | 5 | 6
7 | 8 | 9
Now I am given an input value of "8". I want to know if there is an efficient way to find out that value of "8" belongs to Column_2.
It's a bit strange that you don't know which column the data is in, since columns are meant to have a well-defined function.
[Original response scrubbed.]
EDIT: Your updated post just asks for the column. In that case, you don't need the view, and can just run this query
SELECT col FROM (
SELECT "Column_1" AS col, Column_1 AS value FROM YourTable
UNION ALL SELECT "Column_2", Column_2 FROM YourTable
UNION ALL SELECT "Column_3", Column_3 FROM YourTable
) allValues
WHERE value=8;
When you run this query against your table, it will return "Column_2"
Without knowing more about your app, you have several options:
Use MySQL's built-in full-text search. You can check the MATCH function in the MySQL documentation.
Depending on the needs of your app you could decide to index your whole table with an external full-text search index, like Solr or Sphynx. This provides instant response time, but you'll need to keep the index updated.
You can loop through all the columns in the table doing a LIKE query in MySQL (very expensive in CPU and time)
You're designing this table with repeating groups, which is not satisfying First Normal Form.
You should create a second table and store the values for column1, column2, and column2 in a single column, on three rows.
Learn about the rules of database normalization for more details.