I want to count how many columns in a row are not NULL.
The table is quite big (more than 100 columns), therefore I would like to not do it manually or using php (since I dont use php) using this approach Counting how many MySQL fields in a row are filled (or empty).
Is there a simple query I can use in a select like SELECT COUNT(NOT ISNULL(*)) FROM big_table;
Thanks in advance...
Agree with comments above:
There is something wrong in the data since there is a need for such analysis.
You can't completely make it automatic.
But I have a recipe for you for simplifying the process. There are only 2 steps needed to achieve your aim.
Step 0. In the step1 you'll need to get the name of your table schema. Normally, the devs know in what schema does the table reside, but still... Here is how you can find it
select *
from information_schema.tables
where table_name = 'test_table';
Step 1. First of all you need to get the list of columns. Getting just the list of cols won't help you out at all, but this list is all we need to be able to create SELECT statement, right? So, let's make database to prepare select statement for us
select concat('select (length(concat(',
group_concat(concat('ifnull(', column_name, ', ''###'')') separator ','),
')) - length(replace(concat(',
group_concat(concat('ifnull(', column_name, ', ''###'')') separator ','),
'), ''###'', ''''))) / length(''###'')
from test_table')
from information_schema.columns
where table_schema = 'test'
and table_name = 'test_table'
order by table_name,ordinal_position;
Step 3. Execute statement you've got on step 2.
select (length(concat(.. list of cols ..)) -
length(replace(concat(.. list of cols .. ), '###', ''))) / length('###')
from test_table
The select looks tricky but it's simple: first replace all nulls with some symbols that you're sure you'll never get in those columns. I usually do that replacing nulls with "###". that what all that "ifnull"s are here for.
Next, count symbols with "length". In my case it was 14
After that, replace all "###" with blanks and count length again. It's 11 now. For that I was using "length(replace" functions together
Last, just divide (14 - 11) by a length of a replacement string ("###" - 3). You'll get 1. This is exactly amount of nulls in my test string.
Here's a test case you can play with
Do not hesitate to ask if needed
I have a column in one of my table where I store multiple ids seperated by comma's.
Is there a way in which I can use this column's value in the "IN" clause of a query.
The column(city) has values like 6,7,8,16,21,2
I need to use as
select * from table where e_ID in (Select city from locations where e_Id=?)
I am satisfied with Crozin's answer, but I am open to suggestions, views and options.
Feel free to share your views.
Building on the FIND_IN_SET() example from #Jeremy Smith, you can do it with a join so you don't have to run a subquery.
SELECT * FROM table t
JOIN locations l ON FIND_IN_SET(t.e_ID, l.city) > 0
WHERE l.e_ID = ?
This is known to perform very poorly, since it has to do table-scans, evaluating the FIND_IN_SET() function for every combination of rows in table and locations. It cannot make use of an index, and there's no way to improve it.
I know you said you are trying to make the best of a bad database design, but you must understand just how drastically bad this is.
Explanation: Suppose I were to ask you to look up everyone in a telephone book whose first, middle, or last initial is "J." There's no way the sorted order of the book helps in this case, since you have to scan every single page anyway.
The LIKE solution given by #fthiella has a similar problem with regards to performance. It cannot be indexed.
Also see my answer to Is storing a delimited list in a database column really that bad? for other pitfalls of this way of storing denormalized data.
If you can create a supplementary table to store an index, you can map the locations to each entry in the city list:
CREATE TABLE location2city (
location INT,
city INT,
PRIMARY KEY (location, city)
);
Assuming you have a lookup table for all possible cities (not just those mentioned in the table) you can bear the inefficiency one time to produce the mapping:
INSERT INTO location2city (location, city)
SELECT l.e_ID, c.e_ID FROM cities c JOIN locations l
ON FIND_IN_SET(c.e_ID, l.city) > 0;
Now you can run a much more efficient query to find entries in your table:
SELECT * FROM location2city l
JOIN table t ON t.e_ID = l.city
WHERE l.e_ID = ?;
This can make use of an index. Now you just need to take care that any INSERT/UPDATE/DELETE of rows in locations also inserts the corresponding mapping rows in location2city.
From MySQL's point of view you're not storing multiple ids separated by comma - you're storing a text value, which has the exact same meaing as "Hello World" or "I like cakes!" - i.e. it doesn't have any meaing.
What you have to do is to create a separated table that will link two objects from the database together. Read more about many-to-many or one-to-many (depending on your requirements) relationships in SQL-based databases.
Rather than use IN on your query, use FIND_IN_SET (docs):
SELECT * FROM table
WHERE 0 < FIND_IN_SET(e_ID, (
SELECT city FROM locations WHERE e_ID=?))
The usual caveats about first form normalization apply (the database shouldn't store multiple values in a single column), but if you're stuck with it, then the above statement should help.
This does not use IN clause, but it should do what you need:
Select *
from table
where
CONCAT(',', (Select city from locations where e_Id=?), ',')
LIKE
CONCAT('%,', e_ID, ',%')
but you have to make sure that e_ID does not contain any commas or any jolly character.
e.g.
CONCAT(',', '6,7,8,16,21,2', ',') returns ',6,7,8,16,21,2,'
e_ID=1 --> ',6,7,8,16,21,2,' LIKE '%,1,%' ? FALSE
e_ID=6 --> ',6,7,8,16,21,2,' LIKE '%,6,%' ? TRUE
e_ID=21 --> ',6,7,8,16,21,2,' LIKE '%,21,%' ? TRUE
e_ID=2 --> ',6,7,8,16,21,2,' LIKE '%,2,%' ? TRUE
e_ID=3 --> ',6,7,8,16,21,2,' LIKE '%,3,%' ? FALSE
etc.
Don't know if this is what you want to accomplish. With MySQL there is feature to concatenate values from a group GROUP_CONCAT
You can try something like this:
select * from table where e_ID in (Select GROUP_CONCAT(city SEPARATOR ',') from locations where e_Id=?)
this one in for oracle ..here string concatenation is done by wm_concat
select * from table where e_ID in (Select wm_concat(city) from locations where e_Id=?)
yes i agree with raheel shan .. in order put this "in" clause we need to make that column into row below code one do that job.
select * from table where to_char(e_ID)
in (
select substr(city,instr(city,',',1,rownum)+1,instr(city,',',1,rownum+1)-instr(city,',',1,rownum)-1) from
(
select ','||WM_CONCAT(city)||',' city,length(WM_CONCAT(city))-length(replace(WM_CONCAT(city),','))+1 CNT from locations where e_Id=? ) TST
,ALL_OBJECTS OBJ where TST.CNT>=rownum
) ;
you should use
FIND_IN_SET Returns position of value in string of comma-separated values
mysql> SELECT FIND_IN_SET('b','a,b,c,d');
-> 2
You need to "SPLIT" the city column values. It will be like:
SELECT *
FROM table
WHERE e_ID IN (SELECT TO_NUMBER(
SPLIT_STR(city /*string*/
, ',' /*delimiter*/
, 1 /*start_position*/
)
)
FROM locations);
You can read more about the MySQL split_str function here: http://blog.fedecarg.com/2009/02/22/mysql-split-string-function/
Also, I have used the TO_NUMBER function of Oracle here. Please replace it with a proper MySQL function.
IN takes rows so taking comma seperated column for search will not do what you want but if you provide data like this ('1','2','3') this will work but you can not save data like this in your field whatever you insert in the column it will take the whole thing as a string.
You can create a prepared statement dynamically like this
set #sql = concat('select * from city where city_id in (',
(select cities from location where location_id = 3),
')');
prepare in_stmt from #sql;
execute in_stmt;
deallocate prepare in_stmt;
Ref: Use a comma-separated string in an IN () in MySQL
Recently I faced the same problem and this is how I resolved it.
It worked for me, hope this is what you were looking for.
select * from table_name t where (select (CONCAT(',',(Select city from locations l where l.e_Id=?),',')) as city_string) LIKE CONCAT('%,',t.e_ID,',%');
Example: It will look like this
select * from table_name t where ',6,7,8,16,21,2,' LIKE '%,2,%';
I'm relatively new to SQL and I've been having trouble trying to figure out how to select rows from a table that matches a string.
The name of the table uses the current month and year.
It looks like this:
XXXX.xxx_YY_MM
where XXXX is the database, YY is year, and MM is month.
So normally the query would just look like this:
select * from XXXX.xxx_16_05;
However, I want it to be able to change depending on the date.
So I thought this would work:
select * from (select concat('XXXX.xxx_',date_format(now(), '%y_%m'))));
The concat bit gives me something that looks exactly like the name of the table. But it doesn't work and I'm not sure why. It says every table must have it's own alias. I'm not sure what to do about it.
Alternatively, I was thinking maybe something like this would be ok
select * from (select * from information_schema.tables where table_name like concat('%logsms_',date_format(now(), '%y_%m'),'%'));
But it doesn't work either. What should I do? There is only one table with a name that matches the string.
You can't use an expression for the table name in SQL. You need to use a stored procedure that creates dynamic SQL and executes it using PREPARE and EXECUTE.
SET #sql = CONCAT('SELECT * FROM XXXX.xxx_', DATE_FORMAT(NOW(), '%y_%m'));
PREPARE stmt FROM #sql;
EXECUTE stmt;
A database design that requires this seems like a poor decision. You should have a single table where the date is in a column, not separate tables for each month.
If you're running the query from a programming language, you can use its own date functions to constructure SQL.
Please find the following query to retrieve all table by matching string.
select table_schema as database_name,table_name from information_schema.tables
where table_name like '%your_table_name%' order by table_schema, table_name;
I am new to SQL and I realize I have a lot to learn. I am in the midst of converting a set of ISAM files into MySQL tables. Some of the tables I am creating have hundreds of columns. To make it easier to see if my conversions are working properly, I would like to display only a subset of columns at a time in my SELECT results.
For example, I would like to see only Dates or only Amounts (i.e., Decimals). This would make it easier to see if correct values are in those fields.
I am working from the MySQL command line and I would rather not specify a long list of column names if it can be avoided. Is there a way to get my Select statements to display only the types of columns I am interested in?
(Added 5/12/16)
I used a variant of Uueerdo's syntax to create a view:
create view dates_view as concat('select ', group_concat(column_name), ' ',' from ', table_schema, '.', table_name, ';') from information_schema.columns where table_name = 'binders_tbl' and data_type in ('date');
When I enter select * from dates_view; I get this value:
select binder_effectiveDate,binder_expirationDate,binder_submissionCreatedDate,binder_firstQuotedDate,binder_boundDate,binder_invoicedDate,binder_terminatedDate,binder_cancelledDate from aesdb.binders_tbl;
I tried but could not find a way to use this view with "select * from binders_tbl;" to display only the above date fields and their values.
If anyone can guide me to a "join" or "subquery" solution, I would appreciate it. For now, I am going to stick with listing all the individual columns as needed. Plus, I will study more basic and intermediate SQL. Thank you to those who responded.
(Added 5/20/16)
I found a sort-of work-around to my problem. Here it is in case it helps someone else:
CREATE VIEW DateCols_View AS SELECT Table1_SelectionCriteriaField, Table1_Date1, Table1_Date2, Table1_Date3 FROM Table1;
SELECT * FROM DateCols_View WHERE Table1_SelectionCriteriaField (matches some criteria);
My work-around has 2 drawbacks. The first is that I have to include any selection criteria fields that I plan to use later. But with a little forethought I can create multiple views with different sets of columns for different displays. This will let me display only those columns I wish to see while working with a set of records. I plan to create separate views for Dates, Amounts, Flags, etc.
The other drawback is that I have to create a separate set of views for each table or join combination I plan to work with.
I hope as I learn more about SQL, I will find a way to generalize this technique (to make it more like a script or macro) and simplify it further.
Nope, you have to specify column names. There is no syntax for dynamic field inclusion, however you could maybe work something out with information_schema (like below) to generate a query for you to use.
SELECT CONCAT('SELECT ', GROUP_CONCAT(column_name), ' '
,'FROM `', table_schema, '`.`', table_name, '`;'
FROM information_schema.columns
WHERE table_schema = 'yourschemaname'
AND table_name = 'yourtablename'
AND data_type IN ('timestamp', 'date'[, etc....])
;
I would like to rename the columns in the results of a SELECT expression. Of course, the following doesn't work:
SELECT * AS foobar_* FROM `foobar`
As I'm new to SQL, I think I'm just missing a concept, tool, or keyword that would lead to the answer. A hint in the right direction would be appreciated. Thanks!
UPDATE
I'm looking for a generic way to do this, and MySQL-specific techniques are absolutely fine.
In short, I'm writing a tool that "exports" the results of MySQL queries to Google Spreadsheets (via the Google Data API). Some queries are joins, so to make columns unique I wanted to prefix all column names with their respective table names.
You can alias the column names one by one, like so
SELECT col1 as `MyNameForCol1`, col2 as `MyNameForCol2`
FROM `foobar`
Edit You can access INFORMATION_SCHEMA.COLUMNS directly to mangle a new alias like so. However, how you fit this into a query is beyond my MySql skills :(
select CONCAT('Foobar_', COLUMN_NAME)
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME = 'Foobar'
you have to rename each column
SELECT col1 as MyCol1,
col2 as MyCol2,
.......
FROM `foobar`
select column1 as xyz,
column2 as pqr,
.....
from TableName;