I want to count how many columns in a row are not NULL.
The table is quite big (more than 100 columns), therefore I would like to not do it manually or using php (since I dont use php) using this approach Counting how many MySQL fields in a row are filled (or empty).
Is there a simple query I can use in a select like SELECT COUNT(NOT ISNULL(*)) FROM big_table;
Thanks in advance...
Agree with comments above:
There is something wrong in the data since there is a need for such analysis.
You can't completely make it automatic.
But I have a recipe for you for simplifying the process. There are only 2 steps needed to achieve your aim.
Step 0. In the step1 you'll need to get the name of your table schema. Normally, the devs know in what schema does the table reside, but still... Here is how you can find it
select *
from information_schema.tables
where table_name = 'test_table';
Step 1. First of all you need to get the list of columns. Getting just the list of cols won't help you out at all, but this list is all we need to be able to create SELECT statement, right? So, let's make database to prepare select statement for us
select concat('select (length(concat(',
group_concat(concat('ifnull(', column_name, ', ''###'')') separator ','),
')) - length(replace(concat(',
group_concat(concat('ifnull(', column_name, ', ''###'')') separator ','),
'), ''###'', ''''))) / length(''###'')
from test_table')
from information_schema.columns
where table_schema = 'test'
and table_name = 'test_table'
order by table_name,ordinal_position;
Step 3. Execute statement you've got on step 2.
select (length(concat(.. list of cols ..)) -
length(replace(concat(.. list of cols .. ), '###', ''))) / length('###')
from test_table
The select looks tricky but it's simple: first replace all nulls with some symbols that you're sure you'll never get in those columns. I usually do that replacing nulls with "###". that what all that "ifnull"s are here for.
Next, count symbols with "length". In my case it was 14
After that, replace all "###" with blanks and count length again. It's 11 now. For that I was using "length(replace" functions together
Last, just divide (14 - 11) by a length of a replacement string ("###" - 3). You'll get 1. This is exactly amount of nulls in my test string.
Here's a test case you can play with
Do not hesitate to ask if needed
Related
I am new to SQL and I realize I have a lot to learn. I am in the midst of converting a set of ISAM files into MySQL tables. Some of the tables I am creating have hundreds of columns. To make it easier to see if my conversions are working properly, I would like to display only a subset of columns at a time in my SELECT results.
For example, I would like to see only Dates or only Amounts (i.e., Decimals). This would make it easier to see if correct values are in those fields.
I am working from the MySQL command line and I would rather not specify a long list of column names if it can be avoided. Is there a way to get my Select statements to display only the types of columns I am interested in?
(Added 5/12/16)
I used a variant of Uueerdo's syntax to create a view:
create view dates_view as concat('select ', group_concat(column_name), ' ',' from ', table_schema, '.', table_name, ';') from information_schema.columns where table_name = 'binders_tbl' and data_type in ('date');
When I enter select * from dates_view; I get this value:
select binder_effectiveDate,binder_expirationDate,binder_submissionCreatedDate,binder_firstQuotedDate,binder_boundDate,binder_invoicedDate,binder_terminatedDate,binder_cancelledDate from aesdb.binders_tbl;
I tried but could not find a way to use this view with "select * from binders_tbl;" to display only the above date fields and their values.
If anyone can guide me to a "join" or "subquery" solution, I would appreciate it. For now, I am going to stick with listing all the individual columns as needed. Plus, I will study more basic and intermediate SQL. Thank you to those who responded.
(Added 5/20/16)
I found a sort-of work-around to my problem. Here it is in case it helps someone else:
CREATE VIEW DateCols_View AS SELECT Table1_SelectionCriteriaField, Table1_Date1, Table1_Date2, Table1_Date3 FROM Table1;
SELECT * FROM DateCols_View WHERE Table1_SelectionCriteriaField (matches some criteria);
My work-around has 2 drawbacks. The first is that I have to include any selection criteria fields that I plan to use later. But with a little forethought I can create multiple views with different sets of columns for different displays. This will let me display only those columns I wish to see while working with a set of records. I plan to create separate views for Dates, Amounts, Flags, etc.
The other drawback is that I have to create a separate set of views for each table or join combination I plan to work with.
I hope as I learn more about SQL, I will find a way to generalize this technique (to make it more like a script or macro) and simplify it further.
Nope, you have to specify column names. There is no syntax for dynamic field inclusion, however you could maybe work something out with information_schema (like below) to generate a query for you to use.
SELECT CONCAT('SELECT ', GROUP_CONCAT(column_name), ' '
,'FROM `', table_schema, '`.`', table_name, '`;'
FROM information_schema.columns
WHERE table_schema = 'yourschemaname'
AND table_name = 'yourtablename'
AND data_type IN ('timestamp', 'date'[, etc....])
;
I have a field called 'areasCovered' in a MySQL database, which contains a string list of postcodes.
There are 2 rows that have similar data e.g:
Row 1: 'B*,PO*,WA*'
Row 2: 'BB*, SO*, DE*'
Note - The strings are not in any particular order and could change depending on the user
Now, if I was to use a query like:
SELECT * FROM technicians WHERE areasCovered LIKE '%B*%'
I'd like it to return JUST Row 1. However, it's returning Row 2 aswell, because of the BB* in the string.
How could I prevent it from doing this?
The key to using like in this case is to include delimiters, so you can look for delimited values:
SELECT *
FROM technicians
WHERE concat(', ', areasCovered, ', ') LIKE '%, B*, %'
In MySQL, you can also use find_in_set(), but the space can cause you problems so you need to get rid of it:
SELECT *
FROM technicians
WHERE find_in_set('B', replace(areasCovered, ', ', ',') > 0
Finally, though, you should not be storing these types of lists as strings. You should be storing them in a separate table, a junction table, with one row per technician and per area covered. That makes these types of queries easier to express and they have better performance.
You are searching wild cards at the start as well as end.
You need only at end.
SELECT * FROM technicians WHERE areasCovered LIKE 'B*%'
Reference:
Normally I hate REGEXP. But ho hum:
SELECT * FROM technicians
WHERE concat(",",replace(areasCovered,", ",",")) regexp ',B{1}\\*';
To explain a bit:
Get rid of the pesky space:
select replace("B*,PO*,WA*",", ",",");
Bolt a comma on the front
select concat(",",replace("B*,PO*,WA*",", ",","));
Use a REGEX to match "comma B once followed by an asterix":
select concat(",",replace("B*,PO*,WA*",", ",",")) regexp ',B{1}\\*';
I could not check it on my machine, but it's should work:
SELECT * FROM technicians WHERE areasCovered <> replace(areaCovered,',B*','whatever')
In case the 'B*' does not exist, the areasCovered will be equal to replace(areaCovered,',B*','whatever'), and it will reject that row.
In case the 'B*' exists, the areCovered will NOT be eqaul to replace(areaCovered,',B*','whatever'), and it will accept that row.
You can Do it the way Programming Student suggested
SELECT * FROM technicians WHERE areasCovered LIKE 'B*%'
Or you can also use limit on query
SELECT * FROM technicians WHERE areasCovered LIKE '%B*%' LIMIT 1
%B*% contains % on each side which makes it to return all the rows where value contains B* at any position of the text however your requirement is to find all the rows which contains values starting with B* so following query should do the work.
SELECT * FROM technicians WHERE areasCovered LIKE 'B*%'
All my column headers in a MySQL database are prefixed with a number, 1_X, 2_X, etc... which makes bringing the data into IDL impossible using just a basic select statement to bring in the entire table. I'm not sure but I see two possible ways:
1) Bring in the table with column name aliases. Can I use TRIM or SUBSTRING_INDEX to remove/replace the first two characters?
2) Create a routine that uses the information schema to to recursively go through and delete the first two characters of the column headers and create a new table with those headers and copy the data in.
If there weren't so many different tables (all with 1_X, 2_X, etc...) there'd be no problem manually selecting 1_X AS X but that's not feasible. It would be great to be able to use TRIM/SUBSTRING on column headers in the select statement.
Thanks.
It's not possible to use functions in a SQL statement to alter the identifier assigned to a column being returned. The SQL way of specifying the identifier for the column in a resultset is to use the expr AS alias approach.
Rather than trim off the leading digit characters, you could prepend the identifiers with another valid character. (Trimming off leading characters seems like it would potentially lead to another problem, duplicate and/or zero length column names.)
You could just use a SQL statement to generate the SELECT list for you.
(NOTE: the GROUP_CONCAT function is limited by some system/session variables: group_concat_max_len and max_allowed_packet, it's easy enough to adjust these higher, though changing global max_allowed_packet may require MySQL to be restarted.)
To get it back the SELECT list on all one line (assuming you won't overflow the GROUP_CONCAT limits) something like:
SELECT c.table_schema
, c.table_name
, GROUP_CONCAT(
CONCAT('t.`',c.column_name,'` AS `x',c.column_name,'`')
ORDER BY c.ordinal_position
) AS select_list_expr
FROM information_schema.columns c
FROM information_schema.columns c
WHERE c.table_schema = 'mydatabase'
GROUP BY c.table_schema, c.table_name
Or, you could even get back a whole SELECT statement, if you wrapped that GROUP_CONCAT expression (which produces the select list) in another CONCAT
Something like this:
SELECT CONCAT('SELECT '
, GROUP_CONCAT(
<select_list_expr>
)
, ' FROM `',c.table_schema,'`.`',c.table_name,'` t;'
) AS stmt
FROM information_schema.columns c
WHERE c.table_schema = 'mydatabase'
GROUP BY c.table_schema, c.table_name
You could use a more clever expression for <select_list_expr>, to check for leading "digit" characters, and assign an alias to just those columns that need it, and leave the other columns unchanged, though that again introduces the potential for returning duplicate column names.
That is, if you already have columns named '1_X' and 'x1_X' in the same table. But a carefully chosen leading character may avoid that problem...
The <select_list_expr> could be more clever by doing a conditional test for leading digit character, something like this:
SELECT CONCAT('SELECT '
, GROUP_CONCAT(
CASE
WHEN c.column_name REGEXP '^[[:digit:]]'
THEN CONCAT('t.`',c.column_name,'` AS `x',c.column_name,'`')
ELSE CONCAT('t.`',c.column_name,'`')
END
)
, ' FROM `',c.table_schema,'`.`',c.table_name,'` t;'
) AS stmt
FROM information_schema.columns c
WHERE c.table_schema = 'mydatabase'
GROUP BY c.table_schema, c.table_name
Again, there's a potential for generation "duplicate" column names with this approach. The conditional test "c.column_name REGEXP" could be extended to check for other "invalid" leading characters as well.
As a side note, at some point, someone thought it a "good idea" to name columns with leading digit characters. Just because something is allowed doesn't mean it's a good idea.
Then again, maybe all that rigamarole isn't necessary, and just wrapping the column names in backticks would be sufficient for your application.
I think you can follow option 2. However this will not be quick solution.
Another way around this could be,
Generate schema script for the tables you want to correct.
Open the script in notepad++ or any editor that supports find using regular expression.
Search and replace with [0-9]+_ expression and empty string for replacement.
Create the new tables using this script and copy data into them.
This may sound like a manual approach but you will do this once for all of your tables.
Look into a strategy of doing 2 selects, one for the column name, then one for the data with column alias. You might have to revert to some scripting language, like PHP, for help.
First, get the column names :
show columns from tbl_client;
+-------------------------------+-----------------------------------+------+-----+---------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------------------+-----------------------------------+------+-----+---------------------+-----------------------------+
| 1_X | int(11) | NO | PRI | NULL | auto_increment |
Then, loop through the results and create a list of column alias
Then create your new select
SELECT 1_X as NEW_COLUMN_NAME_FOR_FIELD_1 FROM tbl_client;
I need to fetch data from database for backup in the form of insert statements
I need to do it on a button click in c#. So i think an sql query or stored procedure will be appropriate to do this, rather than mysqldump.
Secondly I need them for all tables. Instead of writing table and column names. They should be fetched from information_schema, because the query will not need to be changed for different scema
If there already exists a solution, please guide me.
Update : I have prepared a solution, it is posted, still looking for the better one.
To get Data of whole database - SqlFiddle Demo
To get data of only one table - - SqlFiddle Demo
I have made a complex but acceptable solution. But needs improvement.
This is a complex procedure with complex coding especially the query which fetches all rows of all columns into a single result by group_concat and formats with a complex concatenation.
Need it simplified, efficient and working in all scenarios.
Some details of my solution : Following is the important part, other is just conditions/Looping (I am not handy with documentation also it needs time and suggestions, someone might help me in its formatting and improvement, Sorry for any inconvenience, however I will be glad for any help from you and me)
Note: group_concat(yourColumn separator ' --anySeparator-- ') is merging all rows of your column as one such that Rows are separated by --anySeparator--
select group_concat(column_name separator '`,`') into #cns1 from
information_schema.columns where table_schema=dn and table_name=#tn;
1 : column_names are got as a single value separated by
`,` => #cs1 = id`,`ename`,`did
select group_concat(column_name separator '`,"\',\'",`') into #cns2
from information_schema.columns where table_schema=dn and table_name=#tn;
2 : column_names are got as a single value separated by
`','` => #cn2 = id`','`ename`','`did
set #cns1=concat("`",#cns1,"`"); set #cns2=concat("`",#cns2,"`");
3: Missing letter (`) is put at beginning and end of Column names
set #res=concat(#res," insert into ",#tn,"(",#cns1,") values ('");
4: Simply makes res= " insert into emp(`id` , `ename` ,`did` ) values(" Here you can see why have I put separators (MySql Formatting is achieved)
set #temp := '';
set #q := concat("select group_concat(concat(",#cns2,") separator \"'),('\")
from ",dn,".",#tn, " into #temp");
Above is the most crucial statement It gets all data rows from table as rows of a single column and further these rows are merged being separated by '),('
5.1 concat(",#cns2,") gets values of all columns in a single one.
5.2 After outer most concat now #q is
#q = "select group_concat(`id`','`ename`','`,did` separator '),(' from
mydb.emp into #temp";
5.3 : group_concat will merge all rows of that combined column into one value.
Columns values will be joined through separators existing in #cns2 and rows level joining will be with '),('
prepare s1 from #q;
execute s1;deallocate prepare s1;
set #res = concat(#res,#temp,");");
#q is executed
set #res = concat(#res,#temp,");");
6 : And We will get result as
res was = insert into emp(`id`,`ename`,`did`) values ('
#temp = 1','e1','4'),('2','e2','4'),
('3','e3','2'),('4','e4','4'),('5','e5','3
And after #res = concat(#res,#temp,");"); we get
insert into emp(`id`,`ename`,`did`) values ('1','e1','4'),('2','e2','4'),
('3','e3','2'),('4','e4','4'),('5','e5','3);
select concat("insert into users (id,name,password) values ('",id,"'"),
concat(",'",username,"'"),
concat(",'",password,"'),") INTO OUTFILE 'c:\\datas\\asd.txt' from users
There are some similar questions around but they aren't quite what I'm looking for, so forgive me if you think this is answered elsewhere.
I am basically looking for an easy way to do things as I have over 4000 tables to get data from. This kind of follows on from my previous post: mysql search for segment of table name
The general situation is that I have a database filled with tables and I only want about a quarter of this which comes to around 4000 tables. I have a list of the individual table names thanks to my previous post, but I want the data that goes with them.
I know that for an individual one I can do SELECT table1.*, table2.*; or something similar but I don't want to go through all 4000 or so.
They all end with the same thing, e.g. staff_name, manager_name, customer_name so I can use
SHOW TABLES LIKE '%_name'
to see the table names that I want in the database. Someone suggested using dynamic mysql, but I don't even know where to start with that. Any suggestions?
Generic example (in PHP):
Constructing dynamic SQL or building your SQL queries with the aid of a programming language would look like this (in PHP for ex.):
$pdos = $pdo->query("SHOW TABLES LIKE '%_name'");
$tables = $pdos->fetchAll();
$query = 'SELECT * FROM '.implode(' UNION SELECT * FROM ');
$pdo->query($query);
The fetchAll method will return an array containing the names of each table selected.
The implode($glue, $array) function takes an array and concatenates every value in the array using the $glue parameter - usually you take an array of values and implode them using $glue = ',' to create a coma separated list of values.
In our case the implode has a partial query as $glue in order to create one big UNION JOIN query.
Once the final $query is build it should look something like:
SELECT * FROM table_1_name
UNION
SELECT * FROM table_2_name
UNION
SELECT * FROM table_3_name
....
....
UNION
SELECT * FROM table_4000_name
The result should contain all of the DISTINCT rows from all 4000 tables.
Specific example (in SQL-only format):
SELECT GROUP_CONCAT(
CONCAT('select * from ', table_name)
SEPARATOR ' union '
)
INTO #my_variable
FROM information_schema.tables
WHERE table_schema = 'dbname'
AND table_name LIKE '%_name';
PREPARE my_statement FROM #my_variable;
EXECUTE my_statement;
The first statement will get all of the table names from the information_schema database;
The CONCAT function prefixes every table name with a a 'SELECT * FROM ' string;
The GROUP_CONCAT does the job that implode would have done in PHP;
The INTO clause makes sure the values are saved inside a variable named my_variable;
The PREPARE statement takes a string value (such as the one you saved in my_variable) and checks if the value is an SQL query;
The EXECUTE statement takes a "prepared statement" and well... executes it.
#my_variable is a temporary variable but it can only be of a scalar type (varchar, int, date, datetime, binary, float, double etc.) it is not an array.
The GROUP_CONCAT function is an "aggregate function" which means it takes an aggregate value (similar concept to an array - in our case the result set of our query) and outputs a simple string result.
I would suggest generating the SQL statement.
Try doing:
select concat('select * from ', table_name) as query
from Information_Schema.tables
where table_schema = <dbname> and
table_name like <whatever>
You can then run this as a bunch of queries by copying into a query editor window.
If you want everything as one query, then do:
select concat('select * from ', table_name, ' union all ') as query
from Information_Schema.tables
where table_schema = <dbname> and
table_name like <whatever>
And remove the final "union all".
This has the table name matching a like. Leave out the table_name part of the WHERE to get all tables. Or, include specific tables using table_name in ().