I would like to convert like 100 tables from MySql to MongoDb
So I think that the easiest way will be to import the data as JSON
Any generic query to convert MySql data to JSON format without using PHP /ruby/Python?
SELECT CONCAT(CONCAT('{"CompanyId":', company_id),"}") AS JSON FROM company;
Try this:
SET #schema = 'test_db';
SET #table = 'test';
SELECT CONCAT(
'SELECT CONCAT(TRIM(TRAILING ', QUOTE(','), ' FROM CONCAT(', QUOTE('{'), ',',
GROUP_CONCAT(QUOTE('"'), ',', QUOTE(COLUMN_NAME), ',',
QUOTE('"'), ',', QUOTE(':'), ',', QUOTE('"'), ',', COLUMN_NAME, ',',
QUOTE('"'),',', QUOTE(',')),
')), ''}'') FROM ', #table
)
INTO #qry FROM
(SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS c
WHERE TABLE_SCHEMA = #schema AND TABLE_NAME = #table) t;
SELECT #qry;
PREPARE stmt FROM #qry;
EXECUTE stmt;
Above query generates table data in JSON format. Using the query you can prepare a stored procedure taking database and table names as input parameters and populate your data. If you want to exclude some of the columns in a table, just modify the query which selects data from INFORMATION_SCHEMA database like SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS c WHERE TABLE_SCHEMA = #schema AND TABLE_NAME = #table AND COLUMN_NAME NOT IN (**columns to exclude**).
I extended oardic's answer to create a mysql procedure to generate json from table name along with null check. Here key is set to value null instead of empty string.
create procedure createJSON (IN tabelSchema varchar(255), IN tableName varchar(255), IN whereCondition varchar(255), OUT outJson longtext)
BEGIN
DROP TEMPORARY TABLE IF EXISTS RowJson;
SELECT CONCAT(
'CREATE TEMPORARY TABLE RowJson AS SELECT CONCAT(TRIM(TRAILING ', QUOTE(','), ' FROM CONCAT(', QUOTE('{'), ',',
GROUP_CONCAT(QUOTE('"'), ',', QUOTE(COLUMN_NAME), ',',
QUOTE('"'), ',', QUOTE(':'), ',', 'IF(ISNULL(', COLUMN_NAME, '), "null", CONCAT(',QUOTE('"'),",", COLUMN_NAME, ",", QUOTE('"'),')),',
QUOTE(',')),
')), ''}'') as json FROM ', tableName, ' WHERE ', whereCondition
)
INTO #qry FROM
(SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS c
WHERE TABLE_SCHEMA = tabelSchema AND TABLE_NAME = tableName) t;
PREPARE stmt FROM #qry;
EXECUTE stmt;
SELECT json into outJson FROM RowJson;
END;
Related
How to search the entire database for column name equal to my rule and specific value as well.
Let's say that i want to search for column name like voucher where it's value contain that word value10
So far i can find the column name but i don't know how to match with value as well.
SELECT column_name FROM information_schema.columns
WHERE TABLE_SCHEMA = 'dbname' AND column_name LIKE '%voucher%'
So the end goal is to find any column name like voucher containing value10 within it's content.
Procedure code:
CREATE PROCEDURE search_tables ( IN column_pattern TEXT,
IN value_pattern TEXT )
BEGIN
SELECT GROUP_CONCAT (CONCAT( ' SELECT ''',
TABLE_NAME,
'.',
COLUMN_NAME,
''' AS `table.column`, ',
COLUMN_NAME,
' AS `value`\nFROM ',
TABLE_NAME,
'\nWHERE ',
COLUMN_NAME,
' LIKE ''',
value_pattern,
'''' )
SEPARATOR ' UNION ALL ')
INTO #query
FROM INFORMATION_SCHEMA.COLUMNS
WHERE column_name LIKE column_pattern
AND TABLE_SCHEMA = DATABASE();
PREPARE stmt FROM #query;
EXECUTE stmt;
DROP PREPARE stmt;
END
Test tables:
CREATE TABLE table1 (val1 VARCHAR(8), val2 TEXT);
INSERT INTO table1 VALUES
('a_01_a','b_11_b'),
('c_211_c','d_311_d'),
('e_55_e','f_00_f');
CREATE TABLE table2 (val3 CHAR(6), field4 VARCHAR(64));
INSERT INTO table2 VALUES
('x_1123','ghjghj_11_tyuyu'),
('8901_t','sdf_SDF_sdf');
Call:
CALL search_tables('%val%', '%11%');
Output:
table.column value
table1.val1 c_211_c
table1.val2 b_11_b
table1.val2 d_311_d
table2.val3 x_1123
fiddle
Create a stored procedure to loop through meta data table INFORMATION_SCHEMA to fetch all tables with column_name of choice. Further dynamic SQL is used to scan each of the tables retrieved for columns having the value of choice.
DDL and DML for setting the data for testing :
create table TESTA(COLMNA char(255),COLMNC char(255));
create table TESTB(COLMNA char(255),COLMNB char(255));
create table TESTC(COLMND char(255),COLMNA char(255));
insert into TESTA values('value0','someothercolmn');
insert into TESTB values('value0','anothersomeothercolmn');
insert into TESTB values('value1','Yetanothercolumn');
Test is to search all tables having column_name as COLMNA with value as value0. The procedure will accept column_name and column_Value, hence can be used across the database, just need to pass values as appropriate.
CREATE PROCEDURE Findtables( colmn_name VARCHAR(64),colmn_value VARCHAR(64) )
BEGIN
DECLARE tablename CHAR(64);
DECLARE c1 CURSOR FOR
SELECT table_name
FROM information_Schema
WHERE column_name = colmn_name;
OPEN c1;
lable_loop:LOOP
FETCH c1 INTO tablename;
select tablename;
SET #sql = CONCAT('SELECT * FROM ', tablename, ' WHERE ',colmn_name,' = "',colmn_value ,'" ;');
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP lable_loop;
CLOSE c1;
END;
Call the stored procedure :
CALL Findtables('COLMNA','value0');
Output :
tablename
TESTA
COLMNA COLMNC
value0 someothercolmn
tablename
TESTB
COLMNA COLMNB
value0 anothersomeothercolmn
tablename
TESTC
COLMND COLMNA
Demonstration of the solution can be found in DBFIDDLE link [https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=4888a6160faf97fb75665832d6610293][1]
PS : I have to create INFORMATION_SCHEMA table in dbfiddle as metadata tables are not accessible.
I need to call a mySQL function for all columns in a table.
I know how to do it for a particular column
Like this:
UPDATE `table_name` set `column_name` = function_name(`column_name`)
But i have no clue how to do it for all columns at once.
Thanks in advance.
Little clarification: I dont want to manually mention all columns, as i probably could have 200 columns table.
But i have no clue how to do it for all columns at once.
You just can't - there is no such shortcut in the update syntax.
You can do this with a single update statement, but you need to enumerate each and every column, like:
update table_name set
column_name1 = function_name(column_name1),
column_name2 = function_name(column_name2),
column_name3 = function_name(column_name3)
An alternative would be to use dynamic SQL to programatically generate the proper query string from catalog table information_schema.columns, and then execute it. This seems uterly complicated for what looks like a one-shot task... But here is sample code for that:
-- input variables
set #table_schema = 'myschema';
set #table_name = 'mytable';
set #function_name = 'myfunction';
-- in case "GROUP_CONCAT()" returns more than 1024 characters
set session group_concat_max_len = 100000;
-- build the "set" clause of the query string
select
#sql := group_concat(
'`', column_name, '` = ', #table_schema, '.', #function_name, '(`', column_name, '`)'
separator ', '
)
from information_schema.columns
where table_schema = #table_schema and table_name = #table_name;
-- entire query string
set #sql := concat('update ', #table_schema, '.', #table_name, ' set ', #sql);
-- debug
select #sql mysql;
-- execute for real
prepare stmt from #sql;
execute stmt;
deallocate prepare stmt;
I want to copy/update data from Table A to Table B. Table B has some more additional columns. I have tried the following options.
1) `REPLACE INTO `B` (SHOW FIELDS FROM 'A') SELECT * FROM `A
2) `REPLACE INTO `B`
(SELECT `COLUMN_NAME` FROM `INFORMATION_SCHEMA`.`COLUMNS`
WHERE `TABLE_SCHEMA`='test1' AND `TABLE_NAME`='A') SELECT * FROM `A
But it throws errors. Can you guys help me how to select names with select query?
UPDATE:
3) As suggested by Jerko,
I have two tables A(warehouse_id,long,lat) B(warehouse_id,long)
Applied the following statement.
SET #query = CONCAT('REPLACE INTO `A` (SELECT ',
(SELECT GROUP_CONCAT(CONCAT('`',column_name, '`'))
FROM information_schema.columns
WHERE `TABLE_SCHEMA`='test2' AND `table_name` = 'A'),
' FROM `B`)');
PREPARE stmt FROM #query;
EXECUTE stmt;
This gives me the error
"#1054 - Unknown column 'lat' in 'field list' "
You can't do this dynamically in mysql like you are trying to do. MySQL expects your list of column names to be provided directly, not from a subquery.
If you want to do this dynamically you'll have to step back upstream to whatever language you are using to interact with MySQL such as PHP or Java.
Have you tried this?
insert into B(col1, . . ., coln)
select col1, . . ., coln
from A;
That is, list the fields from A in the select clause. List the corresponding columns for B in the insert column list.
If you need the list of columns, get them from INFORMATION_SCHEMA.COLUMNS and cut-and-paste into the query.
Actually there is a way
SET #query = CONCAT('REPLACE INTO `A` (',
(SELECT GROUP_CONCAT(CONCAT('`',column_name, '`'))
FROM information_schema.columns
WHERE `TABLE_SCHEMA`='test1' AND `table_name` = 'A'
AND column_name IN (SELECT column_name FROM information_schema.columns WHERE table_schema = 'test1' AND table_name='B')) ,
') (SELECT ',
(SELECT GROUP_CONCAT(CONCAT('`',column_name, '`'))
FROM information_schema.columns
WHERE `TABLE_SCHEMA`='test1' AND `table_name` = 'A'
AND column_name IN (SELECT column_name FROM information_schema.columns WHERE table_schema = 'test1' AND table_name='B')),
' FROM `B`)');
PREPARE stmt FROM #query;
EXECUTE stmt;
Try this
INSERT INTO B (field1,field2,...,fieldN)
SELECT (field1,field2,...,fieldN) FROM A
I would like to have a list of those columns of a table that have at least one non-NULL data entries in them.
In other words, I would like to get the column names for which the following returns at least one entry:
SELECT DISTINCT column_name FROM table WHERE column_name IS NOT NULL
I tried the following:
SELECT column_name
FROM information_schema.columns
WHERE table_name = "table_name"
AND EXISTS (
SELECT DISTINCT column_name FROM table_name WHERE column_name IS NOT NULL
)
But this also returns the column names where all the entries are NULL.
So how do I get only those columns with non-NULL entries?
Create from the INFORMATION_SCHEMA.COLUMNS table a string that contains the SQL you wish to execute, then prepare a statement from that string and execute it.
The SQL we wish to build will look like:
SELECT 'column_a'
FROM table_name
WHERE `column_a` IS NOT NULL
HAVING COUNT(*)
UNION ALL
SELECT 'column_b'
FROM table_name
WHERE `column_b` IS NOT NULL
HAVING COUNT(*)
-- etc.
(One could omit the WHERE clause and substitute COUNT(*) for COUNT(column), but I think that might be less efficient on indexed columns).
This can be done using the following:
SET group_concat_max_len = 4294967295;
SELECT GROUP_CONCAT(
' SELECT ',QUOTE(COLUMN_NAME),
' FROM table_name',
' WHERE `',REPLACE(COLUMN_NAME, '`', '``'),'` IS NOT NULL',
' HAVING COUNT(*)'
SEPARATOR ' UNION ALL ')
INTO #sql
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'table_name';
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
See it on sqlfiddle.
Use this procedure this will print columns names of a table which have atleast one not null rows.
create or replace procedure list_col_notNull(tblName in varchar2)
as
lv_col_name varchar2(200);
lv_ctr number;
lv_sql varchar2(400);
CURSOR cur_col_name is
SELECT column_name
FROM USER_TAB_COLUMNS U
WHERE table_name = tblName order by column_name asc;
begin
open cur_col_name;
LOOP
FETCH cur_col_name INTO lv_col_name;
EXIT WHEN cur_col_name%NOTFOUND;
lv_sql := 'select count(1) From ' || tblName || ' where ' || lv_col_name || ' is not null' ;
EXECUTE IMMEDIATE lv_sql into lv_ctr;
if lv_ctr > 0
then
dbms_output.put_line(lv_col_name);
end if;
I'm using the Kohana framework and I need to convert column names to lowercase. I don't have control of the db table structure. I want to do the following:
SELECT LOWER(*) FROM .....
but MYSQL does not like that. Whats the proper way of outputting the lower case column names if I don't know what the column names will be?
Found here http://dev.mysql.com/doc/refman/5.0/en/columns-table.html
SELECT LOWER(COLUMN_NAME)
FROM INFORMATION_SCHEMA.COLUMNS
WHERE table_name = 'Table'
Below you can see both MSSQL and MySQL syntax for creating a dynamic query using the column results from the query above.
MSSQL Syntax
DECLARE #ColumnNames [nvarchar](1024)
SELECT #ColumnNames = COALESCE(#ColumnNames + ', ', '') + LOWER(COLUMN_NAME)
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'Table'
DECLARE #Sql [nvarchar](1024) = 'SELECT ' + #ColumnNames + ' FROM Table ' --Remember to put spaces after SELECT and before FROM
EXEC(#Sql)
With this you are dynamically building your query and then executing it.
MySQL Syntax
SELECT #ColumnNames := GROUP_CONCAT(LOWER(COLUMN_NAME))
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'Table';
SET #Sql = CONCAT('SELECT ', #ColumnNames, ' FROM Table ');
PREPARE stmt1 FROM #Sql;
EXECUTE stmt1;