MySQL Dynamic Table issue - mysql

Every month we get a CSV file that contains 300+ columns. The layout of the file can change whenever, and columns can be added/removed. I'm in need of a way to create a dynamic table that contains the exact number of columns needed. I don't care about column names, they can be Column1, Column2, etc.. and I plan to set all types to Varchar(500).
Ideally what I would I want to accomplish is a stored procedure that I can simply pass in the number of columns needed and it will loop to create the necessary Table Definition sql and then execute that sql.
Is accomplishing this even possible? I had started to write the following:
BEGIN
Declare loopvar int default 1;
Declare tsql VarChar(5000);
Declare table_definition VarChar(8000);
Declare tablename varchar(20);
Set table_definition = 'Column';
set tablename = 'npi_data';
Set loopvar = 1;
While loopVar < 362 DO
set tsql = table_definition + loopvar + 'varchar(500) DEFAULT NULL' ;
set loopvar = loopvar + 1;
end while;
set tsql = 'CREATE TABLE' + tablename + '(' + tsql + ') ENGINE=InnoDB DEFAULT CHARSET=utf8';
execute tsql;
END;
I'd like something close to that but all I really need is the ability to create a table with any given number of columns.
Thanks!

If you mean you want to be able to pass a list of column names, like 'A,B,C', and use it to generate a list of column definitions, like A varchar(500) DEFAULT NULL,B varchar(500) DEFAULT NULL,C varchar(500) DEFAULT NULL and, eventually, a CREATE TABLE statement with that definition list, you could use an approach like this:
...
SET columnnames = 'A,B,C';
SET sql = CONCAT(
REPLACE(columnnames, ',', ' varchar(500) DEFAULT NULL,'),
' varchar(500) DEFAULT NULL'
);
SET sql = CONCAT('CREATE TABLE ', tablename, ' (', sql, ')');
...
That is, every comma in the name list is expanded so as to complete the definition of the column preceding the comma. As the last item isn't followed by a comma, the type is simply concatenated one more time to the result of REPLACE.
Alternatively, to avoid the repetition of the varchar(500) DEFAULT NULL bit, you could do this:
...
SET columnnames = 'A,B,C';
SET sql = REPLACE(CONCAT(columnnames, ','), ',', ' varchar(500) DEFAULT NULL,');
SET sql = LEFT(sql, LENGTH(sql) - 1);
SET sql = CONCAT('CREATE TABLE ', tablename, ' (', sql, ')');
...
In this case, a comma is added after the last name, then all commas in the resulting string are replaced like in the previous examples. As a final touch to the list, the trailing comma is removed and finally the complete statement is built.
Either way, no loop is needed.

Related

want to Write a Stored Procedure in MYSQL [duplicate]

I have made a stored procedure. I want it to filter the data by different parameters. If I pass one parameter, it should be filtered by one; if I pass two, it should be filtered by two, and so on, but it is not working.
Can anyone help me please?
DROP PROCEDURE IF EXISTS medatabase.SP_rptProvince2;
CREATE PROCEDURE medatabase.`SP_rptProvince2`(
IN e_Region VARCHAR(45)
)
BEGIN
DECLARE strQuery VARCHAR(1024);
DECLARE stmtp VARCHAR(1024);
SET #strQuery = CONCAT('SELECT * FROM alldata where 1=1');
IF e_region IS NOT NULL THEN
SET #strQuery = CONCAT(#strQuery, ' AND (regionName)'=e_Region);
END IF;
PREPARE stmtp FROM #strQuery;
EXECUTE stmtp;
END;
AFAIK, you can't have a variable argument list like that. You can do one of a couple of things:
Take a fixed maximum number of parameters, and check them for null-ness before concatenating:
CREATE PROCEDURE SP_rptProvince2(a1 VARCHAR(45), a2 VARCHAR(45), ...)
...
IF a1 IS NOT NULL THEN
SET #strQuery = CONCAT(#strQuery, ' AND ', a2);
END IF;
If you need predetermined fields to which the criteria in the argument apply (like the e_Region parameter in your existing code), then you modify the CONCAT operation appropriately.
Possible invocation:
CALL SP_rptProvince2('''North''', 'column3 = ''South''')
Take a single parameter that is much bigger than just 45 characters, and simply append it to the query (assuming it is not null).
Clearly, this places the onus on the user to provide the correct SQL code.
Possible invocation:
CALL SP_rptProvince2('RegionName = ''North'' AND column3 = ''South''')
There's not a lot to choose between the two. Either can be made to work; neither is entirely satisfactory.
You might note that there was a need to protect the strings in the arguments with extra quotes; that is the sort of thing that makes this problematic.
I found a JSON-based approach which works with the latest MySQL/MariaDB systems. Check the link below (Original Author is Federico Razzoli): https://federico-razzoli.com/variable-number-of-parameters-and-optional-parameters-in-mysql-mariadb-procedures
Basically, you take a BLOB parameter which is actually a JSON object and then do JSON_UNQUOTE(JSON_EXTRACT(json object, key)) as appropriate.
Lifted an extract here:
CREATE FUNCTION table_exists(params BLOB)
RETURNS BOOL
NOT DETERMINISTIC
READS SQL DATA
COMMENT '
Return whether a table exists.
Parameters must be passed in a JSON document:
* schema (optional). : Schema that could contain the table.
By default, the schema containing this procedure.
* table : Name of the table to check.
'
BEGIN
DECLARE v_table VARCHAR(64)
DEFAULT JSON_UNQUOTE(JSON_EXTRACT(params, '$.table'));
DECLARE v_schema VARCHAR(64)
DEFAULT JSON_UNQUOTE(JSON_EXTRACT(params, '$.schema'));
IF v_schema IS NULL THEN
RETURN EXISTS (
SELECT TABLE_NAME
FROM information_schema.TABLES
WHERE
TABLE_SCHEMA = SCHEMA()
AND TABLE_NAME = v_table
);
ELSE
RETURN EXISTS (
SELECT TABLE_NAME
FROM information_schema.TABLES
WHERE
TABLE_SCHEMA = v_schema
AND TABLE_NAME = v_table
);
END IF;
END;

How to not show certain columns in MySQL? [duplicate]

when I do:
SELECT *
FROM SOMETABLE
I get all the columns from SOMETABLE, but I DON'T want the columns which are NULL (for all records). How do I do this?
Reason: this table has 20 columns, 10 of these are set but 10 of them are null for certain queries. And it is time consuming to type the columnnames....
Thanks,
Voodoo
SQL supports the * wildcard which means all columns. There is no wildcard for all columns except the ones you don't want.
Type out the column names. It can't be more work than asking questions on Stack Overflow. Also, copy & paste is your friend.
Another suggestion is to define a view that selects the columns you want, and then subsequently you can select * from the view any time you want.
It's possible to do, but kind of complicated. You can retrieve the list of columns in a table from INFORMATION_SCHEMA.COLUMNS. For each column, you can run a query to see if any non-null row exists. Finally, you can run a query based on the resulting column list.
Here's one way to do that, with a cursor:
declare #table_name varchar(256)
set #table_name = 'Airports'
declare #rc int
declare #query nvarchar(max)
declare #column_list varchar(256)
declare columns cursor local for select column_name
from INFORMATION_SCHEMA.COLUMNS where TABLE_NAME = #table_name
open columns
declare #column_name varchar(256)
fetch next from columns into #column_name
while ##FETCH_STATUS = 0
begin
set #query = 'select #rc = count(*) from ' + #table_name + ' where ' +
#column_name + ' is not null'
exec sp_executesql #query = #query, #params = N'#rc int output',
#rc = #rc output
if #rc > 0
set #column_list = case when #column_list is null then '' else
#column_list + ', ' end + #column_name
fetch next from columns into #column_name
end
close columns
deallocate columns
set #query = 'select ' + #column_list + ' from ' + #table_name
exec sp_executesql #query = #query
This runs on SQL Server. It might be close enough for Sybase. Hopefully, this demonstrates that typing out a column list isn't that bad :-)

How to check for a character in a string and replace that character before insert

Ok, this question involves one part of a complicated stored procedure which inserts new entities into several tables.
The part that I'm currently having difficulty with needs to work like so:
insert entity with original name
check if name of new entity contains any special characters listed in table A 'Characters'
if yes, than replace that character with a 'replacement character' from table A
EDIT: I've gotten this to partially work but still not finished. I'm still having a problem showing each combination of character replacements. Also in the case of a replacement character occurring more than once, such as the '.', the substitutions needs to happen independently of one another.
ex: #www.test&aol.com -> #wwwtest&aol.com, #www.test&aolcom
Here's a rough start, I know parts of this aren't going to work, but I thought it was a decent starting point:
declare #test varchar(50)
set #test = '#www.test&aol.com'
declare #len int, #ctr int
set #len = LEN(#test)
set #ctr = 1
declare #newName varchar(50)
declare #matchedChar table(match varchar(10),replaceChar varchar(10),processed int default(0))
declare #alternateEntities table(name varchar(50))
declare #repChar varchar(10)
declare #selectedChar varchar(1)
while #ctr<=#len
begin
--Insert matching characters and replacement characters into table variable,
--this is necessary for the # character, which has multiple replacement characters
insert into #matchedChar (match,replaceChar) select Character,ReplacementCharacter from tblTransliterations where Character = SUBSTRING(#test,#ctr,1)
--loop
while (select COUNT(*) from #matchedChar where processed = 0)>0
begin
--get the top character from table variable
set #selectedChar = (select top 1 match from #matchedChar where processed = 0)
--get replacement character
set #repChar = (select top 1 replaceChar from #matchedChar where processed = 0)
--replace character in name string
--set #newName = (select Replace(#test,#selectedChar,#repChar))
set #newName = (select STUFF(#test,CHARINDEX(#selectedChar,#test),1,#repChar))
--update table variable to move onto next character
update #matchedChar set processed = 1 where #repChar = replaceChar
--add name with replaced character to alternate entities table
insert into #alternateEntities (name) values (#newName)
end
set #ctr = #ctr+1
set #len = LEN(#test)
end
select * from #alternateEntities
Instead of looping, use set-based approach.
Create a temp table and populate column 'Words' of type NVARCHAR(100), call the temp table Invalid_Words
Create a column on Invalid_Words for each token and make the col type = bit
Update the temp table bit columns if a word contains the token through a series of update statements
You now have defined which tokens were matched for each word.
The next part is to replace.

What's wrong with the stored procedure

I have a table called Std_Components which acts like an index for list of components with associated tables. The column AssociatedTable holds the name of table that actually contains the component data.
Please check images below -
Here is table data for Std_SteeringPumps
I am trying to create a stored procedure that will copy Std_Components table as well as all associated tables with new name. For ex. Lets say if i provided 001 as a parameter to this stored procedure i should be able create new tables like C001_Components, C001_SteeringPumps and so on.
This is what I have done so far:
ALTER PROCEDURE [dbo].[sgi_sp_CreateTablesForNewCompany]
-- Add the parameters for the stored procedure here
#CompanyId varchar(5)
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- declare variables
declare #qry as varchar(2000)
declare #compTblName as varchar(100)
set #compTblName = 'C'+#companyId +'_Components'
-- Check if table already exists
IF object_id(#compTblName) is not null
return
-- Create main component index table by copying standard component table --
set #qry = 'Select * into '+#compTblName+' From Std_Components;';
--print #qry
--execute (#qry)
set #qry =#qry + 'Update C'+#companyId +'_Components Set AssociatedTable=''C'+#companyId +'''+substring(AssociatedTable,4,200);';
--print #qry
--exec #qry
-- Create all child tables --
Select * Into #TempTbl From dbo.Std_Components
Declare #Id int
While (Select Count(*) From #TempTbl) > 0
Begin
declare #rowTableName as varchar(50)
declare #compNewTbl as varchar(50)
Select Top 1 #rowTableName=AssociatedTable, #Id = Id From #TempTbl
set #compNewTbl = 'C'+#companyId + substring(#rowTableName,4,200);
set #qry = #qry + 'Select * into '+#compNewTbl+' From ' + #rowTableName + ';'
--print #qry
--exec #qry
Delete #TempTbl Where Id = #Id
End
print #qry
exec #qry
END
Here is the output of the print statement for the query it generates -
Select * into C001_Components From Std_Components;
Update C001_Components Set AssociatedTable='C001'+substring(AssociatedTable,4,200);
Select * into C001_SteeringPumps From Std_SteeringPumps;
But when the stored procedure is executed, I get the following error -
Msg 203, Level 16, State 2, Procedure sgi_sp_CreateTablesForNewCompany, Line 56
The name 'Select * into C001_Components From Std_Components;Update C001_Components Set AssociatedTable='C001'+substring(AssociatedTable,4,200);Select * into C001_SteeringPumps From Std_SteeringPumps;' is not a valid identifier.
Can anybody help me out resolve this issue.
Thanks for sharing your time and wisdom.
The error you're getting is because the EXEC statement (the last line of the stored procedure) needs to have brackets around the #qry variable so that it becomes
exec(#qry)
Without the brackets it's treating the entire SQL string as stored procedure name.
The non valid indentifier is around the AssociatedTable part
Set AssociatedTable='C001'+substring(AssociatedTable,4,200); will not run as there is no scope for AssociatedTable to substring - the string needs to contain the name of the table completely to be able to be executed
Instead of
exec #qry;
You need
exec sp_executesql #qry;
You'll also need to change the type of #qry to NVARCHAR. Note that because of the dynamic sql, the proc is prone to SQL Injection and other escaping issues (i.e. ensure that #CompanyId is validated)

Creating a dynamic where clause in SQL Server stored procedure

I am trying to create a stored that will accept two values, a column name and a value. It will then check if there is a record that exists for the passed in column name with the passed in value. I've tried the following
CREATE PROCEDURE p_HasActiveReference
#pi_colName varchar(100)
,#pi_colValue varchar(100)
AS
BEGIN
SET NOCOUNT ON
declare #sql varchar(1000)
set #sql = 'IF EXISTS(SELECT TOP 1 p.PaymentId
FROM Payments p
WHERE ' + #pi_colName + ' = ' + #pi_colValue + 'AND Active = 1)
SELECT ''True'' AS RETVAL
ELSE
SELECT ''False'' AS RETVAL'
exec(#sql)
END
However, I always get this error
Conversion failed when converting the varchar value 'InternalOr' to data type int.
When I call the procedure with the following
p_HasActiveReference 'InternalOrgCode', '10110'
The internalOrgCode column is of value varchar(10)
I am not a SQL expert, so I am not even sure if what I need to achieve is even possible using that technique!
Thanks!
At least one issue: you should be surrounding your string value with single quotes, and to escape those inside a string you need to double them up:
WHERE ' + #pi_colName + ' = ''' + #pi_colValue + ''' AND ...
You also may want to declare your #sql variable as something bigger than 100 characters! Looks like your string is getting truncated.
If the possible values for #pi_colName are finite, the data type is always string, and the columns are collation compatible, you could do something like this and avoid dynamic SQL:
SELECT ...
WHERE CASE #pi_colName
WHEN 'col1' THEN col1
WHEN 'col2' THEN col2
END = #pi_ColValue;