I have quite a few many-to-many relationships. To simplify the process of inserting data into the respective three tables, I have the below function that I adapt for the various M:M relationships, and it works like a charm. However, for situations when dealing with many new records, I would like to simplify the insert process even further.
At the moment I am using an .xls sheet with columns (and their order of sequence) corresponding to how they are written in the function (ex. surname, fname, email, phone, docutype, year, title, citation, digital, url, call_number, report_no, docu_description)
I then import that .xls incl. data to a new table in the database, and using Navicat's 'Copy as insert statement', and further copy & replacing the function-call to the statement, I end up with function call statements for all records in the table looking similar to this:
SELECT junction_insert_into_author_reportav ('Smith', 'Victoria',
some#email.com, NULL, 'Report', '2010', ' Geographical Place Names',
'Some citation, 'f', 'NULL', 'REP/63', NULL, NULL);
This works okay but I would like to reduce the steps involved even further if possible. For example by being able to pass the newly created table that I imported the .xls sheet into, as a parameter to the function -and then deleting the new table again after the insert statements in the function has run. I am just unsure how to do this, and if at all it is possible?
Here is an example of the function as it looks and works at the moment:
CREATE OR REPLACE FUNCTION junction_insert_into_author_reportav (
p_surname VARCHAR,
p_fname VARCHAR,
p_email VARCHAR,
p_phone TEXT,
p_docutype VARCHAR,
p_year int4,
p_title VARCHAR,
p_citation VARCHAR,
p_digital bool,
p_url TEXT,
p_call_no VARCHAR,
p_report_no VARCHAR,
p_docu_description VARCHAR
) RETURNS void AS $BODY$
DECLARE
v_authorId INT;
v_reportavId INT;
BEGIN
SELECT
author_id INTO v_authorId
FROM
author
WHERE
surname = p_surname
AND fname = p_fname;
SELECT
reportav_id INTO v_reportavId
FROM
report_av
WHERE
title = p_title;
IF
( v_authorId IS NULL ) THEN
INSERT INTO author ( surname, fname, email, phone )
VALUES
( p_surname, p_fname, p_email, p_phone ) RETURNING author_id INTO v_authorId;
END IF;
IF
( v_reportavId IS NULL ) THEN
INSERT INTO report_av ( docu_type, YEAR, title, citation, digital, url, call_number, report_no, docu_description )
VALUES
( p_docutype, p_year, p_title, p_citation, p_digital, p_url, p_call_no, p_report_no, p_docu_description ) RETURNING reportav_id INTO v_reportavId;
END IF;
INSERT INTO jnc_author_reportav
VALUES ( v_authorId, v_reportavId );
END;
$BODY$ LANGUAGE plpgsql VOLATILE COST 100
Related
Say, if I have multiple tables that have the same schema:
CREATE TABLE `tbl01`
(
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`name` TINYTEXT,
`data` INT
);
CREATE TABLE `tbl02`
(
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`name` TINYTEXT,
`data` INT
);
CREATE TABLE `tbl03`
(
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`name` TINYTEXT,
`data` INT
);
-- etc. ------------------
INSERT INTO `tbl01` (`name`, `data`) VALUES
('row 1', 1),
('row 2', 1),
('row 3', 3);
INSERT INTO `tbl02` (`name`, `data`) VALUES
('cube', 1),
('circle', 0);
INSERT INTO `tbl03` (`name`, `data`) VALUES
('one', 1);
and then one table that contains names of all other tables in one of its columns:
CREATE TABLE `AllTbls`
(
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`tblnm` VARCHAR(64) NOT NULL UNIQUE,
`desc` TINYTEXT,
`flgs` BIGINT UNSIGNED
);
INSERT INTO `AllTbls` (`tblnm`, `desc`, `flgs`) VALUES
('tbl01', 'Table 1', 0),
('tbl02', 'Table two', 1),
('tbl03', '3rd table', 0);
So if I want to write a query to retrieve contents of AllTbls and also in one column to include count of rows in each of corresponding tables, I thought the following would be the way to do it:
SELECT *, `tblnm` as TblName, (SELECT COUNT(*) FROM TblName) as cntRws
FROM `AllTbls` ORDER BY `id` ASC LIMIT 0,30;
But this returns an error:
#1146 - Table 'database.TblName' doesn't exist
I know that I can do this in multiple queries (using a loop in a programming language), but is it possible to do it in one query?
PS. I'm using MySQL v.5.7.28
The simple answer is: "you can't"
Table names are not supposed to be used like variables, to hold data, in this way. What you're supposed to have is one table:
tblContractCounts
Client, ContractCount
-------------------
IBM, 1
Microsoft, 3
Google, 2
Not three tables:
tblIBMContractCounts
ContractCount
1
tblMicrosoftContractCounts
ContractCount
3
tblGoogleContractCounts
ContractCount
2
If your number of tables is known and fixed you can perhaps remedy things by creating a view that unions them all back together, or embarking on an operation to put them all into one table, with separate views named the old names so things carry in working til you can change them. If new tables are added all the time it's a flaw in the data modelling and need to be corrected. In that case you'd have to use a programming language (front end or stored procedure) to build a single query:
//pseudo code
strSql = ""
for each row in dbquery("Select name from alltbls")
strSql += "select '" + row.name + "' as tbl, count(*) as ct from " + row.name + " union all "
next row
strSql += "select 'dummy', 0"
result = dbquery(strSql)
It doesn't have to be your front end that does this - you could also do this in mysql and leverage the dynamic sql / EXECUTE. See THIS ANSWER how we can concatenate a string using logic like above so that the string contains an sql query and then execute the query. The information schema will give you the info you need to get a list of all current table names
But all you're doing is working around the fact that your data modelling is broken; I recommend to fix that instead
ps: the INFORMATION_SCHEMA has rough counts for tables with their names, which may suffice for your needs in this particular case
select table_name, table_rows from infornation_schema.tables where table_name like ...
I managed to solve the problem using the following stored procedure.
-- DROP PROCEDURE sp_Count_Rows;
Delimiter $$
CREATE PROCEDURE sp_Count_Rows()
BEGIN
DECLARE table_name TEXT DEFAULT "";
DECLARE finished INTEGER DEFAULT 0;
DECLARE table_cursor
CURSOR FOR
SELECT tblnm FROM alltbls;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET finished = 1;
OPEN table_cursor;
DROP TABLE IF EXISTS RowsCount;
CREATE TABLE IF NOT EXISTS RowsCount(Tlbnm text, ctnRws int);
table_loop: LOOP
FETCH table_cursor INTO table_name;
IF finished = 1 THEN
LEAVE table_loop;
END IF;
SET #s = CONCAT("insert into RowsCount select '", table_name ,"', count(*) as cntRws from ", table_name);
PREPARE stmt1 FROM #s;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
END LOOP table_loop;
CLOSE table_cursor;
SELECT * FROM RowsCount;
DROP TABLE RowsCount;
END
$$
And then when you call the procedure
CALL sp_Count_Rows();
You get this result
I am moving data from Spreadsheets to MySQL.
So we know that in Spreadsheets usually there is no ID, instead, just text.
City;Country;...
New York;USA;...
Berlim;Germany;...
Munich,Germany,...
With that in mind, let's consider two tables:
Country : [ID, name]
City : [ID , country (FK) , name]
I dont want to create several countries with the same name -- but I want to use the existing one. Perfect, so, let's add a FUNCTION in the INSERT state that searches, insert (if needed) and return the Country ID.
So I created a Function to FIRST assess whether the Country exists if not then create a country
getCountry (parameter IN strCountry varchar(100))
BEGIN
SELECT ID INTO #id from `country` WHERE country.country = strCountry ;
IF (#id is NULL OR #id= 0) THEN
INSERT INTO `country` (country) VALUES (strCountry);
if (ROW_COUNT()>0) THEN
SET #id = LAST_INSERT_ID();
else
SET #id = NULL;
END IF;
END IF ;
RETURN #id;
END
And then I have DOZENS OF THOUSANDS of INSERTS such as
INSERT INTO city (name, country) VALUES ('name of the city', getCountry('new or existing one'));
The Function works well when executed alone, such as
SELECT getCountry('Aruba');
However, when I execute that in that VERY LONG SQL (22K+ rows) then it does not work.... it uses basically the latest ID that was created BEFORE starting the execution. Maybe I should "wait" the function execute and return a proper result? But How?
What am I doing wrong?
Instead of function why not use a Stored Procedure, then the procedure will process the checking and insertion.
https://www.mysqltutorial.org/getting-started-with-mysql-stored-procedures.aspx
DELIMITER $$
CREATE PROCEDURE `sp_city_add`(in p_city varchar(100), in p_country varchar(100))
BEGIN
DECLARE country_id INT;
IF (SELECT COUNT(1) FROM country WHERE country.country = p_country) = 0 THEN
INSERT INTO country (country) VALUE (p_country);
SET country_id = LAST_INSERT_ID();
ELSE
SELECT ID INTO country_id FROM country WHERE country.country = p_country;
END IF;
INSERT INTO city (name, country) VALUES (p_city, country_id);
END$$
DELIMITER ;
And if you want to execute a procedure
CALL sp_city_add('Bogota', 'Colombia');
CALL sp_city_add('Phnom Penh', 'Cambodia');
CALL sp_city_add('Yaounde', 'Cameroon');
CALL sp_city_add('Ottawa', 'Canada');
CALL sp_city_add('Santiago', 'Chile');
CALL sp_city_add('Beijing', 'China');
CALL sp_city_add('Bogotá', 'Colombia');
CALL sp_city_add('Moroni', 'Comoros');
You can also add a condition to check if the city and country exists to prevent duplicate entry.
I can't find any documentation of it, but maybe there's a conflict when you do an INSERT in a function that's called during another INSERT. So try splitting them up using a variable:
SELECT #country := getCountry('new or existing one');
INSERT INTO city (name, country) VALUES ('name of the city', #country);
Using the idea of #Barman, PLUS adding COMMIT to each row I could solve that:
SELECT #id := getCountry("Colombia");INSERT into city ( city, country) VALUES ('Bogota',#id);COMMIT;
SELECT #id := getCountry("Colombia");INSERT into city ( city, country) VALUES ('Medelin',#id);COMMIT;
SELECT #id := getCountry("Brazil");INSERT into city ( city, country) VALUES ('Medelin',#id);COMMIT;
SELECT #id := getCountry("Brazil");INSERT into city ( city, country) VALUES ('Sao Paulo',#id);COMMIT;
SELECT #id := getCountry("Brazil");INSERT into city ( city, country) VALUES ('Curitiba',#id);COMMIT;
SELECT #id := getCountry("USA");INSERT into city ( city, country) VALUES ('Boston',#id);COMMIT;
SELECT #id := getCountry("USA");INSERT into city ( city, country) VALUES ('DallaS',#id);COMMIT;
Without the COMMIT at the end of each row, MySQL was not calculating the variable anymore, instead, just throwing some last result it collected.
After reading this question, I'm trying to convert some SQL from MySQL to PostgreSQL. Thus I need variable assignation:
INSERT INTO main_categorie (description) VALUES ('Verbe normal');
SET #PRONOMINAL := SELECT LAST_INSERT_ID();
INSERT INTO main_mot (txt,im,date_c,date_v_d,date_l)
VALUES ('je m''abaisse',1,NOW(),NOW(),NOW());
SET #verbe_149 = SELECT LAST_INSERT_ID();
INSERT INTO main_motcategorie (mot_id,categorie_id) VALUES (#verbe_149,#PRONOMINAL);
How would you do this with PostgreSQL? No useful sample in the documentation of v9 and v8 (almost the same).
NB: I dont want to use a stored procedure like here, I just want "raw sql" so I can inject it through CLI interface.
There are no variables in Postgres SQL (you can use variables only in procedural languages).
Use RETURNING in WITH query:
WITH insert_cat AS (
INSERT INTO main_categorie (description)
VALUES ('Verbe normal')
RETURNING id
),
insert_mot AS (
INSERT INTO main_mot (txt,im,date_c,date_v_d,date_l)
VALUES ('je m''abaisse',1,NOW(),NOW(),NOW())
RETURNING id
)
INSERT INTO main_motcategorie (mot_id,categorie_id)
SELECT m.id, c.id
FROM insert_mot m, insert_cat c;
As an alternative, you can use custom configuration parameters in the way described in this post.
Create two functions:
create or replace function set_var (name text, value text)
returns void language plpgsql as $$
begin
execute format('set mysql.%s to %s', name, value);
end $$;
create or replace function get_var (name text)
returns text language plpgsql as $$
declare
rslt text;
begin
execute format('select current_setting(''mysql.%s'')', name) into rslt;
return rslt;
end $$;
With the functions you can simulate variables, like in the example:
INSERT INTO main_categorie (description)
VALUES ('Verbe normal');
SELECT set_var('PRONOMINAL', (SELECT currval('main_categorie_id_seq')::text));
INSERT INTO main_mot (txt,im,date_c,date_v_d,date_l)
VALUES ('je m''abaisse',1,NOW(),NOW(),NOW());
SELECT set_var('verbe_149', (SELECT currval('main_mot_id_seq')::text));
INSERT INTO main_motcategorie (mot_id,categorie_id)
SELECT get_var('verbe_149')::int, get_var('PRONOMINAL')::int;
This is certainly not an example of good code.
Particularly the necessity of casting is troublesome.
However, the conversion can be done semi-automatically.
You can run PostgreSQL scripts outside of a function using the do construct. Here's an example with Donald Ducks' nephews. First the nephew will be added to the nephew table, and then we'll add a baseball cap using the newly inserted nephew's id.
First, create two tables for nephews and baseball caps:
drop table if exists nephew;
drop table if exists cap;
create table nephew (id serial primary key, name text);
create table cap (id serial, nephewid bigint, color text);
Now add the first nephew:
do $$declare
newid bigint;
begin
insert into nephew (name) values ('Huey') returning id into newid;
insert into cap (nephewid, color) values (newid, 'Red');
end$$;
The returning ... into ... does in Postgres what currval does in MySQL. Huey's new id is assigned to the newid variable, and then used to insert a new row into the cap table. You can run this script just like any other SQL statement. Continue with Dewey and Louie:
do $$declare
newid bigint;
begin
insert into nephew (name) values ('Dewey') returning id into newid;
insert into nephew (name) values ('Louie') returning id into newid;
insert into cap (nephewid, color) values (newid, 'Green');
end$$;
And you end up with:
# select * from nephew;
id | name
----+-------
1 | Huey
2 | Dewey
3 | Louie
(3 rows)
# select * from cap;
id | nephewid | color
----+----------+-------
1 | 1 | Red
2 | 3 | Green
(2 rows)
See it working at SQL Fiddle.
1) I have a query that writes the list of actions to temp table1:
declare #table1 table (ActionName varchar(250), ActionDate datetime, ... Description varchar(100))
insert into #table1(ActionName, ActionDate, ... Description)
select --Search actions by ActionName
The actions can have multiple action dates, which is important and it means we can have multiple records with the same action name, but different action dates.
2) I have to reduce table1 to get all action names appears only once, but with additional field DateList (comma separated string) that should contain the list of all date actions (ActionDate field) for that action:
declare #table2 table (ActionName varchar(250), ActionDate datetime, ... Description varchar(100), DateList varchar(4000))
I know how to create the list of action dates:
declare #dateList varchar(4000)
select #dateList = coalesce(#dateList + ', ', '') + convert(varchar, ActionDate, 120)
from #temp
But, I'm not sure how to incorporate it in the whole solution.
You need to wrap your list-creation code in a function and then use that on your table like this:
insert into #table2
select ActionName, uf_GetDateList(ActionName)
from #table1
group by ActionName
Now the function itself is a tricky part. You need to pass the #table1 down there somehow. You can either turn #table1 into a temporary table #table1 then you would do something like this:
select #dateList = coalesce(#dateList + ', ', '') + convert(varchar, ActionDate, 120)
from #temp1
where ActionName=#ActionName
Or you can use table-valued parameter to pass the table variable into the function.
I created the following stored procedure:
CREATE DEFINER=`root`#`localhost` PROCEDURE `add_summit`(IN `assoc_code` CHAR(5), IN `assoc_name` CHAR(50), IN `reg_code` CHAR(2), IN `reg_name` CHAR(100), IN `code` CHAR(20), IN `name` CHAR(100), IN `sota_id` CHAR(5), IN `altitude_m` SMALLINT(5), IN `altitude_ft` SMALLINT(5), IN `longitude` DECIMAL(10,4), IN `latitude` DECIMAL(10,4), IN `points` TINYINT(3), IN `bonus_points` TINYINT(3), IN `valid_from` DATE, IN `valid_to` DATE)
BEGIN
declare assoc_id SMALLINT(5);
declare region_id SMALLINT(5);
declare summit_id MEDIUMINT(8);
-- ASSOCIATION check if an association with the given code and name already exists
SELECT id INTO assoc_id FROM association WHERE code = assoc_code LIMIT 1;
IF (assoc_id IS NULL) THEN
INSERT INTO association(code, name) VALUES (assoc_code, assoc_name);
set assoc_id = (select last_insert_id());
END IF;
-- REGION check if a region with the given code and name already exists
SET region_id = (SELECT id FROM region WHERE code = reg_code AND name = reg_name AND association_id = assoc_id);
IF (region_id IS NULL) THEN
INSERT INTO region(association_id, code, name) VALUES (assoc_id, reg_code, reg_name);
set region_id = (select last_insert_id());
END IF;
-- SUMMIT check if a summit with given parameters already exists
SET summit_id = (SELECT id FROM summit WHERE association_id = assoc_id AND region_id = region_id);
IF (summit_id IS NULL) THEN
INSERT INTO summit(code, name, sota_id, association_id, region_id, altitude_m, altitude_ft, longitude,
latitude, points, bonus_points, valid_from, valid_to)
VALUES (code, name, sota_id, assoc_id, region_id, altitude_m, altitude_ft, longitude, latitude,
points, bonus_points, valid_from, valid_to);
END IF;
END$$
basically, it should check if a record exists in some tables and, if it doesn't, it should insert it and use the inserted id (auto increment).
The problem is that even if the record exists (for instance in the association table), assoc_id keeps returning null and that leads to record duplication.
I'm new to stored procedures so I may be doing some stupid errors. I've been trying to debug this SP for hours but I cannot find the problem.
A newbie mistake.
I forgot to specify the table name in the field comparison and that leads to some conflicts with param names (for example the param name).
A good idea is to specify some kind of prefix for parameters (like p_) and always specify the name of the table in the SP.