I've got a SQL 2008 R2 table defined like this:
CREATE TABLE [dbo].[Search_Name](
[Id] [bigint] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](300) NULL),
CONSTRAINT [PK_Search_Name] PRIMARY KEY CLUSTERED ([Id] ASC))
Performance querying the Name field using CONTAINS and FREETEXT works well.
However, I'm trying to keep the values of my Name column unique. Searching for an existing entry in the Name column is unbelievably slow for a large number of names (usually batches of 1,000), even with an index on the Name field. Query plans indicate I'm using the index as expected.
To search for an existing value, my query looks like this:
SELECT TOP 1 Id, Name from Search_Name where Name = 'My Name Value'
I've tried duplicating the Name column to another column and searching on the new column, but the net effect was the same.
At this point, I'm thinking I must be mis-using this feature.
Should I just stop trying to prevent duplication? I'm using a linking table to join these search name values to the underlying data. It seems somehow 'dirty' to just store a whole bunch of duplicate values...
...or is there faster way to take a list of 1,000 names and see which ones are already stored in the database?
The first change to make is to get the entire list to SQL Server at one time. Regardless of how you add the names to the existing table, doing it as a set operation will make a big difference in performance.
Passing the List as a table-valued parameter (TVP) is a clean way to handle it. Have a look here for an example. You can still use an OUTPUT clause to track which rows did or didn't make the cut, for example:
-- Some sample existing names.
declare #Search_Name as Table ( Id Int Identity, Name VarChar(32) );
insert into #Search_Name ( Name ) values ( 'Bob' ), ( 'Carol' ), ( 'Ted' ), ( 'Alice' );
select * from #Search_Name;
-- Some (prospective) new names.
declare #New_Names as Table ( Name VarChar(32) );
insert into #New_Names ( Name ) values ( 'Ralph' ), ( 'Alice' ), ( 'Ed' ), ( 'Trixie' );
select * from #New_Names;
-- Add the unique new names.
declare #Inserted as Table ( Id Int, Name VarChar(32) );
insert into #Search_Name
output inserted.Id, inserted.Name into #Inserted
select New.Name
from #New_Names as New left outer join
#Search_Name as Old on Old.Name = New.Name
where Old.Id is NULL;
-- Results.
select * from #Search_Name;
-- The names that were added and their id's.
select * from #Inserted;
-- The names that were not added.
select New.Name
from #New_Names as New left outer join
#Inserted as I on I.Name = New.Name
where I.Id is NULL;
Alternatively, you could use a MERGE statement and OUTPUT the names that were added, those that weren't, or both.
Related
It's been my first question to this website, I'm sorry if I used any wrong keywords. I have been with one problem from quite a few days.
The Problem is, I have a MYSQL table named property where I wanted to add a ref number which will be a unique 6 digit non incremental number so I alter the table to add a new column named property_ref which has default value as 1.
ALTER TABLE property ADD uniqueIdentifier INT DEFAULT (1) ;
Then I write a script to first generate a number then checking it to db if exist or not and If not exist then update the row with the random number
Here is the snippet I tried,
with cte as (
select subIdentifier, id from (
SELECT id, LPAD(FLOOR(RAND() * (999999 - 100000) + 100000), 6, 0) AS subIdentifier
FROM property as p1
WHERE "subIdentifier" NOT IN (SELECT uniqueIdentifier FROM property as p2)
) as innerTable group by subIdentifier
)
UPDATE property SET uniqueIdentifier = (
select subIdentifier from cte as c where c.id = property.id
) where property.id != ''
this query returns a set of record for almost all the rows but I have a table of entries of total 20000,
but this query fills up for ~19000 and rest of the rows are null.
here is a current output
[current result picture]
If anyone can help, I am extremely thanks for that.
Thanks
Instead of trying to randomly generate unique numbers that do not exist in the table, I would try the approach of randomly generating numbers using the ID column as a seed; as long as the ID number is unique, the new number will be unique as well. This is not technically fully "random" but it may be sufficient for your needs.
https://www.db-fiddle.com/f/iqMPDK8AmdvAoTbon1Yn6J/1
update Property set
UniqueIdentifier = round(rand(id)*1000000)
where UniqueIdentifier is null
SELECT id, round(rand(id)*1000000) as UniqueIdentifier FROM test;
I've got the following table:
productId price
1 price_value1
2 price_value2
3 price_value3
I would like to insert a new product into the table and assign it a new productId. In this case its value equals to 4.
So I want my new table to look like so:
productId price
1 price_value1
2 price_value2
3 price_value3
4 price_value4
So as far as I understand, in order to do that I have to somehow retrieve the max value of productId and insert it using INSERT INTO mytable VALUES (productId + 1, price_value4).
But how do I find out the maximum value of productId?
I tried INSERT INTO mytable VALUES (SELECT MAX(productId) + 1 FROM mytable, price_value4) but it didn't work.
This should Work:
Select the max(productID) and price_value4 as a columns from mytable and insert the result.
INSERT INTO mytable (SELECT MAX(productId) + 1, 'price_value4' FROM mytable);
However, if you are not going to jump some number you can just add an auto increment id key to product_id and then you will have only to insert the price, the product ID will be incremented automatically..
This will do so :
ALTER TABLE mytable
MODIFY COLUMN `productId` INT(10) UNSIGNED PRIMARY KEY AUTO_INCREMENT;
you can change INT(10) with the INT(5) for example depanding on the size you want to give to your productId column
EDIT :
In return to the OP question in comments why his solution wouldn't work
Some suggetions says you have to make the SELECT statment in insert always between parenthesis
INSERT INTO mytable VALUES ( (SELECT MAX(ID)+1 FROM mytable) , price_value4)
.. In my Case it Return
(1093): You can't specify target table
'mytable' for update in FROM clause
AND HERE IS WHY (Quoting From the documentation)
When selecting from and inserting into the same table, MySQL creates
an internal temporary table to hold the rows from the SELECT and then
inserts those rows into the target table. However, you cannot use
INSERT INTO t ... SELECT ... FROM t when t is a TEMPORARY table,
because TEMPORARY tables cannot be referred to twice in the same
statement
BUT there is away to overcome by using a query instead of the table itself in the FROM, which has the effect of copying the requested table values instead of referencing the one that you are updating..
INSERT INTO mytable VALUES (
(SELECT MAX(ID)+1 FROM (SELECT * FROM mytable ) as mytmp ),
'price_value4');
OR (Quoting From the documentation)
To avoid ambiguous column reference problems when the SELECT and the
INSERT refer to the same table, provide a unique alias for each table
used in the SELECT part, and qualify column names in that part with
the appropriate alias.
INSERT INTO mytable Values ( (SELECT MAX(ID)+1 FROM mytable as mytmp) , 'price_value4')
This is a duplicate question. In order to take advantage of the auto-incrementing capability of the column, do not supply a value for that column when inserting rows.
A simple syntax to create table
CREATE TABLE Product (
productId MEDIUMINT NOT NULL AUTO_INCREMENT,
price INT NOT NULL,
PRIMARY KEY (productid)
);
While inserting supplied default or leave column as blank or supplied value as NULL. Take a look at below code snippet.
INSERT INTO Product (price) VALUES
('10'),('20'),('4'),
('30');
refer this link
I know, deleting duplicates from mysql is often discussed here. But none of the solution work fine within my case.
So, I have a DB with Address Data nearly like this:
ID; Anrede; Vorname; Nachname; Strasse; Hausnummer; PLZ; Ort; Nummer_Art; Vorwahl; Rufnummer
ID is primary Key and unique.
And i have entrys for example like this:
1;Herr;Michael;Müller;Testweg;1;55555;Testhausen;Mobile;012345;67890
2;Herr;Michael;Müller;Testweg;1;55555;Testhausen;Fixed;045678;877656
The different PhoneNumber are not the problem, because they are not relevant for me. So i just want to delete the duplicates in Lastname, Street and Zipcode. In that case ID 1 or ID 2. Which one of both doesn't matter.
I tried it actually like this with delete:
DELETE db
FROM Import_Daten db,
Import_Daten dbl
WHERE db.id > dbl.id AND
db.Lastname = dbl.Lastname AND
db.Strasse = dbl.Strasse AND
db.PLZ = dbl.PLZ;
And insert into a copy table:
INSERT INTO Import_Daten_1
SELECT MIN(db.id),
db.Anrede,
db.Firstname,
db.Lastname,
db.Branche,
db.Strasse,
db.Hausnummer,
db.Ortsteil,
db.Land,
db.PLZ,
db.Ort,
db.Kontaktart,
db.Vorwahl,
db.Durchwahl
FROM Import_Daten db,
Import_Daten dbl
WHERE db.lastname = dbl.lastname AND
db.Strasse = dbl.Strasse And
db.PLZ = dbl.PLZ;
The complete table contains over 10Mio rows. The size is actually my problem. The mysql runs on a MAMP Server on a Macbook with 1,5GHZ and 4GB RAM. So not really fast. SQL Statements run in a phpmyadmin. Actually i have no other system possibilities.
You can write a stored procedure that will each time select a different chunk of data (for example by rownumber between two values) and delete only from that range. This way you will slowly bit by bit delete your duplicates
A more effective two table solution can look like following.
We can store only the data we really need to delete and only the fields that contain duplicate information.
Let's assume we are looking for duplicate data in Lastname , Branche, Haushummer fields.
Create table to hold the duplicate data
DROP TABLE data_to_delete;
Populate the table with data we need to delete ( I assume all fields have VARCHAR(255) type )
CREATE TABLE data_to_delete (
id BIGINT COMMENT 'this field will contain ID of row that we will not delete',
cnt INT,
Lastname VARCHAR(255),
Branche VARCHAR(255),
Hausnummer VARCHAR(255)
) AS SELECT
min(t1.id) AS id,
count(*) AS cnt,
t1.Lastname,
t1.Branche,
t1.Hausnummer
FROM Import_Daten AS t1
GROUP BY t1.Lastname, t1.Branche, t1.Hausnummer
HAVING count(*)>1 ;
Now let's delete duplicate data and leave only one record of all duplicate sets
DELETE Import_Daten
FROM Import_Daten LEFT JOIN data_to_delete
ON Import_Daten.Lastname=data_to_delete.Lastname
AND Import_Daten.Branche=data_to_delete.Branche
AND Import_Daten.Hausnummer = data_to_delete.Hausnummer
WHERE Import_Daten.id != data_to_delete.id;
DROP TABLE data_to_delete;
You can add a new column e.g. uq and make it UNIQUE.
ALTER TABLE Import_Daten
ADD COLUMN `uq` BINARY(16) NULL,
ADD UNIQUE INDEX `uq_UNIQUE` (`uq` ASC);
When this is done you can execute an UPDATE query like this
UPDATE IGNORE Import_Daten
SET
uq = UNHEX(
MD5(
CONCAT(
Import_Daten.Lastname,
Import_Daten.Street,
Import_Daten.Zipcode
)
)
)
WHERE
uq IS NULL;
Once all entries are updated and the query is executed again, all duplicates will have the uq field with a value=NULL and can be removed.
The result then is:
0 row(s) affected, 1 warning(s): 1062 Duplicate entry...
For newly added rows always create the uq hash and and consider using this as the primary key once all entries are unique.
I need to do this: On inserted record I need to store Inserted item identity and selected item identity. (Example below)
I'm using after insert trigger (basically I copy one row from one table into another and do some more modifications.
I have a table parameter like this:
DECLARE #Tempequipment TABLE
(Equipment_Id int,
DefaultEquipment_Id INT)
Then I insert into table like this:
INSERT INTO dbo.tblEquipmentType
( Name, EquipmentType_Id)
SELECT name,(SELECT Equiment_Id FROM INSERTED)
FROM dbo.tblDefaultEquipmentType
This works fine!
What I need to do is: I need to insert into #TempEquipment EquipmentTypeId's that were just ineserted (can be more than one) and DefaultEquipmentTypeId's that were just copied.
I was thinking about doing something like:
INSERT INTO dbo.tblEquipmentType
( Name, EquipmentType_Id)
Output EquipmentTypeId, DefaultEquipmentTypeId into #TempEquipment
SELECT name,(SELECT Equipment_Id FROM INSERTED)
FROM dbo.tblDefaultEquipmentType
but of course this is not going to work, since it cannot get values from select statement, and not written correctly.
Any help is appreciated!
UPDATE:
I have an Item. Item can be built on different equipment. Equipment has types (foreign key. And equipmentType has attributes (foreignkey).
So this mean that we have four tables Item->Equipment->EquipmentType->EquipmentAttribute.
I need to store default EquipmentTypes and default EquipmentAtrributes for that type.
So I also got these replationship: Equipment->DefaultEquipmentType->DefaultEquipmentAttribute.
Now, When I insert new Item and select an equipment I want to copy defaults over to real tables (EquipmentType, EquipmentAttribute).
Is it clear at least a little?
Aside from how you're trying to do this (which isn't working), what specifically are you trying to do?
It may be that this can be resolved by changing / normalizing your paradigm, instead of some kind of exotic code. For example, it looks odd to have a customers table with an orderID field in it. Unless your customers only ever order one thing... I would have expected to see a customers table, an items table, and then an orders table that joined customers with items.
Hope that makes sense -- but anyway, if not, can you post your table structure, and maybe be a little more clear on what you know ahead of time (e.g., I imagine you know who your customers are, and what they ordered...before you do the insert...yes?)
For an INSERT statement you can only access the columns which are in the insert column list, so the solution is to rewrite the statement as a MERGE statement which can access all the columns including columns which are in the INSERT target table for instance IDENTITY columns.
In the demo I've used dbo.INSERTED to emulate the virtual table INSERTED from the trigger.
USE master
GO
IF DB_ID('MergeOutputExample') IS NOT NULL
DROP DATABASE MergeOutputExample
GO
CREATE DATABASE MergeOutputExample
GO
USE MergeOutputExample
GO
DECLARE #Tempequipment TABLE
(EquipmentId int,
DefaultEquipmentId INT,
ID int);
CREATE TABLE dbo.INSERTED
(
EquipmentTypeId int PRIMARY KEY
);
CREATE TABLE dbo.tblEquipmentType
(
ID int IDENTITY(1,1),
Name varchar(50),
EquipmentTypeId int PRIMARY KEY
);
CREATE TABLE dbo.tblDefaultEquipmentType
(
EquipmentTypeId int,
DefaultEquipmentTypeId int IDENTITY(1,1) PRIMARY KEY,
Name varchar(50)
);
INSERT dbo.inserted
(
EquipmentTypeId
)
VALUES (1);
INSERT dbo.tblDefaultEquipmentType
(
EquipmentTypeId,
Name
)
VALUES (
1,
'Hammer'
);
MERGE dbo.tblEquipmentType AS ET
USING (
SELECT DE.EquipmentTypeId,
DE.DefaultEquipmentTypeId,
DE.Name
FROM dbo.tblDefaultEquipmentType DE
INNER JOIN dbo.INSERTED I
ON DE.EquipmentTypeId = I.EquipmentTypeId
) AS DET
ON ET.EquipmentTypeId = DET.EquipmentTypeId
WHEN NOT MATCHED BY TARGET
THEN INSERT
(
Name,
EquipmentTypeID
)
VALUES
(
DET.Name,
DET.EquipmentTypeID
)
OUTPUT DET.EquipmentTypeId,
DET.DefaultEquipmentTypeId,
INSERTED.ID
INTO #Tempequipment;
SELECT *
FROM #Tempequipment;
SELECT *
FROM dbo.tblEquipmentType;
Simply Asking, Is there any function available in mysql to split single row elements in to multiple columns ?
I have a table row with the fields, user_id, user_name, user_location.
In this a user can add multiple locations. I am imploding the locations and storing it in a table as a single row using php.
When i am showing the user records in a grid view, I am getting problem for pagination as i am showing the records by splitting the user_locations. So I need to split the user_locations ( single row to multiple columns).
Is there any function available in mysql to split and count the records by character ( % ).
For Example the user_location having US%UK%JAPAN%CANADA
How can i split this record in to 4 columns.
I need to check for the count values (4) also. thanks in advance.
First normalize the string, removing empty locations and making sure there's a % at the end:
select replace(concat(user_location,'%'),'%%','%') as str
from YourTable where user_id = 1
Then we can count the number of entries with a trick. Replace '%' with '% ', and count the number of spaces added to the string. For example:
select length(replace(str, '%', '% ')) - length(str)
as LocationCount
from (
select replace(concat(user_location,'%'),'%%','%') as str
from YourTable where user_id = 1
) normalized
Using substring_index, we can add columns for a number of locations:
select length(replace(str, '%', '% ')) - length(str)
as LocationCount
, substring_index(substring_index(str,'%',1),'%',-1) as Loc1
, substring_index(substring_index(str,'%',2),'%',-1) as Loc2
, substring_index(substring_index(str,'%',3),'%',-1) as Loc3
from (
select replace(concat(user_location,'%'),'%%','%') as str
from YourTable where user_id = 1
) normalized
For your example US%UK%JAPAN%CANADA, this prints:
LocationCount Loc1 Loc2 Loc3
4 US UK JAPAN
So you see it can be done, but parsing strings isn't one of SQL's strengths.
The "right thing" would be splitting the locations off to another table and establish a many-to-many relationship between them.
create table users (
id int not null auto_increment primary key,
name varchar(64)
)
create table locations (
id int not null auto_increment primary key,
name varchar(64)
)
create table users_locations (
id int not null auto_increment primary key,
user_id int not null,
location_id int not null,
unique index user_location_unique_together (user_id, location_id)
)
Then, ensure referential integrity either using foreign keys (and InnoDB engine) or triggers.
this should do it
DELIMITER $$
DROP PROCEDURE IF EXISTS `CSV2LST`$$
CREATE DEFINER=`root`#`%` PROCEDURE `CSV2LST`(IN csv_ TEXT)
BEGIN
SET #s=CONCAT('select \"',REPLACE(csv_,',','\" union select \"'),'\";');
PREPARE stmt FROM #s;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END$$
DELIMITER ;
You should do this in your client application, not on the database.
When you make a SQL query you must statically specify the columns you want to get, that is, you tell the DB the columns you want in your resultset BEFORE executing it. For instance, if you have a datetime stored, you may do something like select month(birthday), select year(birthday) from ..., so in this case we split the column birthday into 2 other columns, but it is specified in the query what columns we will have.
In your case, you would have to get exactly that US%UK%JAPAN%CANADA string from the database, and then you split it later in your software, i.e.
/* get data from database */
/* ... */
$user_location = ... /* extract the field from the resultset */
$user_locations = explode("%", $user_location);
This is a bad design, If you can change it, store the data in 2 tables:
table users: id, name, surname ...
table users_location: user_id (fk), location
users_location would have a foreign key to users thorugh user_id field