i have two tables, with same columns. but there are duplicates. i want to create third one but without duplicate rows. what is the best way to do this keeping in mind that tables have over million records?
Table have two columns, ean and price
If both the tables have same structure then , I think you should try union.
Sample Query--
Assuming 2 table from which the data needs to be retrieved as
table_src_1
table_src_2
Final Table-table_unique_records
Your Query-
create table table_unique_records as
(select * from table_src_1
union
select * from table_src_2)
Related
Struggling with a large dataset in my mariaDB database. I have two tables, where table A contains 57 million rows and table B contains around 500. Table B is a subset of ids related to a column in table A. I want to delete all rows from A which do not have a corresponding ID in table B.
Example table A:
classification_id
Name
20
Mercedes
30
Kawasaki
80
Leitz
70
HP
Example table B:
classification_id
Type
20
car
30
bike
40
bus
50
boat
So in this example the last two rows from table A would be deleted (or a mirror table would be made containing only the first two rows, thats also fine).
I tried to do the second one using an inner join but this query took a few minutes before giving an out of memory exception.
Any suggestions on how to tackle this?
try this:
delete from "table A" where classification_id not in (select classification_id from "table B");
Since you say that the filter table contains a relatively small number of rows, your best bet would be creating a separate table that contains the same columns as the original table A and the rows that match your criteria, then replace the original table and drop it. Also, with this number of IDs you probably want to use WHERE IN () instead of joins - as long as the field you're using there is indexed, it will usually be way faster. Bringing it all together:
CREATE TABLE new_A AS
SELECT A.* FROM A
WHERE classification_id IN (SELECT classification_id FROM B);
RENAME TABLE A TO old_A, new_A to A;
DROP TABLE old_A;
Things to be aware of:
Backup your data! And test the queries thoroughly before running that DROP TABLE. You don't want to lose 57M rows of data because of a random answer at StackOverflow.
If A has any indexes or foreign keys, these won't be copied over - so you'll have to recreate them all manually. I'd recommend running SHOW CREATE TABLE A first and making note on its structure. Alternatively, you may consider creating the table new_A explicitly using the output of SHOW CREATE TABLE A as a template and then performing INSERT INTO new_A SELECT ... instead of CREATE TABLE new_A AS SELECT ... with the same query after this.
In my DB a table contains 20 long-blob columns(columns can't be moved in another table) Each row contains a data size of 0.55 MB approximately. i created indexes on that table but it doesn't improve the performance.We need to store more than 1 million records in this table so i am planning to partition this table.
My question is whether Partitioning will improve the performance or not (Read/Write)?
Can anybody help me on this how to improve it.Suggest if any alternate way to improve.
Try splitting the blobs into a separate table.
For example, you could have a table table1 containing the columns id and name, and another table table2 containing the columns id and data. If you only want the name, you don't need to query the table2 table at all; if you need both the name and the data, you can join the tables together using the shared primary key id:
SELECT id, name, data
FROM table1 JOIN table2 USING (id)
WHERE id = ...
I have a huge DB table(38 GB) in which there are many columns which are having all its value as null. Problem is that before creating table you are not aware which columns will have data and due to that we have to keep all columns while creating table. But due to this performance of queries are very bad.
So need to find all columns which have all its value null reduce size of table. Also when inner joins are done it takes too much of time. So is it the case that inner join on large tables takes more time.
Do a
SELECT Count(distinct colName) FROM myTable
For eacht column. This way you will get als only result if there is no other value. You can then
ALTER TABLE myTable DROP COLUMN colName
to drop the col.
An alternative might be that you do a
SELECT * FROM myTable procedure Analyse()
This way you will get an overview on your table with all columns containing two interesting columns: Empties_or_zeros and Nulls. Both contain the Count of empty rows.
I have two tables that I am trying to perform matching on with the following query:
select * from task1,task2
where task1.From_Number=task2.To_Number
and task1.Start_Time<task2.Start_Time;
It will work eventually but is taking forever. The tables have 33 columns and one has around 45k rows and the other 500k rows. There are duplicates in various columns and no column is unique so there isn't a primary key. The tables were imported from spreadsheets.
There are a bunch of phone call logs and as mentioned, there are several duplicates in each column. What can I do to get this query to run faster? I am only matching against a few columns but need to print all columns and output the result into a csv.
The best thing you can do is create an non-unique index on both columns in both tables.
Read the MySQL documentation on creating an index.
Something like:
create index task1_idx
on task1
( From_Number
, Start_Time
)
And:
create index task2_idx
on task2
( To_Number
, Start_Time
)
I've been trying to figure this out, but can't seem to come up with a simple solution.
Say for instance I have a table that has similar data throughout 3 columns (i.e. different types of activities spanning 3 columns) but I want to have those three columns inserted into a separate table (Table2) so I can keep the like data together and perform a JOIN to match it with its respective data in Table1.
I'm not talking about performing a CONCAT or CONCAT_WS, but moving those three columns from Table1 into one column in Table2, each item with its own row.
Is there a way to do this through a query without having to manually insert each entry into Table2?
Thank you in advance!
It might be as simple as:
insert into table2
(field)
select column1 from table1
union
select column2 from table1
union
select column3 from table1
But, before you do this, decide what you want to do if two columns in table1 have the same value.