I have to join 3 tables from 3 different database.
I am fetching records from table 1 (every hour and around 100K record every hour). Using the key from table1, I will have to get record from table 2 and then using key from table 2, i will have to fetch record from table 3.
I am thinking of using IN clause however I am not sure if that would be good option.
Also, Records in table2 and table3 are less in numbers and wont change frequently. So there is an option of using second level cache to get records of these two tables in cache and then filter records based on what is required for table1.
should I join tables approach or use cache for less frequently updated tables.
Please suggest.
Related
I'm working on automating the process of building a database. This is a database that needs daily updates after one build.
This database has 51 tables, divided into 3 schemas (there are 17 tables in each schema), and has a total of 20 million records, each record with a PK of manage_number .
I need to update 2000~3000 records every day, but I don't know which method to use.
Make a table for PK indexing
This is a method to separately create a table with a PK and a table name with a PK in the same database. To create a table with metadata about which table the manage_number is stored in. Currently, this methodology is applied. The problem is that the build time takes 5-6 times longer than before (increased from 2 minutes to 12 minutes).
Multi-table update query
This is how to perform update query on 17 tables with the same schema. However, in this case, the load in the FROM clause is expected to be very high, and I think it will be a burden on the server.
Update query may looks like below.
UPDATE table1, table2, table3, ..., table17
SET data_here
WHERE manage_number = 'TARGET_NUMBER';
Please share which way is better, or if you have a better way.
Thank you.
I have two big tables for example:
'tbl_items' and 'tbl_items_transactions'
First table keeping some items metadata which may have 20 (varchar) columns with millions rows... and second table keeping each transaction of first table.
for example if a user insert new record to tbl_items then automatically a new record will be adding to tbl_items_transactions with same data plus date, username and transaction type to keep each row history.
so in the above scenario two tables have same columns but tbl_items_transactions have 3 extra columns date, username, transaction_type to keep each tbl_items history
now assume we have 1000 users that wants to Insert, Update, Delete tbl_items records with a web application. so these two tables scale very soon (maybe billion rows in tbl_items_transactions)
I have tried MySQL, MariaDB, PostgreSQL... they are very good but when table scale and millions rows inserted they are slow when run some select queries on tbl_items_transactions... but sometimes PostgreSQL is faster than MySQL or MariaDB
now I think I'm doing wrong things... If you was me... do you use MariaDB or PostgreSQL or somthing like that and structure your database like what I did?
Your setup is wrong.
You should not duplicate the columns from tbl_items in tbl_items_transactions, rather you should have a foreign key in the latter table pointing to the former.
That way data integrity is preserved, and tbl_items_transactions will be much smaller. This technique is called normalization.
To speed up queries when the table get large, define indexes on them that match the WHERE and JOIN conditions.
I have to large tables, a main one (TableA) with around 14 million records and a second one I want to merge into it with 20 million records (TableB). For the most part the first is a subset of the second.
I tried making a Unique Index using 2 or 3 fields combined that would identify records as such but MySql wouldn't do it.
I then made my own field 'Unique' by concatenating those three fields.
My question is how do I an import TableB into TableA using only unique records i.e. ones where the value in Unique field in TableB does not already exist in Unique field in TableA. Since I could not make the Unique field an actual unique index will/should I try to make each one a PK and or ordinary index in the respective tables?
Any thoughts on how to do this efficiently appreciated.
Use the sql union statement.
"select * from tableA join tableb"
I have two tables T1 and T2 and want to update one field of T1 from T2 where T2 holds massive data.
What is more efficient?
Updating T1 in a for loop iteration over the values
or
Left join it with T2 and update.
Please note that i'm updating these tables in a shell script
In general, the JOIN will always work much better than a loop. The size should not be an issue if it is properly indexed.
There is no simple answer which will be more effective, it will depend on table size and data size to which you are going to update in one go.
Suppose you are using innodb engine and trying to update 1,000 or more rows in one go with 2 heavy tables join and it is quite frequent then it will not be good idea on production server as it will lock your table for some time and due to this locking some other operations also can be hit on your production server.
Option1: If you are trying to update few rows and based on proper indexed fields (preferred based on primary key) then you can go with join.
Option2: If you are trying to update a large amount of data based on multiple tables join then below option will be better:
Step1: Create a stored procedure.
Step2: Keep below query results in a cursor.
suppose you want TO UPDATE corresponding field2 DATA of TABLE table2 IN field1 of TABLE table1:
SELECT a.primary_key,b.field2 FROM table1 a JOIN table2 b ON a.primary_key=b.foreign_key WHERE [place CONDITION here IF any...];
Step3: Now update all rows one by one based on primary key using stored values in cursor.
Step4: You can call this stored procedure from your script.
the scripts i've been working with in SQL work with close to 40,000 records and i've noticed a huge increase in execution time for when i use an UPDATE command
in 2 tables that have like 10 fields in them each, INSERT executes quicker for both combined than this UPDATE command
UPADTE table1
INNER JOIN table2 ON table1.primarykey = table2.primarykey
SET table1.code = table2.code
really what the UPDATE is doing is copying the code from one table to another where the identical records exists, this is because table1 is a staging table between 2 databases while table2 is a possessing table to insert staging table's data across multiple tables, both tables have the same number of record which is about 40,000
now to me UPDATE should be executing a lot quicker, considering it's only connecting 2 identical tables and inserting data for 1 field it should be running quicker than 2 INSERTS where 40,000 records are being created over 10 fields (so in other words, inserting 800,000 pieces of data) and i'm running the queries in a SQL console windows to avoid php timeouts
is UPDATE somehow more resource hungry than INSERT and is there any way to get it to go faster (apart from changing the fact that i use a separate table for processing, the staging table updates frequently so i copy the data like a snapshot and work with that, the code field is NULL to begin with so i only copy over records with a NULL code meaning records where code is not NULLhave already been worked with)
Is that UPDATE command the actual SQL? Because you need a WHERE clause to avoid updating every record in the table...
Also, INSERT doesn't first need to find the record to update from 2 joined tables.