My original problem is that I need to insert a lot of records to DB, so to speed up, I want to use mysqlimport which takes a file of row values and load them to specified table. So suppose I have model Book, I couldn't simply use book.attributes.values as one of the fields is a hash that is serialized to db (using serialize), so I need to know what is the format this hash will be stored in in the db. Same for time and dates fields. Any help?
How about using SQL insert statements instead of serialization?
book = Book.new(:title => 'Much Ado About Nothing', author: 'William Shakespeare')
sql = book.class.arel_table.create_insert
.tap { |im| im.insert(record.send(
:arel_attributes_with_values_for_create,
record.attribute_names)) }
.to_sql
Related
I'm working a .NET app where the user selects some filters like date, id, etc.
What I need to do is query a SQL Server database table with those filters, and dump them into a MySQL table. I don't need all fields in the table, only a few.
So far, I need to loop through all records in the SQL Server Dataset and insert them one by one on my MySQL table.
Is there anyway of achieving better performance? I've been playing with Dapper but cant figure out a way to do something like:
Insert into MySQLTable (a,b,c)
Select a,b,c from SQLServerTable
where a=X and b=C
Any ideas?
Linked server option is not possible because we have no access to the SQL server configuration, so looking for the most efficient way of bulk inserting data.
If I were to do this inside .NET with dapper I'd use c# and do the following;
assumptions:
a table in both tables with the same schema;
CREATE Events (
EventId int,
EventName varchar(10));
A .Net class
public class Event
{
public int EventId { get; set; }
public string EventName { get; set; }
}
The snippet below should give you something you can use as a base.
List<Event> Events = new List<Event>();
var sqlInsert = "Insert into events( EventId, EventName ) values (#EventId, #EventName)";
using (IDbConnection sqlconn = new SqlConnection(Sqlconstr))
{
sqlconn.Open();
Events = sqlconn.Query<Event>("Select * from events").ToList();
using (IDbConnection mySqlconn = new SqlConnection(Sqlconstr))
{
mySqlconn.Open();
mySqlconn.Execute(sqlInsert, Events);
}
}
The snippet above selects the rows from the events table in SQL Server and populates the Events list. normally Dapper will return an IEnumerable<>, but you are casting that ToList(). Now with the Events list, you connect to MySQL and execute the insert statements against the Events list.
This snippet is just a barebones example. Without a transaction on the Execute, each row will be autocommitted. If you add a Transaction, it will commit when all the items in the Events list are inserted.
Of course there are disadvantages doing it this way. One important thing to realize is that if you are trying to insert 1million rows from SQL to MySQL, that list will contain 1million entries which will increase the memory footprint. In those cases I'd use Dapper's Buffered = false option. This will return the 1million rows 1 row at a time. Your c# code can them enumerate over the results and add the row to a list and keep a counter. after 1000 rows have been inserted into list you can do the insert part into MySQL then clear the list and contine enumerating over the rows.
this will keep the memory footprint of your application small while processing a large number of rows.
With all that said, nothing beats bulk insert at the server level.
-HTH
I am trying to write a Spring Batch Starter job that reads a CSV file and inserts the records into a MySQL DB. When it begins I want to save the start time in a tracking table, and when it ends, the end time in that same table. The table structure is like:
TRACKING : id, start_time, end_time
DATA: id, product, version, server, fk_trk_id
I am unable to find an example project that does such a thing. I believe this needs to be a Spring Batch Starter project that can handle multiple queries. i.e.
// insert start time
1. INSERT INTO tracking (start_time) VALUES (NOW(6));
// get last inserted id for foreign key
2. SET #last_id_in_tracking = LAST_INSERT_ID();
// read from CSV and insert data into 'data' DB table
3. INSERT INTO data (product, version, server, fk_trk_id) VALUES (mysql, 5.1.42, Server1, #last_id_in_tracking);
4. INSERT INTO data (product, version, server, fk_trk_id) VALUES (linux, 7.0, Server2, #last_id_in_tracking);
5. INSERT INTO data (product, version, server, fk_trk_id) VALUES (java, 8.0, Server3, #last_id_in_tracking);
// insert end time
6. UPDATE tracking SET end_time = NOW(6) WHERE fk_trk_id = #last_id_in_table1;
I'd like sample code and explanation on how to use those queries to multiple tables in the same Spring Batch Starter job.
start of edit section - additional question
I do have an additional question. In my entities I have them set-up to represent the relationships with annotations (i.e #ManyToOne, #JoinColumn)...
In your code, how would I get the trackingId from a referenced object? Let me explain:
My Code (Data.java):
#JsonManagedReference
#ManyToOne
#JoinColumn(name = "id")
private Tracking tracking;
Your code (Data.java):
#Column(name = "fk_trk_id")
private Long fkTrkId;
Your code (JobConfig.java):
final Data data = new Data();
data.setFkTrkId(trackingId);
How do I set the id with "setFkTrkId" when the relationship in my Entity is an object?
end of edit section - additional question
Here is an example app that does what you're asking. Please see the README for details.
https://github.com/joechev/examples/tree/master/csv-reader-db-writer
I have created a project for you as an example. Please refer to https://bigzidane.wordpress.com/2018/02/25/spring-batch-mysql-reader-writer-processor-listener/
This example simply has a Reader/Processor/Writer. The reader will read a CSV file and then process something and then write to database.
And we have a listener to capture StartJob and EndJob. For Start Job, we will insert an entry to DB and then return a generatedId. We will pass the same ID to writer when we stored entries.
Note: I'm sorry I'm reused an example I have already. So it may not match 100% as your question but technically it should be the same.
Thanks,
Nghia
I am trying to insert the contents of an JSON to a MySql database using Mule ESB. The JSON looks like:
{
"id":106636,
"client_id":9999,
"comments":"Credit",
"salesman_name":"Salvador Dali",
"cart_items":[
{"citem_id":1066819,"quantity":3},
{"citem_id":1066820,"quantity":10}
]
}
On mule I want to insert all data on a step like:
Insert INTO order_header(id,client_id,comments,salesman_name)
Insert INTO order_detail(id,citem_id,quantity)
Insert INTO order_detail(id,citem_id,quantity)
Currently i have come this far on Mule:
MuleSoft Flow
Use Bulk Execute operation of Database Connector.
You will insert into multiple tables.
for ex :
Query text
Insert INTO order_header(payload.id,payload.client_id,payload.comments,payload.salesman_name);
Insert INTO order_detail(payload.id,payload.cart_items[0].citem_id,payload.cart_items[0].quantity); etc..
There is an excellant article here http://www.dotnetfunda.com/articles/show/2078/parse-json-keys-to-insert-records-into-postgresql-database-using-mule
that should be of help. You may need to modify as you need to write the order_header data first and then use a collection splitter for the order_detail and wrap the whole in a transaction.
Ok. Since, you have already converted JSON into Object in the flow, you can refer individual values with their object reference like obj.id, obj.client_id etc.
Get a database connector next.
Configure your MySQL database in "Connector Configuration".
Operation: Choose "Bulk execute"
In "Query text" : Write multiple INSERT queries and pass appropriate values from Object (converted from JSON). Remember to separate multiple queries with semicolon (;) in Query text.
That's it !! Let me know if you face any issue. Hope it works for you..
I have saved a source code .ccp type file under column 'file' in file_details table in a mysql database. It is done not by using a program but by directly inserting through localhost/phpmyadmin interface. The type of the data field is BLOB. Then by accessing saved codes I am calculating the number of spaces of each source codes and I want to insert the values to the database column called 'spaces' in the same table. Here I have given the query that I used.
Statement stmt = conn.createStatement();
Statement count_to_db=conn.createStatement();
String query = "SELECT prog_num,file FROM file_details";
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
String file_content = rs.getString("file");
CalculatingSpaces calSpaces = new CalculatingSpaces();
String num_of_Spaces= calSpaces.CalculatingSpaces(rs, pw, file_content);
int spaces=Integer.parseInt(num_of_Spaces);
String space_query="INSERT INTO file_details(spaces) VALUES ('"+spaces+"')";
count_to_db.executeUpdate(space_query);
}
But when I am inserting the spaces they are inserted to a new row but not for the same row where I have uploaded the BLOB files. Is there any way to insert the number of spaces in-front of the BLOB files which spaces related to.
PS : So the problem is Can't I insert data column by column to a database?
If you're adding your table row for the first time, you'll have to use the SQL INSERT command.
If you want to update an existent row, you have to use the SQL UPDATE command.
Don't forget that you need to know which row(s) you want to UPDATE, so you'll also need to define a way to uniquely identify your rows (aka the Primary Key).
See http://dev.mysql.com/doc/refman/5.0/en/update.html for details
I have a table with 10 columns. Each row in the table was is originally a JSON Object that I receive in this format.
{"mainEntity":
"atlasId": 1234567
"calculatedGeography": False
"calculatedIndustry" : False
"geography": "G:6J"
"isPublic" = False
"name" = XYZ, Inc
"permId" = 12345678987
primaryRic=""
type=corporation
}
I am using jdbc and a mysql driver. The problem is my insert statements look very long and ugly(see example below) because of the high number of columns. Is there a way to solve this or is this the only way. Also, is there a way to insert multiple records at the same time with jdbc?
"INSERT INTO table_name VALUES(1234567, False, False, "G:6J", False, "XYZ, Inc", 12345678987, "", corporation"
Are you only wondering about style or also performance? always use prepared statements when you make inserts, this will unclutter your code and make sure the datatypes are all correct.
If it is about speed, you might try transactions, or even "load data infile". The load data method requires you to make a temporary CSV file that is directly loader into the database.