How much take a time to load 1GB csv file into MySQL?
System Configuration:-
4GB RAM
Windows 2007
I don't have much of an idea about windows, but if you can ensure that your biggest single table fits in the storage engine buffer pool, it should not take more than 30 - 45 minutes (worst case) even with your given system configuration.
Related
I'm trying to create a database with data collected from google n-grams. It's actually a lot of data, but after the creation of the CSV files the insertion was pretty fast. The problem is that, immediately after the insertion, the neo4j-import tool indexes the data, and this step its taking too much time. It's been more than an hour and it looks like it achieved 10% of progress.
Nodes
[*>:9.85 MB/s---------------|PROPERTIES(2)====|NODE:198.36 MB--|LABE|v:22.63 MB/s-------------] 25M
Done in 4m 54s 828ms
Prepare node index
[*SORT:295.94 MB-------------------------------------------------------------------------------] 26M
This is the console info atm. Does anyone have a suggestion about what to do to speed up this process?
Thank you. (:
Indexing takes a long time depending on number of nodes. I tried indexing with 10 million nodes and it took around 35 minutes, but you can still try these settings :
Increase your page cache size which is stored in '/var/lib/neo4j/conf/neo4j.properties' file (in my ubuntu system). Edit the following line
dbms.pagecache.memory=4g
according to your RAM, allocate size, here, 4g means 4gb space. Also, you can try changing java memory size which is stored in neo4j-wrapper.conf
wrapper.java.initmemory=1024
wrapper.java.maxmemory=1024
You can also read neo4j documentation on this - http://neo4j.com/docs/stable/configuration-io-examples.html
We have a MySQL master database which replicates to a MySQL slave. We were experiencing issues where MySQL was showing a high number of writes (but not an increased number of queries being ran) for a short period of time (a few hours). We are trying to investigate the cause.
Normally our binary logs are 1 GB in file size but during the period that we were experiencing these issues, the log files jumped to 8.5 GB.
When I run mysqlbinlog --short-form BINARYLOG.0000 on one of the 8.5 GB binary log it only returns 196 KB of queries and data. When I run mysqlbinlog --short-form on a normal binary log (1 GB) it returns around 8,500 KB worth of queries and database activity. That doesn't make sense because it has 7 GB more of data yet returns less than a 1 GB binary log file.
I see lots of these statements with very sequential timestamps, but I'm not sure if that's related to the problem because they're in both the normal period as well as when we experienced these issues.
SET TIMESTAMP=1391452372/*!*/;COMMIT/*!*/;
SET TIMESTAMP=1391452372/*!*/;BEGIN/*!*/;COMMIT/*!*/;
SET TIMESTAMP=1391452372/*!*/;BEGIN/*!*/;COMMIT/*!*/;
SET TIMESTAMP=1391452372/*!*/;BEGIN/*!*/;COMMIT/*!*/;
How can I determine what caused those binary logs to balloon in size which also caused high writes, so much so it took the server offline at points, almost like a DDoS attack would?
How could mysqlbinlog return so much less data, even though the binary log file itself had 7 GB more? What can I do to identify the difference between a normal period where the binary logs are 1 GB to the period we had issues with the 8 GB binary log? Thank you for any help you can provide.
Bill
I would guess that your log contains some form of LOAD DATA [LOCAL] INFILE commands and the data files associated with them. These commands do not generate much SQL output as their data is written to a temporary file by mysqlbinlog during processing. Can you check if the output contains any such LOAD DATA commands?
I exported one database, in phpmyadmin, this database info: size: 2.7 GiB, overhead:43.5 KiB, after I exported to my local computer(win 7), the size is :495 MB
Questions:
1.size: 2.7 GiB = 2.9 GB, why after exported to local computer, it is only 495 MB?
2.what does this mean 'overhead'?
If there are deletes or updates some dbs don't free the memory slots by itself and the difference to the really needed space can be huge, especially after many updates and deletes.
For example an update can lead to a new used slot and a slot marked as empty They just keep it for future needs. Some have a special command for compacting. But after exporting and importing all the unused space is freed.
Just try optimize table on the tables of your old db and compare them after.
I am new to cassandra and now I am trying to make a production server.
In documentation I read that data and commitlog should be on separated drives (btw I use hdd),
I tought the commitlog will increase to many Gb of data and I created 2 hard drives (both 100Gb), on first will be data (sstables) on second commitlog. But now in config I see:
commitlog_total_space_in_mb: 4096, and I think this shoul be 'the maximum heap size'. If commitlog reached this limit, then it seems that memtables increased in size and needs to be flushed to disk and also the data that was contained in memtables is removed from commitlog.
So tell me please if am I right: commitlog is like a backup of heap and cannot increase to hundreds of Gb?
And I do not need to create 100Gb hard drive for that, will be enough a 4Gb partition (on another hard drive, not the same where the data (sstables) is stored)?
commitlog is like a backup of heap and cannot increase to hundreds of Gb?
The commitlog is temporary storage used for a copy of data stored in the memtables. This is in case power is lost or the server otherwise crashes before the memtables can be written out as SSTables. As soon as a memtable is flushed to disk, the commitlog segments are deleted.
The rest of the data on heap (caches, in-flight data, etc.) is never stored in a commitlog. So your commitlog will normally be significantly smaller than your heap.
And I do not need to create 100Gb hard drive for that, will be enough a 4Gb partition (on another hard drive, not the same where the data (sstables) is stored) ?
A smaller partition will be fine, but you may want to leave a little headroom above 4 GB.
How can I insert huge BLOBs into a MySQL database (InnoDB)?
Fields of type LONGBLOB support data sizes of up to 4 GB according to the MySQL manual. But how does data of such a huge size get into the database?
I tried to use
INSERT INTO table (bindata) VALUES ( LOAD_FILE('c:/tmp/hugefile') );
which fails if the size of hugefile is bigger than about 500 MB. I have set max_allowed_packet to an appropriate size; the value of innodb_buffer_pool_size doesn't seem to have an influence.
My server machine runs Windows Server 2003 and has 2 GB RAM. I'm using MySQL 5.0.74-enterprise-nt.
BLOBs are cached in memory, that's why you will have 3 copies of a BLOB as you are inserting it into a database.
Your 500 MB BLOB occupies 1,500 MB in RAM, which seems to hit your memory limit.
I do not know which client/api you use, but when trying to use blobs from own Java and Objective-C clients it seems, MySQL does not really support streaming of blobs. You need enough memory to hold the whole blob as byte array in ram (server and client side) more than once! Moving to a 64 bit linux helps, but is not desired solution.
MySQL ist not made for BLOB handling (ok for small BOBs :-). It occupies twice or three times the ram to be stored/read in the "BLOB".
You have to use an other database like PostgreSQL to get real BLOB support, sorry.