how to pull 300-400k data stored in mysql to mongoDB - mysql

Question

I was developing a spring boot api to pull data from remote mysql database table. This table contains 300k - 400k data daily. We need to migrate this data to mongoDB now. I tried GridFS technique to store collected json file to mongoDB. I was able to this on local machine. But when I tried this scenario with live server, the JVM threw error :
2018-12-18 17:59:26.206 ERROR 4780 --- [r.BlockPoller-1] o.a.tomcat.util.net.NioBlockingSelector :
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.ArrayList.iterator(ArrayList.java:840) ~[na:1.8.0_181]
at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496) ~[na:1.8.0_181]
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) ~[na:1.8.0_181]
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) ~[na:1.8.0_181]
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) ~[na:1.8.0_181]
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:339) ~[tomcat-embed-core-8.5.14.jar:8.5.14]
2018-12-18 17:59:27.865 ERROR 4780 --- [nio-8083-exec-1] o.a.c.c.C.[.[.[.[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [/datapuller/v1] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded] with root cause
java.lang.OutOfMemoryError: GC overhead limit exceeded
I tried to exceed the heap size with -Xmx3048m by opening java utility from control panel.. but same result. What should I do next to resolve this issue. I have not posted code here because I hope it was all fine as it was running OK on local machine with 60k to 70k record data.

Answer 1

The most performant way is always to bypass all those abstraction.
Since you are not locked to Sprint Boot, I will suggest you to dump the data as csv from mysql, either via mysqldump, or
echo 'SELECT * FROM table' | mysql -h your_host -u user -p -B <db_schema>
Then you can import this csv to MongoDB.
mongoimport --host=127.0.0.1 -d database_name -c collection_name --type csv --file csv_location --headerline
https://docs.mongodb.com/manual/reference/program/mongoimport/