How can I create a parquet file bigger than node's assigned memory? - apache-drill

Question

I'm trying to create a parquet file from a table stored in mysql. The source contains millions of rows and I get a GC Overhead limit exception after a couple of minutes.
Can apache drill be configured in a way that allows operations to use disk temporarily in case there is no more RAM available?
This were my steps before getting the error:
Put the mysql jdbc connector inside jars/3rdparty
Execute sqlline.bat -u "jdbc:drill:zk=local"
Navigate to http://localhost:8047/storage
Configure a new storage pluggin to connect to mysql
Navigate to http://localhost:8047/query and execute the following queries
ALTER SESSION SET `store.format` = 'parquet';
ALTER SESSION SET `store.parquet.compression` = 'snappy';
create table dfs.tmp.`bigtable.parquet` as (select * from mysql.schema.bigtable)
Then I get the error and the aplication ends:
Node ran out of Heap memory, exiting.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2149)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1956)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3308)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:463)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3032)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2280)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2673)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2546)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2504)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1370)
at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
at org.apache.drill.exec.store.jdbc.JdbcRecordReader.setup(JdbcRecordReader.java:177)
at org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:101)
at org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:128)
at org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:40)
at org.apache.drill.exec.store.jdbc.JdbcBatchCreator.getBatch(JdbcBatchCreator.java:33)
at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:151)
at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:131)
at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:174)
at org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:105)
at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:79)
at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:230)
at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Answer 1

Check drill-env.sh located in <drill_installation_directory>/conf
By default values are:
DRILL_MAX_DIRECT_MEMORY="8G"
DRILL_HEAP="4G"
The default memory for a Drillbit is 8G, but Drill prefers 16G or more depending on the workload.
If you have sufficient RAM you can configure it as 16G
You can read in detail in Drill's documentation.