Uploading 40k+ rows from SQL to Google Sheet - google-apps-script

Question

I am trying to populate from SQL to Google Sheets a recordset of 40k rows. Unfortunately the script continues to exceed the maximum amount of time allowed (6 mins?). Here is the code used:
var qry = "select custid from customers ";
var con = Jdbc.getConnection(connection,user,pw);
var stmt = con.createStatement();
var rs = stmt.executeQuery(qry);
dataRange = sheet.getRange("a2")
while(rs.next()) {
dataRange.setValue(rs.getString(1));
dataRange = dataRange.offset(1,0);
}
rs.close();
stmt.close();
con.close();
Is there a way to do this more efficiently? Is there a similar function to copyFromRecordset in Excel? (This sample code only shows one column value. I get less rows returned the more columns I add to the query.)
If network speed is factor, we are on a 50mb down / 10mb up connection.
Thanks in advance for your help!

Answer 1

Assuming that the timeout is happening on the database side, then batching will help. Instead of getting all rows in a single query, get the first few hundred or few thousand. When you're done with the query, close the statement and reopen it with new parameters to get the next few hundred or few thousand. You can keep the same connection for all the queries.
Exactly how to do that DB paging depending on what your SQL database is.
Another option is to cache the data in memory, close the DB connection, then update the spreadsheet.

Answer 2

I never used jdbc on apps script but your script for each loop add data to the sheet this is time consuming. Best way to do that is to build an array and after you push all data in the array you push to spreadsheet.
var qry = "select custid from customers ";
var con = Jdbc.getConnection(connection,user,pw);
var stmt = con.createStatement();
var rs = stmt.executeQuery(qry);
dataRange = sheet.getRange("a2")
var tab = [];
while(rs.next()) {
tab.push([rs.getString(1)]);
//dataRange = dataRange.offset(1,0);
}
sheet.getRange(1, 2, tab.length, tab[0].length).setValues(tab);
rs.close();
stmt.close();
con.close();
This will make you save lot of time in the script but not sure 40 000 rows will fit the 6 minutes. Don't know performance of the jdbc service.
Stéphane