Prevent MySQL query from timing out in Apps Script - mysql

Question

Our client's online shop uses Opencart, for marketing purposes they wanted to sync Opencart orders automatically with Google Sheets. I modified a script in google sheets to extract customer orders from the past year (based on https://www.transpacific-software.com/blog/opencart-to-google-docs-pull-and-sync-data-auto-through-scripting and https://gist.github.com/pradeepbheron/e6129c9fd9bc74e814d0)
The SQL query is:
SELECT order_id, firstname AS first_name, lastname AS last_name,
email, date_added AS order_date, CONCAT("£", FORMAT(total,2))
order_value, payment_address_1 AS billing_address_1,
payment_address_2 AS billing_address_2, payment_city AS
billing_city, payment_postcode AS billing_postcode,
payment_country AS billing_country, items_purchased FROM
(
SELECT order_id, firstname, lastname, email, date_added,
total, payment_address_1, payment_address_2, payment_city,
payment_postcode, payment_country
FROM ocbw_order
GROUP BY order_id
) AS A
JOIN (
SELECT order_id AS product_order_id, GROUP_CONCAT(name
SEPARATOR ", ") AS items_purchased
FROM ocbw_order_product
GROUP BY product_order_id
) AS B
ON A.order_id=B.product_order_id
WHERE date_added >= DATE_SUB(NOW(),INTERVAL 1 YEAR)
AND firstname != ''
It runs fine in phpMyAdmin but Google Script Editor generates an "Exceeded maximum execution time" error.
It looks like there are 7357 rows (exported from myPhpAdmin). Is there a better way to write the query? Also I am trying to rename the column headers but can only two works, i.e:
GROUP_CONCAT(name SEPARATOR ", ") AS items_purchased
and
CONCAT("£", FORMAT(total,2)) order_value
Any thoughts
QUICK UPDATE: Fri Nov 9 11:54:03 2018
1) As request by #ThisGuyHasTwoThumbs, here is a screenshot of the explain table result
2) I looked into the Best Practices doc mentioned by #bcperth. I tried to rewrite the google sheets script but ran into issues.
Here is the amended script.
function p1MySQLFetchData() {
// Change it as per your database credentials
var conn =
Jdbc.getConnection('jdbc:mysql://[dbHostIp]:3306/[dbName]',
'[dbUsername]', '[dbPassword]');
var stmt = conn.createStatement();
var start = new Date(); // Get script starting time
//change table name as per your database structure
var rs = stmt.executeQuery('[sqlQuery]');
// It sets the limit of the
// maximum nuber of rows in a ResultSet object
// Returns the currently active spreadsheet
var doc = SpreadsheetApp.getActiveSpreadsheet();
var cell = doc.getRange('a1');
var row = 0;
// Mysql table column name count.
var getCount = rs.getMetaData().getColumnCount();
// Create array to hold mysql data
var tempArray = [];
// get row and column count for later
var colCount = getCount;
// ATTEMPT TO GET ROW COUNT 1
//rs.last();
//var rowCount = rs.getRow();
//rs.beforeFirst(); // resets rs cursor
//Logger.log(RowCount);
// DOESN'T WORK! result => ReferenceError: "RowCount" is not
// defined. (line 28, file "Code")
// ATTEMPT TO GET ROW COUNT 2
//var rsCount = stmt.executeQuery('SELECT COUNT(*) AS rowcount FROM
//[sqlQuery]);
// It sets the limit of the maximum number of rows in a ResultSet
// object
//rsCount.next();
//var rowCount = rsCount.getString("rowcount");
//Logger.log(rowCount);
// DOESN'T WORK! result => 0
// Build TempArray using MySQL data
for (var i = 0; i < getCount; i++){
tempArray[0][i] = rs.getMetaData().getColumnName(i+1);
// DOESNT WORK => ERROR
// TypeError: Cannot set property "0.0" of undefined to "order_id".
// (line 39, file "Code")
//Logger.log(rs.getMetaData().getColumnName(i+1));
}
var row = 1;
while (rs.next()) {
for (var col = 0; col < rs.getMetaData().getColumnCount();
col++) {
tempArray[row][col] = rs.getString(col + 1);
//Logger.log(rs.getString(col + 1));
}
row++;
}
// BELOW DOESNT AS I CANT GET A ROW COUNT (SEE ABOVE)
// Fill Spreadsheet from tempArray
//for (var row = 0; row < rowCount; row++) {
//for (var col = 0; col < colCount; col++) {
//cell.offset(row, col).setValue(tempArray[row][col + 1]);
//}
// }
rs.close();
stmt.close();
conn.close();
var end = new Date(); // Get script ending time
Logger.log('Time elapsed: ' + (end.getTime() - start.getTime()));
// To generate script log. To view log click on View -> Logs.
}
But as you can see from the comments, I get loads of errors. Am not sure what to do next.
UPDATE Fri Nov 30 15:26:14 2018
In answer to #Tedinoz comment below.
PhpMyAdmin generates 6862 results and the query took 13.3074 seconds.
When I ran the script in Googles Script Editor, it took around 2 minutes 30 to complete and only pulls 6348 records (92.5%). The records stop after 3rd October 2018.
Regarding your suggestions:
1) I tried running a modified script, setting the INTERVAL to:
1 MONTH => I get no results in Google (it should be 529)
2 MONTH => I get 14 results (should be 1029)
3 MONTH => I get 299 results (should be 1669).
They all took about 4-7 second in myPhpAdmin vs 5 - 20 seconds for Google Script Editor
2) Do you mean exporting a csv from phpMyAdmin and importing to Google Sheets? Well I did that and it works fine.
Another thing I have noticed is that the order_id's in Google Sheets don't match that from phpMyAdmin. Weird.