jdbc sqlserver Exception: The query has timed out [duplicate] - google-apps-script

I am trying to fetch data from mySQL database on Google Cloud SQL using JDBC from Google Apps Script. However, I got this error:
Exception: Statement cancelled due to timeout or client request
I can fetch some other data successfully. However, some data I can't.
I execute one of the successful queries and one of the unsuccessful queries on mySQL workbench. I can execute the unsuccessful query with no problem on mySQL workbench.
I compared the durations.
Duration / Fetch
-------------------------------------------
Successful query: 0.140 sec / 0.016 sec
Unsuccessful query: 0.406 sec / 0.047 sec
The unsuccessful query seems to take longer. So, I set query timeout with:
stmt.setQueryTimeout(0);
intending to set no timeout (when the value is set to zero it means that the execution has no timeout limit). Then, I executed it on Google Apps Script.
However, it doesn't work and get the same error. Could you tell me a solution for this?

This seems to be a known issue. Star ★ and comment on the issue to get Google developers to prioritise the issue. Until the issue is fixed, you can switch back to rhino runtime.

Update to add 2nd fix
After some trial and error I figured out what solved this for me -- Some queries worked, others returned this error.
Fix 1
The common denominator was that it was the queries that had been converted to the multi-line format by the V8 engine / new editor that had this issue.
As an example, switching to the new editor / V8 converted long text strings to be similar to the following:
var query = "select Street_Number, street_name, street_suffix, street_dir_prefix, postal_code, city, mls, address, unitnumber "
+"as unit, uspsid,latitude,longitude from properties.forsale where status = 'active' and zpid is null and property_type = 'residential'"
This query resulted in the error as described. The fix is changing longer queries to be continuous strings like the following:
var query = "select Street_Number, street_name, street_suffix, street_dir_prefix, postal_code, city, mls, address, unitnumber as unit, uspsid,latitude,longitude from properties.forsale where status = 'active' and zpid is null and property_type = 'residential'"
Fix 2
This one was a bit more frustrating. V8 is not as forgiving when it comes to connections to the database. Previous to V8 It would automatically close any connections that lingered, however, it looks like V8 does not like that. I had originally written my scripts to share as few connections as possible but I noticed there were some that I got this error on and it was the ones where a connection might be 'split.' For example:
var conn = getconnection() //this is a connection function I have written
var date = date || Utilities.formatDate(new Date(), "America/Chicago", "yyyy-MM-dd");
var keyobj = getkeys(undefined,undefined,conn);
conn = keyobj.conn;
var results = sqltojson(query, true, conn);
The function above was causing this error, and I'm assuming it's because the 'conn' variable was being returned from the function but, I'm going to make a wild assumption, that the connection for whatever reason cannot be in two separate functions at once since it was being both returned and continued to be a value in the object 'keyobj' and also as 'conn'. Adding delete keyobj.conn immediately after defining the conn variable did the trick:
var conn = getconnection() //this is a connection function I have written
var date = date || Utilities.formatDate(new Date(), "America/Chicago", "yyyy-MM-dd");
var keyobj = getkeys(undefined,undefined,conn);
conn = keyobj.conn;
delete keyobj.conn;
var results = sqltojson(query, true, conn);
Doing both of these fixes stopped this error and allowed the script to continue without problems.

The issue was the same as mentioned above and it failed on today so I tried to change the old version and it works for me. FYI

#TheMaster: Trying out connection to MySQL, same time out issue, even when I tried the example at https://developers.google.com/apps-script/guides/jdbc > Write 500 rows of data to a table in a single batch.
Even worse, when I reverted to the rhino interface, the result was the same. That crashes my whole development approach! :(
[Edit] FWIW, it seems to me that both Rhino and V8 don't like keeping a connection (or is it the statement?) open long enough for the above prepareStatement to complete.
So I tried inserting the 500 records as per above linked example, using a prepared SQL statement, which worked OK:
var conn = sqlGetConnection();
var start = new Date();
// conn.setAutoCommit(false);
// var stmt = conn.prepareStatement('INSERT INTO entries '
// + '(guestName, content) values (?, ?)');
// for (var i = 0; i < 500; i++) {
// stmt.setString(1, 'Name ' + i);
// stmt.setString(2, 'Hello, world ' + i);
// stmt.addBatch();
// }
// var batch = stmt.executeBatch();
// conn.commit();
var sql = 'INSERT INTO ' +
'entries (guestName, content) ' +
'values ';
for (var i = 0; i < 500; i++) {
var col1 = "'" + 'Name ' + i + "'"; // Note that the strings had to be ecapsulated
var col2 = "'" + 'Hello, world ' + i + "'"; // in quotes to work with this method
sql = sql + '(' + col1 + ', ' + col2 + '),';
}
sql = sql.substr(0,sql.length-1);
var stmt = conn.createStatement();
var response = stmt.executeUpdate(sql); // executeQuery is only for SELECT statements
conn.close();
var end = new Date();
Logger.log('Time elapsed: %sms, response: %s rows', end - start, response);
}

I got this issue when I submit the update statement for the same records in mysql table.
I set the breakpoint before update statement in my program, and I start the 2 process to run this program. so, the first process will update the mysql table correctly and the second process will get this exception later.
you need add the 'for update ' in you select statement. so the second process will got the zero not the exception when you update the record in the transaction.

Related

Unable to connect Apps Script to Cloud SQL using JDBC

So... I have been trying to set up the connection to a Cloud SQL for MySQL database from an Apps Script function, however I am unable to figure out why it is not working. I have been following the documentation for JDBC, however no matter what I do, the connection always fails.
The following is the code I am using:
const connectionName = '<project_id>:<region>:<instance_id>';
const user = '<user>';
const userPwd = '<user_pass>';
const db = '<database>';
const dbUrl = 'jdbc:google:mysql://' + connectionName + '/' + db;
function connect() {
const conn = Jdbc.getCloudSqlConnection(dbUrl, user, userPwd);
let start = new Date();
let stmt = conn.createStatement();
stmt.setMaxRows(1000);
let results = stmt.executeQuery('SELECT * FROM entries');
let numCols = results.getMetaData().getColumnCount();
while (results.next()) {
let rowString = '';
for (let col = 0; col < numCols; col++) {
rowString += results.getString(col + 1) + '\t';
}
Logger.log(rowString);
}
results.close();
stmt.close();
let end = new Date();
Logger.log('Time elapsed: %sms', end - start);
}
When running this code I get the following error:
Every time I run the code, I can see a new line in the mysqlerr logs:
I have the Cloud SQL instance set with the public IP, but I have not whitelisted any networks. The documentation does not mention it as a required step. Also I have set the project number for the GCP project of this Apps Script project, but still I get this error.
EDIT
As requested, I am adding more screenshots about the instance:
I am able to connect to the Cloud SQL instance using the gcloud sql instances connect command, and once inside I am able to query the table I want. (Bear in mind that the table is currently empty, as we are in the early stages of development at this point)
I believe this rules out the possibility of me using the wrong user/password.
As you can see I am using the default value for the max_allowed_packet flag. To be honest I have not set any flags on this instance yet.
About the connection name, I am using the format shared in the code snippet above, and actually I have copy and pasted it from the Cloud Console.
The next screenshot is the summary of the instance's settings:
Thanks in advance! 😁
I work on the team that maintains Google Cloud SQL Connector libraries.
I was able to get your code sample working with my own project, so I don't think your code is the problem. The first thing I would try is double checking the values of user, userPwd, and connectionName. If those are all correct, then I would also ensure that the user you are trying to log in as has access to the database.
As for those errors you're seeing in the logs, this troubleshooting documentation has some suggestions regarding things you can try. Once suggestion would be to increase the max_allowed_packet flag
If you're sure that all of those values are correct, then please follow up to this comment and we can try to debug further.

Google Sheets SQL Queries Timing Out [duplicate]

I am trying to fetch data from mySQL database on Google Cloud SQL using JDBC from Google Apps Script. However, I got this error:
Exception: Statement cancelled due to timeout or client request
I can fetch some other data successfully. However, some data I can't.
I execute one of the successful queries and one of the unsuccessful queries on mySQL workbench. I can execute the unsuccessful query with no problem on mySQL workbench.
I compared the durations.
Duration / Fetch
-------------------------------------------
Successful query: 0.140 sec / 0.016 sec
Unsuccessful query: 0.406 sec / 0.047 sec
The unsuccessful query seems to take longer. So, I set query timeout with:
stmt.setQueryTimeout(0);
intending to set no timeout (when the value is set to zero it means that the execution has no timeout limit). Then, I executed it on Google Apps Script.
However, it doesn't work and get the same error. Could you tell me a solution for this?
This seems to be a known issue. Star ★ and comment on the issue to get Google developers to prioritise the issue. Until the issue is fixed, you can switch back to rhino runtime.
Update to add 2nd fix
After some trial and error I figured out what solved this for me -- Some queries worked, others returned this error.
Fix 1
The common denominator was that it was the queries that had been converted to the multi-line format by the V8 engine / new editor that had this issue.
As an example, switching to the new editor / V8 converted long text strings to be similar to the following:
var query = "select Street_Number, street_name, street_suffix, street_dir_prefix, postal_code, city, mls, address, unitnumber "
+"as unit, uspsid,latitude,longitude from properties.forsale where status = 'active' and zpid is null and property_type = 'residential'"
This query resulted in the error as described. The fix is changing longer queries to be continuous strings like the following:
var query = "select Street_Number, street_name, street_suffix, street_dir_prefix, postal_code, city, mls, address, unitnumber as unit, uspsid,latitude,longitude from properties.forsale where status = 'active' and zpid is null and property_type = 'residential'"
Fix 2
This one was a bit more frustrating. V8 is not as forgiving when it comes to connections to the database. Previous to V8 It would automatically close any connections that lingered, however, it looks like V8 does not like that. I had originally written my scripts to share as few connections as possible but I noticed there were some that I got this error on and it was the ones where a connection might be 'split.' For example:
var conn = getconnection() //this is a connection function I have written
var date = date || Utilities.formatDate(new Date(), "America/Chicago", "yyyy-MM-dd");
var keyobj = getkeys(undefined,undefined,conn);
conn = keyobj.conn;
var results = sqltojson(query, true, conn);
The function above was causing this error, and I'm assuming it's because the 'conn' variable was being returned from the function but, I'm going to make a wild assumption, that the connection for whatever reason cannot be in two separate functions at once since it was being both returned and continued to be a value in the object 'keyobj' and also as 'conn'. Adding delete keyobj.conn immediately after defining the conn variable did the trick:
var conn = getconnection() //this is a connection function I have written
var date = date || Utilities.formatDate(new Date(), "America/Chicago", "yyyy-MM-dd");
var keyobj = getkeys(undefined,undefined,conn);
conn = keyobj.conn;
delete keyobj.conn;
var results = sqltojson(query, true, conn);
Doing both of these fixes stopped this error and allowed the script to continue without problems.
The issue was the same as mentioned above and it failed on today so I tried to change the old version and it works for me. FYI
#TheMaster: Trying out connection to MySQL, same time out issue, even when I tried the example at https://developers.google.com/apps-script/guides/jdbc > Write 500 rows of data to a table in a single batch.
Even worse, when I reverted to the rhino interface, the result was the same. That crashes my whole development approach! :(
[Edit] FWIW, it seems to me that both Rhino and V8 don't like keeping a connection (or is it the statement?) open long enough for the above prepareStatement to complete.
So I tried inserting the 500 records as per above linked example, using a prepared SQL statement, which worked OK:
var conn = sqlGetConnection();
var start = new Date();
// conn.setAutoCommit(false);
// var stmt = conn.prepareStatement('INSERT INTO entries '
// + '(guestName, content) values (?, ?)');
// for (var i = 0; i < 500; i++) {
// stmt.setString(1, 'Name ' + i);
// stmt.setString(2, 'Hello, world ' + i);
// stmt.addBatch();
// }
// var batch = stmt.executeBatch();
// conn.commit();
var sql = 'INSERT INTO ' +
'entries (guestName, content) ' +
'values ';
for (var i = 0; i < 500; i++) {
var col1 = "'" + 'Name ' + i + "'"; // Note that the strings had to be ecapsulated
var col2 = "'" + 'Hello, world ' + i + "'"; // in quotes to work with this method
sql = sql + '(' + col1 + ', ' + col2 + '),';
}
sql = sql.substr(0,sql.length-1);
var stmt = conn.createStatement();
var response = stmt.executeUpdate(sql); // executeQuery is only for SELECT statements
conn.close();
var end = new Date();
Logger.log('Time elapsed: %sms, response: %s rows', end - start, response);
}
I got this issue when I submit the update statement for the same records in mysql table.
I set the breakpoint before update statement in my program, and I start the 2 process to run this program. so, the first process will update the mysql table correctly and the second process will get this exception later.
you need add the 'for update ' in you select statement. so the second process will got the zero not the exception when you update the record in the transaction.

JDBC on Google Apps Script. Exception: Statement cancelled due to timeout or client request

I am trying to fetch data from mySQL database on Google Cloud SQL using JDBC from Google Apps Script. However, I got this error:
Exception: Statement cancelled due to timeout or client request
I can fetch some other data successfully. However, some data I can't.
I execute one of the successful queries and one of the unsuccessful queries on mySQL workbench. I can execute the unsuccessful query with no problem on mySQL workbench.
I compared the durations.
Duration / Fetch
-------------------------------------------
Successful query: 0.140 sec / 0.016 sec
Unsuccessful query: 0.406 sec / 0.047 sec
The unsuccessful query seems to take longer. So, I set query timeout with:
stmt.setQueryTimeout(0);
intending to set no timeout (when the value is set to zero it means that the execution has no timeout limit). Then, I executed it on Google Apps Script.
However, it doesn't work and get the same error. Could you tell me a solution for this?
This seems to be a known issue. Star ★ and comment on the issue to get Google developers to prioritise the issue. Until the issue is fixed, you can switch back to rhino runtime.
Update to add 2nd fix
After some trial and error I figured out what solved this for me -- Some queries worked, others returned this error.
Fix 1
The common denominator was that it was the queries that had been converted to the multi-line format by the V8 engine / new editor that had this issue.
As an example, switching to the new editor / V8 converted long text strings to be similar to the following:
var query = "select Street_Number, street_name, street_suffix, street_dir_prefix, postal_code, city, mls, address, unitnumber "
+"as unit, uspsid,latitude,longitude from properties.forsale where status = 'active' and zpid is null and property_type = 'residential'"
This query resulted in the error as described. The fix is changing longer queries to be continuous strings like the following:
var query = "select Street_Number, street_name, street_suffix, street_dir_prefix, postal_code, city, mls, address, unitnumber as unit, uspsid,latitude,longitude from properties.forsale where status = 'active' and zpid is null and property_type = 'residential'"
Fix 2
This one was a bit more frustrating. V8 is not as forgiving when it comes to connections to the database. Previous to V8 It would automatically close any connections that lingered, however, it looks like V8 does not like that. I had originally written my scripts to share as few connections as possible but I noticed there were some that I got this error on and it was the ones where a connection might be 'split.' For example:
var conn = getconnection() //this is a connection function I have written
var date = date || Utilities.formatDate(new Date(), "America/Chicago", "yyyy-MM-dd");
var keyobj = getkeys(undefined,undefined,conn);
conn = keyobj.conn;
var results = sqltojson(query, true, conn);
The function above was causing this error, and I'm assuming it's because the 'conn' variable was being returned from the function but, I'm going to make a wild assumption, that the connection for whatever reason cannot be in two separate functions at once since it was being both returned and continued to be a value in the object 'keyobj' and also as 'conn'. Adding delete keyobj.conn immediately after defining the conn variable did the trick:
var conn = getconnection() //this is a connection function I have written
var date = date || Utilities.formatDate(new Date(), "America/Chicago", "yyyy-MM-dd");
var keyobj = getkeys(undefined,undefined,conn);
conn = keyobj.conn;
delete keyobj.conn;
var results = sqltojson(query, true, conn);
Doing both of these fixes stopped this error and allowed the script to continue without problems.
The issue was the same as mentioned above and it failed on today so I tried to change the old version and it works for me. FYI
#TheMaster: Trying out connection to MySQL, same time out issue, even when I tried the example at https://developers.google.com/apps-script/guides/jdbc > Write 500 rows of data to a table in a single batch.
Even worse, when I reverted to the rhino interface, the result was the same. That crashes my whole development approach! :(
[Edit] FWIW, it seems to me that both Rhino and V8 don't like keeping a connection (or is it the statement?) open long enough for the above prepareStatement to complete.
So I tried inserting the 500 records as per above linked example, using a prepared SQL statement, which worked OK:
var conn = sqlGetConnection();
var start = new Date();
// conn.setAutoCommit(false);
// var stmt = conn.prepareStatement('INSERT INTO entries '
// + '(guestName, content) values (?, ?)');
// for (var i = 0; i < 500; i++) {
// stmt.setString(1, 'Name ' + i);
// stmt.setString(2, 'Hello, world ' + i);
// stmt.addBatch();
// }
// var batch = stmt.executeBatch();
// conn.commit();
var sql = 'INSERT INTO ' +
'entries (guestName, content) ' +
'values ';
for (var i = 0; i < 500; i++) {
var col1 = "'" + 'Name ' + i + "'"; // Note that the strings had to be ecapsulated
var col2 = "'" + 'Hello, world ' + i + "'"; // in quotes to work with this method
sql = sql + '(' + col1 + ', ' + col2 + '),';
}
sql = sql.substr(0,sql.length-1);
var stmt = conn.createStatement();
var response = stmt.executeUpdate(sql); // executeQuery is only for SELECT statements
conn.close();
var end = new Date();
Logger.log('Time elapsed: %sms, response: %s rows', end - start, response);
}
I got this issue when I submit the update statement for the same records in mysql table.
I set the breakpoint before update statement in my program, and I start the 2 process to run this program. so, the first process will update the mysql table correctly and the second process will get this exception later.
you need add the 'for update ' in you select statement. so the second process will got the zero not the exception when you update the record in the transaction.

Inserting Null values to Mysql database from Sheets using Google Apps Script

I'm currently working on a data ingestion add-on using Google apps script. The main idea is that the users of an application can insert data from sheets to a database. To do so, i'm using the JDBC api that apps script provides
The problem i'm currently having is that when I read a cell from the sheet that is empty apps script uses the type undefined, therefore producing an error a the moment of insertion. How could I do such thing?
My current insert function:
function putData(row, tableName) {
var connectionName = '****';
var user = '****';
var userPwd = '*****';
var db = '******';
var dbUrl = 'jdbc:google:mysql://' + connectionName + '/' + db;
var conn = Jdbc.getCloudSqlConnection(dbUrl, user, userPwd);
var stmt = conn.createStatement();
var data = row
var query = "INSERT INTO "+ db + '.' + tableName +" VALUES (" ;
var i = 0
//The following loop is just to build the query from the rows taken from the sheet
// if the value is a String I add quotation marks
for each (item in row){
if ((typeof item) == 'string'){
if (i == row.length-1){
query += "'" + item + "'";
} else {
query += "'" + item + "',";
}
}else {
if (i == row.length-1){
query += item;
} else {
query += item + ",";
}
}
i++
}
query += ")"
results = stmt.executeUpdate(query)
stmt.close();
conn.close();
}
When I try to insert the word "NULL" in some cases in thinks it is a string and brings out an error on other fields.
When trying to get the data from the Spreadsheet, more precisely from a cell, the value will be automatically parsed to one of these types: Number, Boolean, Date or String.
According to the Google getValues() documentation:
The values may be of type Number, Boolean, Date, or String, depending on the value of the cell. Empty cells are represented by an empty string in the array.
So essentially, the undefined type may be an issue present in the way you pass the row parameter (for example, trying to access cells which are out of bounds).
If you want to solve your issue, you should add an if statement right after the for each (item in row) { line:
if (typeof item == 'undefined')
item = null;
The if statement checks if the row content is of type undefined and if so, it automatically parses it to null. In this way, the content will be of type null and you should be able to insert it into the database.
The recommended way to do what you are doing actually is by using the JDBC Prepared Statements, which are basically precompiled SQL statements, making it easier for you to insert the necessary data. More exactly, you wouldn't have to manually prepare data for the insertion, like you did in the code you provided above. They are also the safer way, making your data less prone to various attacks.
Also, the for each...in statement is a deprecated one and you should consider using something else instead such as the for loop or the while loop.
Furthermore, I suggest you take a look at these links, since they might be of help:
Class JdbcPreparedStatement;
Class Range Apps Script - getValues().

Uploading 40k+ rows from SQL to Google Sheet

I am trying to populate from SQL to Google Sheets a recordset of 40k rows. Unfortunately the script continues to exceed the maximum amount of time allowed (6 mins?). Here is the code used:
var qry = "select custid from customers ";
var con = Jdbc.getConnection(connection,user,pw);
var stmt = con.createStatement();
var rs = stmt.executeQuery(qry);
dataRange = sheet.getRange("a2")
while(rs.next()) {
dataRange.setValue(rs.getString(1));
dataRange = dataRange.offset(1,0);
}
rs.close();
stmt.close();
con.close();
Is there a way to do this more efficiently? Is there a similar function to copyFromRecordset in Excel? (This sample code only shows one column value. I get less rows returned the more columns I add to the query.)
If network speed is factor, we are on a 50mb down / 10mb up connection.
Thanks in advance for your help!
Assuming that the timeout is happening on the database side, then batching will help. Instead of getting all rows in a single query, get the first few hundred or few thousand. When you're done with the query, close the statement and reopen it with new parameters to get the next few hundred or few thousand. You can keep the same connection for all the queries.
Exactly how to do that DB paging depending on what your SQL database is.
Another option is to cache the data in memory, close the DB connection, then update the spreadsheet.
I never used jdbc on apps script but your script for each loop add data to the sheet this is time consuming. Best way to do that is to build an array and after you push all data in the array you push to spreadsheet.
var qry = "select custid from customers ";
var con = Jdbc.getConnection(connection,user,pw);
var stmt = con.createStatement();
var rs = stmt.executeQuery(qry);
dataRange = sheet.getRange("a2")
var tab = [];
while(rs.next()) {
tab.push([rs.getString(1)]);
//dataRange = dataRange.offset(1,0);
}
sheet.getRange(1, 2, tab.length, tab[0].length).setValues(tab);
rs.close();
stmt.close();
con.close();
This will make you save lot of time in the script but not sure 40 000 rows will fit the 6 minutes. Don't know performance of the jdbc service.
Stéphane