UrlFetchApp Service Time Out - google-apps-script

Question

the urlfetch has a limited data allowed so in order to load a large CSV or TSV file (50K) I've created a proxy server (REST)
which makes the request under the hood, map it to list and cache it for 6 hours.
then, I added an endpoint to fetch data based on a range (see signature below)
signature:
var PROXY = 'http://lagoja-services.rcb.co.il/api/adwordsProxy/feed/{base64URL}/{seperator}/{from}/{to}'
when someone reaches the range endpoint, I create a CSV on-the-fly.
the first loading of this large file on the server takes between 60-80 seconds, other followed requests take the content from the cache so it takes only 2 seconds.
the problem I have is that I'm getting a timeout after 60 seconds using the urlfetch built-in class so it Alternately works (sometimes it works and sometimes getting timeout exception)
there is no way of reducing the first load!
it seems that I can't control the client timeout (from some unknown reason)
any other solution for me?
thanks, Roby (-:
hi, I've created a dedicated "test" endpoint for you to reproduce the issue. this endpoint tries to load the massive file every time (no cache)
function main() {
var response = UrlFetchApp.fetch('http://lagoja-services.rcb.co.il/api/adwordsProxy/test');
var status_code = response.getResponseCode();
Logger.log('status code %s', status_code);
}
endpoint:
http://lagoja-services.rcb.co.il/api/adwordsProxy/test
exception:
Timeout: http://lagoja-services.rcb.co.il/api/adwordsProxy/test
hi all,
i found a solution (which i don't like of course)
but i didn't have any better suggestion.
the idea is to ping the server to fetch the file, wait till it loaded into the cache and then request the loaded content... a shitty solution but at least it works lol
api:
[HttpGet]
[Route("api/fire")]
public HttpResponseMessage FireGetCSVFeed(...) {
HostingEnvironment.QueueBackgroundWorkItem(async ct => {
// make a request to the original CSV file - takes 60-80 seconds
});
return Request.CreateResponse(HttpStatusCode.OK, "Working...");
}
adwords script:
function sleep(milliseconds) {
var start = new Date().getTime();
while (new Date().getTime() < start + milliseconds);
}
function main() {
var response = UrlFetchApp.fetch(baseURL + 'fire');
sleep(1000 * 120); // yes i know, i'm blocking everything )-:
response = UrlFetchApp.fetch(baseURL + 'getCSV');
status_code = response.getResponseCode();
Logger.log('status code %s', status_code);
}