I'm using Sheets.Spreadsheets.batchUpdate(); to import CSV data from a .csv file to Google Sheets. It takes like 8 seconds to update the import Sheet in my test document, but in production document it takes more than 60 seconds. That's because a lot of formulas are referencing the CSV import Sheet, and Sheets recalculate formulas while the script is running.
I don't know how to manage this bottleneck and don't let sheets recalculate until script is finished. Any ideas?
for (const [txtFileSheet, txtFileId] of list) {
sheetPaste = ss.getSheetByName(`${txtFileSheet}`) || ss.insertSheet(`${txtFileSheet}`, 200); sh.activate();
sheetId = sheetPaste.getSheetId();
sheetPaste.clearContents();
SpreadsheetApp.flush();
tsvUrl = `https://drive.google.com/uc?id=${txtFileId}&x=.tsv`;
tsvContent = UrlFetchApp.fetch(tsvUrl).getContentText();
resource = {requests: [{pasteData: {data: tsvContent, coordinate: {sheetId: sheetId}, delimiter: '\t'}},]};
Sheets.Spreadsheets.batchUpdate(resource, ss_id);
}
This is the loop that import all the CSV files. I read that the SpreadsheetApp.flush() line could cause the recalculation, but I tried and deleting .flush() cause data not being updated in target sheets.
Related
I want to list all the folderName and their folderID present in a team Drive(more than 3000 folders). I am using speedsheet and running following code in script-
function listFilesInFolder(folderName) {
var sheet = SpreadsheetApp.getActiveSheet();
sheet.appendRow(['Name','File-Id']);
//change the folder ID below to reflect your folder's ID (look in the URL when you're in your folder)
var folder = DriveApp.getFolderById('folder ID');
var contents = folder.getFolders();
var cnt = 0;
var folder;
while (contents.hasNext()) {
var folder = contents.next();
cnt++;
data = [
folder.getName(),
folder.getId(),
];
sheet.appendRow(data);
};
}
But this is getting Error Exceeded maximum execution time which is 6 min by default.
I tried adding triggers from script app, but after triggering it get start from beginning and script still ends after 6min.
How to add a triggers which starts from where it left?
Answer:
The slow part of this script is the repeated call to sheet.appendRow(). You can speed this up by pushing the results to an array and setting the values at the end of the loop, rather than appending a row on each iteration of the while loop.
More Information:
Using the built-in services such as SpreadsheetApp often can be slow when making many changes to a sheet in a short space of time. You can combat this by minimising the number of calls to the built-in Apps Script services as possible, relying on pure JavaScript to do your processing.
Code Change:
function listFilesInFolder(folderName) {
const sheet = SpreadsheetApp.getActiveSheet()
//change the folder ID below to reflect your folder's ID (look in the URL when you're in your folder)
let folder = DriveApp.getFolderById('')
let contents = folder.getFolders()
let cnt = 0
let data = [['Name','File-Id']]
while (contents.hasNext()) {
folder = contents.next()
cnt++
data.push([
folder.getName(),
folder.getId(),
])
}
sheet.insertRows(sheet.getMaxRows(), data.length)
sheet.getRange(2, 1, data.length, 2).setValues(data)
}
Code changes:
data is declared as an array initialised with the header row, as opposed to appending it directly to the sheet
On each iteration of the loop, the current folder's name and ID is appended to the data array as a new row of data.
After all folders have been looped through, the number of rows in the sheet is extended by the number of rows in data so to not hit an out of bounds error
All rows inside data are added to the sheet using setValues().
In my test environment, I had the following set up:
Drive folder containing 3424 folders
Using the appendRow() method inside the while loop, execution took 1105.256 seconds (or 18 minutes)
Using push() with the .setValues() method outside the loop, execution took 4.478 seconds.
References:
Class Range - setValues() | Apps Script | Google Developers
Good morning.
I have a Sheet tab called "Tsv Export" containing 16 columns of data.
I'd like to export all these 16 columns each on a separated tsv file.
If possible, each tsv file should be named as the corresponding exported column number (ex 01.tsv, 02.tsv etc)
The 16 tsv files should be saved in the same drive folder where the Sheets document is located.
Note: the script might support more columns, not just 16, and stop the loop at the first empty column.
Thanks so much
I believe your goal as follows.
You want to retrieve the values from the sheet "Tsv Export" and export each column as the TSV file.
You want to give the value of index + 1 as the filename.
You want to export the file to the same folder with the Spreadsheet.
When the column without no values is retrieved at the loop, you want to stop the script.
You want to achieve this using Google Apps Script.
For this, how about this answer? In this answer, I would like to propose the following flow.
Retrieve the folder.
Retrieve values from sheet of "Tsv Export";
Transpose the retrieved values.
Export each column as the TSV file.
Sample script:
function myFunction() {
const sheetName = "Tsv Export";
// 1. Retrieve the folder.
const ss = SpreadsheetApp.getActiveSpreadsheet();
const folder = DriveApp.getFileById(ss.getId()).getParents().next();
// 2. Retrieve values from sheet of "Tsv Export";
const sheet = ss.getSheetByName(sheetName);
const values = sheet.getDataRange().getValues();
// 3. Transpose the retrieved values.
const converted = values[0].map((_, i) => values.map(r => r[i]));
// 4. Export each column as the TSV file.
for (let i = 0; i < converted.length; i++) {
const v = converted[i];
if (v.every(e => !e)) {
break;
} else {
folder.createFile(`${('0' + (i + 1)).slice(-2)}.tsv`, v.join("\n"), "text/tab-separated-values");
}
}
}
Note:
Please confirm the sheet name again.
When you want to export a column to a row as TSV format, please modify as follows.
From
folder.createFile(`${('0' + (i + 1)).slice(-2)}.tsv`, v.join("\n"), "text/tab-separated-values");
To
folder.createFile(`${('0' + (i + 1)).slice(-2)}.tsv`, v.join("\t"), "text/tab-separated-values");
Please use this script with V8.
References:
getValues()
getParents()
createFile(name, content, mimeType)
I have two files in an Apps Script project. One is a .gs file acting as a "server" and one is a .html file containing JavaScript as per Google's Apps Script guidelines.
Everything has been going swimmingly for the first 40 hours of development on this project. I have the following line of code outside of any function, in between two tags in the .html file:
google.script.run.withSuccessHandler(setSheetData).getSheetData();
Documentation: https://developers.google.com/apps-script/guides/html/reference/run#withSuccessHandler(Function)
According to the documentation, getSheetData() should first execute in the .gs file, and return a value that is then passed into setSheetData which exists in the .html file.
Server file:
function getSheetData() {
var ss = SpreadsheetApp.getActive();
var activeSheet = ss.getActiveSheet();
var sheetName = activeSheet.getName();
var sheetVals = activeSheet.getDataRange().getValues();
return [sheetName, sheetVals];
}
Html file:
function setSheetData(data) {
alert(data);
sheetName = data[0];
sheetData = data[1];
headers = sheetData[0];
document.getElementById('sheetLook').innerHTML = 'Looking at sheet: ' + sheetName;
}
How I know it is a matter of execution speed:
Currently the alert() call just prints out null. The sheet it is drawing from contains 4 rows of data. However, all other things remaining the same, if I simply am looking at a sheet with 0-1 rows of data, it correctly alerts the entire data vals.
Inside of getSheetData() if I add Logger.log(sheetVals) it correctly logs the entire sheet's data regardless of size. The issue is that the successhandler is executing before it has time to evaluate.
Possible cause(s):
Illegal return values in rows 3-4 of data
No active sheet is present.
Solution:
Avoid returning illegal values like Date objects. Or JSON.stringify() them before returning to client.
getSheetByName or number instead of getting it by activeness.
References:
google.script.run § myFunction § return
Is there a way to export cloud SQL data to excel sheet without copying it to Google Spreadsheet using Google Apps Script.
Since there is limitation of Google Spreadsheet of 4,00,000 cells, I am looking to export data directly to Excel sheet rather than copying it to Spreadsheet.
I specifically want to implement it using Google Apps Script.
Yes, you can serve a CSV file, which you can set up to be downloaded. I attached a sample script to show you how it might work.
Warning: Because this is a Comma Separated Values file, you must ensure that your data does not contain commas.
function doGet(e) {
if (parseInt(e.parameter.download) == 1) {
var someData = [{name:"Jerry",
age:27,
married:true},
{name:"Harry",
age:16,
married:false},
{name:"Gary",
age:65,
married:true},
{name:"Larry",
age:41,
married:false}];
var output = "Name,Age,Married?\n";
for (var i in someData) {
output += someData[i].name + ","+someData[i].age + ","+someData[i].married + "\n";
}
return ContentService.createTextOutput(output).downloadAsFile("data.csv");
} else {
var app = UiApp.createApplication();
var anchor = app.createAnchor("Download the CSV File",ScriptApp.getService().getUrl()+"?download=1");
app.add(anchor);
return app;
}
}
Note: Because you are looking for such a large file, this actually might not work. Your script will probably time out. Suppose each entry that you have is 100 bytes, then you will have a 400 MB excel file? That is just large in general.
Could avoid limits of g apps and spreadsheets by using odbc/jdbc directly from excel https://developers.google.com/cloud-sql/docs/external
I'm sorry if this is an obvious question, I'm still pretty new to the API.
I'm using the python drive api library, and trying to download a google spreadsheet as a csv.
When I used files.get, it spat out a file with no downloadUrl, and with no 'text/csv' key in the export links field.
If it's not possible, I can find a workaround, but I'm hoping it is, since it is possible to do manually (file->download_as->csv)
Do I need to use the google document list api?
thanks,
Matt
Update: I have posted another answer that works with the Spreadsheets v4 API.
Old Answer:
The answer from Alain is correct, but you also need to set the gid=parameter to specify which worksheet to export.
For example, if your 'application/pdf' export link is like this:
docs.google.com/feeds/download/spreadsheets/Export?key=<FILE_ID>&exportFormat=pdf
You can just change it to this to download the first worksheet:
docs.google.com/feeds/download/spreadsheets/Export?key<FILE_ID>&exportFormat=csv&gid=0
There is a bit of a problem, though as there is no reliable way to get the gid for a given worksheet through the API and they are not zero based indexes. If you delete a worksheet, that gid does not get reused. You can see the gid in the URL in your browser though, so if your worksheet information is constant you can just get that from there. See http://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=1813 and http://code.google.com/a/google.com/p/apps-api-issues/issues/detail?id=3240 for more info on that problem.
As a lot of other people have pointed out, my original answer is somewhat outdated. So here is my answer updated for v4 of the Google Spreadsheets API. Now there's a way to get the gids, but we can't use the the drive files.export API because it only exports first worksheet in the spreadsheet (even if you specify the gid).
To export all of the worksheets as CSV files, you need to get the gids for the worksheets you want to export using the spreadsheets.get API. That API call returns a bunch of information about the spreadsheet including each of the worksheets. You can get the gid from the properties.sheetId property for each worksheet.
Once you have that, you can just build the same URL that the Sheets uses when you select File->Download As->CSV. You can take the data.spreadsheetUrl value from spreadsheets.get and replace /edit with /export and then add the gid as the parameter. You will also need to include Authorization Bearer <auth token> in the HTTP header in the request.
Here's a python script based on their quickstart example that downloads all of the sheets for the spreadsheet with a specified ID. You need to replace <spreadsheet id> with the ID for a spreadsheet you have access to:
import apiclient.discovery
import httplib2
import oauth2client.file
import oauth2client.tools
import re
import requests
import shutil
import urllib.parse
SCOPES = 'https://www.googleapis.com/auth/drive.readonly'
SPREADSHEET_ID = '<spreadsheet id>'
store = oauth2client.file.Storage('credentials.json')
creds = store.get()
if not creds or creds.invalid:
flow = oauth2client.client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = oauth2client.tools.run_flow(flow, store)
service = apiclient.discovery.build('sheets', 'v4', http=creds.authorize(httplib2.Http()))
result = service.spreadsheets().get(spreadsheetId = SPREADSHEET_ID).execute()
urlParts = urllib.parse.urlparse(result['spreadsheetUrl'])
path = re.sub("\/edit$", '/export', urlParts.path)
urlParts = urlParts._replace(path=path)
headers = {
'Authorization': 'Bearer ' + creds.access_token,
}
for sheet in result['sheets']:
params = {
'id': SPREADSHEET_ID,
'format': 'csv',
'gid': sheet['properties']['sheetId'],
}
queryParams = urllib.parse.urlencode(params)
urlParts = urlParts._replace(query=queryParams)
url = urllib.parse.urlunparse(urlParts)
response = requests.get(url, headers = headers)
filePath = '/tmp/foo-%s.csv' % (+ params['gid'])
with open(filePath, 'wb') as csvFile:
csvFile.write(response.content)
The exportLinks collection doesn't expose the CSV format as this will only export the first worksheet of a spreadsheet. If retrieving the first worksheet as a CSV is the behavior you are looking for, you can build the link manually and set the ?exportFormat= query parameter to ?exportFormat=csv.
Here's an implementation of Alain's suggestion that works for me:
downloadUrl = entry.get('exportLinks')['application/pdf']
# Strip "=pdf" and replace with "=csv"
downloadUrl = downloadUrl[:-4] + "=csv"
resp, content = drive_service._http.request(downloadUrl)
Not sure if it's what the OP needed, but in the new Google Sheets version it seems that it became a little hard to hot link a csv version of your spreadsheet.
In case you are interested in a Google apps script that will export all sheets in a spreadsheet to individual csv files (instead of downloading each one individually), Here you go:
function onOpen() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var csvMenuEntries = [{name: "export as csv files", functionName: "saveAsCSV"}];
ss.addMenu("csv", csvMenuEntries);
};
function saveAsCSV() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheets = ss.getSheets();
// create a folder from the name of the spreadsheet
var folder = DocsList.createFolder(ss.getName().toLowerCase().replace(/ /g,'_') + '_csv_' + new Date().getTime());
for (var i = 0 ; i < sheets.length ; i++) {
var sheet = sheets[i];
// append ".csv" extension to the sheet name
fileName = sheet.getName() + ".csv";
// convert all available sheet data to csv format
var csvFile = convertRangeToCsvFile_(fileName, sheet);
// create a file in the Docs List with the given name and the csv data
folder.createFile(fileName, csvFile);
}
Browser.msgBox('Files are waiting in a folder named ' + folder.getName());
}
function convertRangeToCsvFile_(csvFileName, sheet) {
// get available data range in the spreadsheet
var activeRange = sheet.getDataRange();
try {
var data = activeRange.getValues();
var csvFile = undefined;
// loop through the data in the range and build a string with the csv data
if (data.length > 1) {
var csv = "";
for (var row = 0; row < data.length; row++) {
for (var col = 0; col < data[row].length; col++) {
if (data[row][col].toString().indexOf(",") != -1) {
data[row][col] = "\"" + data[row][col] + "\"";
}
}
// join each row's columns
// add a carriage return to end of each row, except for the last one
if (row < data.length-1) {
csv += data[row].join(",") + "\r\n";
}
else {
csv += data[row];
}
}
csvFile = csv;
}
return csvFile;
}
catch(err) {
Logger.log(err);
Browser.msgBox(err);
}
}
Note: This script uses the DocsList.createFile() method, which is only available for Google Apps accounts.
If you need further explanation, go here: http://drzon.net/export-all-google-sheets-to-csv/
(Jul 2016) This question is phrased correctly, but in essence is a duplicate of another thread (Download a spreadsheet from Google Docs using Python). While some of the previous answers to this question below may still work (although answers are in JS/Apps Script not Python), a new Drive API version (v3) and new Sheets API version (v4) make them slightly outdated although the previous versions of both have not been deprecated (yet). Modern Google API access occurs using API keys or OAuth2 authorization, primarily with the Google APIs Client Libraries, including the one for Python.
To perform the task requested in/by the OP, you would perhaps query for specific Sheets to download, then perform the actual export(s) with the Drive API. Since this is likely a common operation, I wrote a blogpost sharing a code snippet that does this for you. If you wish to pursue exporting further, I've got another pair of posts along with a video that outlines how to upload files to and download files from Google Drive.
Note that there is also a Google Sheets API, but it's primarily for spreadsheet-oriented operations, i.e., inserting data, reading spreadsheet rows, cell formatting, creating charts, adding pivot tables, etc., not file-based requests like exporting where the Drive API is the correct one to use.
Note, as of April 2015 DocsList was depreciated, and has been replaced by DriveApp. Many of the DriveApp methods are identical to DocsList. So, in many cases, you can simply replace DocsList with DriveApp. So replace DocsList.createFile() with DriveApp.createFile()
How to update DocsList to DriveApp in my code