Error uploading .xls file and converting to Google Spreadsheet via GAS - google-apps-script

Prior Research
Please do not close this question as a duplicate because my question deals with how to resolve the specific error message I am receiving and not the general question of whether my objective is achievable or not — as some other related questions, yielded by my research and below detailed, have asked.
Related questions and why they do not apply here
This question, asked 7/27/2012, does not apply because it: (1) is too old (after 10 months, new solutions/methods might exist) and (2) does not deal with the specific error message I am experiencing.
This question, asked 10/12/2012, fails to apply for similar reasons.
My below code was copied from here which was forked from here. These are presumably, working solutions because they have been referenced as such from other question/answer exchanges here on Stack Overflow.
Objective
Programmatically, I am trying to:
Search my email inbox.
Find Excel (.xls) file attachments.
Upload those .xls file attachments to Google Drive.
While uploading, convert the .xls files into a Google Spreadsheet file format.
Problem
When I execute processInbox() (code shown at the bottom of this question), it fails and I get the error message shown below.
Error Message
Request failed for returned code 403.
Server response:
{
"error":{
"errors":[
{
"domain":"usageLimits",
"reason":"accessNotConfigured",
"message":"AccessNotConfigured"
}
],
"code":403,
"message":"AccessNotConfigured"
}
}
(line 13, file "DriveUpload")
Question
What am I doing wrong? And how can I fix it?
For example, do I need to do something special in my API console relative to setting up my project to, say, access Google Drive or something? What am I missing?
Note: I have not yet successfully implemented oAuth in any of my applications, yet.
Error Source
Line 13
(This is the code line referenced by the error message.)
var uploadRequest = UrlFetchApp.fetch("https://www.googleapis.com/upload/drive/v2/files/?uploadType=media&convert=true&key="+key, params); // convert=true convert xls to google spreadsheet
Code
The complete body of code I am working with is shown below for your reference. I extracted the error-triggering, “line 13,” and highlighted it above to help us focus on the proximate cause of the problem.
DriveUpload.js
function uploadXls(file) {
authorize();
var key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; // <-- developer key
var metadata = { title: file.getName() }
var params = {method:"post",
oAuthServiceName: "drive",
oAuthUseToken: "always",
contentType: "application/vnd.ms-excel",
contentLength: file.getBytes().length,
payload: file.getBytes()
};
// convert=true convert xls to google spreadsheet
var uploadRequest = UrlFetchApp.fetch("https://www.googleapis.com/upload/drive/v2/files/?uploadType=media&convert=true&key="+key, params);
var uploadResponse = Utilities.jsonParse(uploadRequest.getContentText());
var params = {method:"put",
oAuthServiceName: "drive",
oAuthUseToken: "always",
contentType: "application/json",
payload: Utilities.jsonStringify(metadata)
};
var metaRequest = UrlFetchApp.fetch("https://www.googleapis.com/drive/v2/files/"+uploadResponse.id+"?key="+key, params)
return DocsList.getFileById(uploadResponse.id);
}
function authorize() {
var oauthConfig = UrlFetchApp.addOAuthService("drive");
var scope = "https://www.googleapis.com/auth/drive";
oauthConfig.setConsumerKey("anonymous");
oauthConfig.setConsumerSecret("anonymous");
oauthConfig.setRequestTokenUrl("https://www.google.com/accounts/OAuthGetRequestToken?scope="+scope);
oauthConfig.setAuthorizationUrl("https://accounts.google.com/OAuthAuthorizeToken");
oauthConfig.setAccessTokenUrl("https://www.google.com/accounts/OAuthGetAccessToken");
}
function processInbox() {
// get all threads in inbox
var threads = GmailApp.getInboxThreads();
for (var i = 0; i < threads.length; i++) {
// get all messages in a given thread
var messages = threads[i].getMessages();
// iterate over each message
for (var j = 0; j < messages.length; j++) {
// log message subject
var subject = messages[j].getSubject()
//Logger.log(subject);
if ( subject == "with xls attach" ){
Logger.log(messages[j].getSubject());
var attach = messages[j].getAttachments()[0];
var name = attach.getName();
var type = attach.getContentType();
//var data = attach.getDataAsString();
Logger.log( name + " " + type + " " );
var file = uploadXls(attach);
SpreadsheetApp.open(file);
}
}
}
};

Drive API is already built in GAS: https://developers.google.com/apps-script/reference/drive/
Use DriveApp and your problems go away ;-)

This maybe a temp solution
Step 1: Use a Google Form to Collect Data to a Google spreadsheet
Step 2: Add the Zoho Sheet App to your Google Drive
In a Zoho Sheet
Goto
Data Menu
»Link External Data
Select either
CSV
RSS/Atom Feed
or HTML Page
You can schedule it to update at specific time intervals
What I like is the VBA and Macros in Zoho
You can also do Pivot Charts and Tables
You can copy and paste Excel VBA into Zoho !
I have an Unpivot VBA that I will run on my Tabular dataset
before I can do a PivotChart
It is hard to beat all the functionality of Excel and I often fall back on familiar tools !
If I hear of anything I will post it
Good luck

Related

Google Apps Script for Yahoo Finance Returns Empty Cell

A previously working solution that was resolved here by #tanaike suddenly returns an empty cell upon execution. I don't get an error message and in the google apps scripts edit page I get "Notice Execution completed".
It looks like it's working in the background but having trouble returning a value to the cell, my guess would be something wrong with the last line that may resolve it?
function pressReleases(code) {
var url = 'https://finance.yahoo.com/quote/' + code + '/press-releases'
var html = UrlFetchApp.fetch(url).getContentText().match(/root.App.main = ([\s\S\w]+?);\n/);
if (!html || html.length == 1) return;
var obj = JSON.parse(html[1].trim());
// --- I modified the below script.
const { _cs, _cr } = obj;
if (!_cs || !_cr) return;
const key = CryptoJS.algo.PBKDF2.create({ keySize: 8 }).compute(_cs, JSON.parse(_cr)).toString();
const obj2 = JSON.parse(CryptoJS.enc.Utf8.stringify(CryptoJS.AES.decrypt(obj.context.dispatcher.stores, key)));
var res = obj2.StreamStore.streams["YFINANCE:" + code + ".mega"].data.stream_items[0].title;
// ---
return res || "No value";
}
The CryptoJS code saved as a script in google apps script is here
When I tested this script, at if (!_cs || !_cr) return;, I confirmed that the values of _cs and _cr are undefined. From this result, I understood that recently, the specification of the key for decrypting the data has been changed at the server side. When I saw this thread, I confirmed the same situation. In the thread, I noticed that, in the current stage, the key can be simply retrieved from the HTML data. So, as with the current script, how about the following modification?
Usage:
1. Get crypto-js.
Please access https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js. And, copy and paste the script to the script editor of Google Apps Script, and save the script.
2. Modify script.
function pressReleases(code) {
var url = 'https://finance.yahoo.com/quote/' + code + '/press-releases'
var html = UrlFetchApp.fetch(url).getContentText().match(/root.App.main = ([\s\S\w]+?);\n/);
if (!html || html.length == 1) return;
var obj = JSON.parse(html[1].trim());
var key = Object.entries(obj).find(([k]) => !["context", "plugins"].includes(k))[1];
if (!key) return;
const obj2 = JSON.parse(CryptoJS.enc.Utf8.stringify(CryptoJS.AES.decrypt(obj.context.dispatcher.stores, key)));
var res = obj2.StreamStore.streams["YFINANCE:" + code + ".mega"].data.stream_items[0].title;
// console.log(res); // Check the value in the log.
return res || "No value";
}
When this script is run with code = "PGEN", the value of Precigen Provides Pipeline and Corporate Updates at the 41st Annual J.P. Morgan Healthcare Conference is obtained.
Note:
If you want to load crypto-js directly, you can also use the following script. But, in this case, the process cost becomes higher than that of the above flow. Please be careful about this.
const cdnjs = "https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js";
eval(UrlFetchApp.fetch(cdnjs).getContentText());
I can confirm that this method can be used for the current situation (January 14, 2023). But, when the specification in the data and HTML is changed in the future update on the server side, this script might not be able to be used. Please be careful about this.
Reference:
crypto-js

Google Apps Script error - multiple rows from Google Sheets but only one file in Google Drive - "iterator has reached the end"

I have a script that pulls fields from a Google Sheet and inserts them into an email template and sends the emails off. That works fine.
I recently wanted to include a PDF as an attachment to the emails. It would be the same PDF for every email. I uploaded the PDF into Google Drive. When I run the script, the first email is sent off fine with the attachment but the following emails are not sent because I encounter this error: "Cannot retrieve the next object: iterator has reached the end"
Pretty sure it has to deal with the attachment/file and me not handling the iteration correctly. Can someone help? Below is the code:
function send2Email()
{
var filename= 'even_overview2020.pdf';
var file = DriveApp.getFilesByName(filename);
var spread =SpreadsheetApp.getActiveSpreadsheet();
var contactSheet =spread.getSheetByName(contactSheetName);
var bodySheet =spread.getSheetByName(templateSheetName);
var contactData =contactSheet.getDataRange().getValues();
var bodyData =bodySheet.getDataRange().getValues();
var fname,company,sign,template,email,subject,body,sender,file;
for (var i =1;i<contactData.length;i++)
{
contactData[i][statusCol-1]="";
}
contactSheet.getDataRange().setValues(contactData);
for (var i =1;i<contactData.length;i++)
{
fname=trim_(contactData[i][fnameCol-1]);
company=trim_(contactData[i][companyCol-1]);
sign=trim_(contactData[i][signCol-1]);
template=trim_(contactData[i][templateCol-1]);
email=trim_(contactData[i][emailCol-1]);
sender=trim_(contactData[i][senderCol-1]);
Logger.log(email);
for(var j=1;j<bodyData.length;j++)
{
if(trim_(bodyData[j][tempRefCol-1]).toUpperCase()==String(template).toUpperCase())
{
body=bodyData[j][bodyCol-1];
subject=bodyData[j][subjectCol-1];
}
}
Logger.log(j+","+email+','+body+','+subject);
body=body.replace(/\n/g,"<br>");
body=body.replace("(w)",sign).replace("(x)",fname).replace("(y)",company).replace("(s)",sender.split(" ")[0]);
Logger.log(email+','+body+','+subject);
MailApp.sendEmail({to:email,subject:subject,name:sender,htmlBody:body,attachments: [file.next().getAs(MimeType.PDF)]});
contactSheet.getRange(i+1, statusCol).setValue('Y');
}
}
How about this modification?
Modification point:
In your script, attachments: [file.next().getAs(MimeType.PDF)]} is used in the loop. By this, the 1st loop works by file.next(). But after 2nd loop, an error occurs at file.next() because the file of filename is one file in Google Drive. I think that this is the reason of your issue.
In order to avoid this issue, how about the following modification?
Modified script:
From:
var file = DriveApp.getFilesByName(filename);
To:
var files = DriveApp.getFilesByName(filename);
var file;
if (files.hasNext()) {
file = files.next().getAs(MimeType.PDF);
} else {
throw new Error("No file");
}
And also, please modify as follows.
From:
MailApp.sendEmail({to:email,subject:subject,name:sender,htmlBody:body,attachments: [file.next().getAs(MimeType.PDF)]});
To:
MailApp.sendEmail({to:email,subject:subject,name:sender,htmlBody:body,attachments: [file]});
Reference:
Class FileIterator

Google Forms Upload File - using/modifying the response

Within a larger process (not relevant here) I would like to use the native Google Forms "File Upload" Question in a form.
I have been trying to understand how to use the response to :
rename the file uploaded
move/copy/save the file to a specific location
(this will be in the end determined by another response item, but for the moment, lets just call it "myDesitinationFolder"
I can of course, in the responses, in the Google interface, or in a Spreadsheet, see the files (and from the spreadsheet, the URL to the file)
but I would like to do my processing before the file is "saved to Drive" or in any case, during the onFormSubmit()
All i find though is a value [randomalphanumeric] and I am unable to find the documentation that explains what this [element] is. (if it is an object - but the square [] would seems to say this is NOT the case)
(Updated after clarification)
the sample code used to get the value comes from Stack.
.
fonction onFormSubmit(e){
var form = FormApp.getActiveForm()
var formResponses = form.getResponses();
for (var i = 0; i < formResponses.length; i++) {
var formResponse = formResponses[i];
var itemResponses = formResponse.getItemResponses();
for (var j = 0; j < itemResponses.length; j++) {
var itemResponse = itemResponses[j];
Logger.log('Response #%s to the question "%s" was "%s"',
(i + 1).toString(),
itemResponse.getItem().getTitle(),
itemResponse.getResponse());
console.log('Response #%s to the question "%s" was "%s"',
(i + 1).toString(),
itemResponse.getItem().getTitle(),
itemResponse.getResponse());
}
}
}
OUTPUT :
Response #19 to the question "fileUpload" was "[1OQeNOTREALCODE5SoAaPiw4s2M-cfRDJ]"
(NOTREALCODE added by me to anonymise response)
my Question : is that a file object? can i access the "save" process to change the destination and the name before hand?
or would my best option be to do a post submit process on an updated sheet (where i have the file URL etc..) (I really dont want to go that way, I would prefer to have it all within the form (I don't need the sheet otherwise)
thanks
When a file upload form is submitted, Google Form response stores the ID of the file in Google Drive. You can use the DriveApp service to access the uploaded file by ID and rename it, move it or copy to another folder.
function renameFile(id) {
var file = DriveApp.getFileById(id);
file.setName("Some name here");
}

How can I automatically download a .csv file from a hyperlink in a GMAIL message and add its contents to a Google Spreadsheet

I receive an email with a hyperlink that when clicked starts a download of a csv file to my Gmail account. It's not an actual attachment. When I receive this email (which has a unique subject line), I need a way to automatically add the contents of the downloaded .csv
Trigger:
An email with a specific subject line is received to my gmail account
Action 1:
Download a .csv file from a hyperlink within the body of the email
Action 2:
Add the contents of the .csv file to a Google Sheet file
I need an already built service that does this or suggestions on how to approach it.
If I can get this Google script to run, I should be able to find a working solution. The problem is the script keeps giving me errors.
function downloadFile(fileURL,folder) {
var fileName = "";
var fileSize = 0;
var response = UrlFetchApp.fetch(fileURL, {muteHttpExceptions: true});
var rc = response.getResponseCode();
if (rc == 200) {
var fileBlob = response.getBlob()
var folder = DocsList.getFolder(folder);
if (folder != null) {
var file = folder.createFile(fileBlob);
fileName = file.getName();
fileSize = file.getSize();
}
}
var fileInfo = { "rc":rc, "fileName":fileName, "fileSize":fileSize };
return fileInfo;
}
This is something I recently tackled at work, fully automating data pulls from my emails to a database. I am not going to write a solution for you, but I will provide you with the information and links you need to do it yourself.
Note: Your question is very broad, and covers a large range of different problems, each of which should be tackled one at a time with their own question (Many of which already have multiple answers on StackOverflow). This is a process to follow with linked documentation, and a couple code snippets so you can do it yourself and tackle each problem along the way.
The Proposed Process:
Open the email with the GmailApp Service
Extract the link via the script below
Get the CSV from the link via the code linked below. This utilizes UrlFetchAp, the Blob datatype, and the parseCsv utility (which you have to escape commas first, because it's buggy)
Modify the contents of the resulting array to your liking
Use the SpreadsheetApp Service to open a spreadsheet and get a range
Set the values of that range to your array of data.
Extract href link from email (assumes only 1 link):
//Retrieves a URL from a HTML string from an href. Only applicable if there is only one link in the string
function GetHrefURLsFromString(string){
var href = string.match(/href="([^"]*)/)[1];
if(href){
return href;
} else {
throw "No URL Found"
}
}
Extract CSV from link:
//Gets a CSV from a provided link, and parses it.
function GetCSVFromLink(link){
var urlData = UrlFetchApp.fetch(link);
if(urlData.getBlob().getContentType() == 'csv'){
var stringData = urlData.getContentText();
var escapedStringData = stringData.replace(/(?=["'])(?:"[^"\\]*(?:\\[\s\S][^"\\]*)*"|'[^'\\]\r\n(?:\\[\s\S][^'\\]\r\n)*')/g, '\r\n');
var CSV = Utilities.parseCsv(escapedStringData);
return CSV;
}
Logger.log('DataType Not CSV')
return null;
}

Google Script UrlFetchApp.fetch returns a 404 eventhough the page exist

I am sending automated report with Google spreadsheet and Google script.
So far it was working perfectly. But somehow when I try to create a new report to be emailed, the function "UrlFetchApp.fetch" return a 404. The same situation happened when I tried copying the old report.
The line with "UrlFetchApp.fetch" give me this error:
Request failed for https://docs.google.com/spreadsheets/d/1qm_bCKn4MbLKy7AIuIeu7bTZcTk8ObYBln0GAxwfsX8/pub?gid=195635557&single=true&output=pdf returned code 404. Truncated server response
It seems that I am not the only one having the issue but I cannot find any solution to it.
Here is the code:
function emailSpreadsheetAsCSV() {
var ss = SpreadsheetApp.openById("1qm_bCKn4MbLKy7AIuIeu7bTZcTk8ObYBln0GAxwfsX8");
var url = ss.getUrl();
url = url.replace(/edit$/,'');
var token = ScriptApp.getOAuthToken();
var sheets = ss.getSheets();
//make an empty array to hold your fetched blobs
var blobs = [];
for (var i=0; i<sheets.length; i++) {
var response = UrlFetchApp.fetch("https://docs.google.com/spreadsheets/d/1qm_bCKn4MbLKy7AIuIeu7bTZcTk8ObYBln0GAxwfsX8/pub?gid=195635557&single=true&output=pdf", {
headers: {
'Authorization': 'Bearer ' + token
},
'muteHttpExceptions': false
});
//convert the response to a blob and store in our array
blobs[i] = response.getBlob().setName(sheets[i].getName() + '.csv');
}
//create new blob that is a zip file containing our blob array
var zipBlob = Utilities.zip(blobs).setName(ss.getName() + '.zip');
return blobs[0];
}
Thanks a lot for your help.
Aymeric.
I've gotten this problem too and after some research, Spence Easton (question 34573295) solved my issue. I simply added this bogus call to drive so access is granted, and so the script can now get to your file.
Add this near the top before trying to grab the url:
var bogus = DriveApp.getRootFolder();
Now it runs fine with no 404 errors.
Remove the headers from the request:
var response = UrlFetchApp.fetch("https://docs.google.com/spreadsheets/d/1qm_bCKn4MbLKy7AIuIeu7bTZcTk8ObYBln0GAxwfsX8/pub?gid=195635557&single=true&output=pdf", {
'muteHttpExceptions': false
});
Check if it works for you.
You use the "pub" url so are you sure the spreadsheet is published on the web ? If you get a 404 it means the page is not published.
If file is published the bearer is not necessary as it is public now.
After I don't really understand your code because you iterate all the sheets of your file
for (var i=0; i<sheets.length; i++)
but the param "gid" (pub?gid=195635557&single=true&output=pdf) for the url in your urlfetch is fixed ?
Second point you get the file as pdf and after you create a csv ?
Why not get the file as csv directly : https://docs.google.com/spreadsheets/export?id=TheIdOfTheFile&exportFormat=csv
By adapting this code you can get the csv directly, see : https://stackoverflow.com/a/28503601/3556215
Take care you will get the first page and not others.
Stéphane
Try with:
var response = UrlFetchApp.fetch(url, options);
var result = JSON.parse(res.getContentText());