I have built a simple custom function in Apps Script using URLFetchApp to get the follower count for TikTok accounts.
function tiktok_fans() {
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
var handle = '#charlidamelio';
var web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
var match_text = raw_data.exec(web_content);
var result = (match_text[2]);
Logger.log(result)
return result
}
The Log comes back with the correct number for followers.
However, when I change the code to;
function tiktok_fans(handle) {
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
//var handle = '#charlidamelio';
var web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
var match_text = raw_data.exec(web_content);
var result = (match_text[2]);
Logger.log(result)
return result
}
and use it in a spreadsheet for example =tiktok_fans(A1), where A1 has #charlidamelio I get an #ERROR response in the cell
TypeError: Cannot read property '2' of null (line 6).
Why does it work in the logs but not in the spreadsheet?
--additional info--
Still getting the same error after testing #Tanaike answer below, "TypeError: Cannot read property '2' of null (line 6)."
Have mapped out manually to see the error, each time the below runs, a different log returns "null". I believe this is to do with the ContentText size/in the cache. I have tried utilising Utilities.sleep() in between functions with no luck, I still get null's.
code
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
//tiktok urls
var qld = UrlFetchApp.fetch('https://www.tiktok.com/#thisisqueensland?lang=en').getContentText();
var nsw = UrlFetchApp.fetch('https://www.tiktok.com/#visitnsw?lang=en').getContentText();
var syd = UrlFetchApp.fetch('https://www.tiktok.com/#sydney?lang=en').getContentText();
var tas = UrlFetchApp.fetch('https://www.tiktok.com/#tasmania?lang=en').getContentText();
var nt = UrlFetchApp.fetch('https://www.tiktok.com/#ntaustralia?lang=en').getContentText();
var nz = UrlFetchApp.fetch('https://www.tiktok.com/#purenz?lang=en').getContentText();
var aus = UrlFetchApp.fetch('https://www.tiktok.com/#australia?lang=en').getContentText();
var vic = UrlFetchApp.fetch('https://www.tiktok.com/#visitmelbourne?lang=en').getContentText();
//find folowers with regex
var match_qld = raw_data.exec(qld);
var match_nsw = raw_data.exec(nsw);
var match_syd = raw_data.exec(syd);
var match_tas = raw_data.exec(tas);
var match_nt = raw_data.exec(nt);
var match_nz = raw_data.exec(nz);
var match_aus = raw_data.exec(aus);
var match_vic = raw_data.exec(vic);
Logger.log(match_qld);
Logger.log(match_nsw);
Logger.log(match_syd);
Logger.log(match_tas);
Logger.log(match_nt);
Logger.log(match_nz);
Logger.log(match_aus);
Logger.log(match_vic);
Issue:
From your situation, I remembered that the request of UrlFetchApp with the custom function is different from the request of UrlFetchApp with the script editor. So I thought that the reason for your issue might be related to this thread. https://stackoverflow.com/a/63024816 In your situation, your situation seems to be the opposite of this thread. But, it is considered that this issue is due to the specification of the site.
In order to check this difference, I checked the file size of the retrieved HTML data.
The file size of HTML data retrieved by UrlFetchApp executing with the script editor is 518k bytes.
The file size of HTML data retrieved by UrlFetchApp executing with the custom function is 9k bytes.
It seems that the request of UrlFetchApp executing with the custom function is the same as that of UrlFetchApp executing withWeb Apps. The data of 9k bytes are retrieved by using this.
From the above result, it is found that the retrieved HTML is different between the script editor and the custom function. Namely, the HTML data retrieved by the custom function doesn't include the regex of ("followerCount":)([0-9]+). By this, such an error occurs. I thought that this might be the reason for your issue.
Workaround:
When I tested your situation with Web Apps and triggers, the same issue occurs. By this, in the current stage, I thought that the method for automatically executing the script might not be able to be used. So, as a workaround, how about using a button and the custom menu? When the script is run by the button and the custom menu, the script works. It seems that this method is the same as that of the script editor.
The sample script is as follows.
Sample script:
Before you run the script, please set range. For example, please assign this function to a button on Spreadsheet. When you click the button, the script is run. In this sample, it supposes that the values like #charlidamelio are put to the column "A".
function sample() {
var range = "A2:A10"; // Please set the range of "handle".
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
var sheet = SpreadsheetApp.getActiveSheet();
var r = sheet.getRange(range);
var values = r.getValues();
var res = values.map(([handle]) => {
if (handle != "") {
var web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
var match_text = raw_data.exec(web_content);
return [match_text[2]];
}
return [""];
});
r.offset(0, 1).setValues(res);
}
When this script is run, the values are retrieved from the URL and put to the column "B".
Note:
This is a simple script. So please modify it for your actual situation.
Reference:
Related thread.
UrlFetchApp request fails in Menu Functions but not in Custom Functions (connecting to external REST API)
Added:
About the following additional question,
whilst this works for 1 TikTok handle, when trying to run a list of multiple it fails each time, with the error TypeError: Cannot read property '2' of null. After doing some investigating and manually mapping out 8 handles, I can see that each time it runs, it returns "null" for one or more of the web_content variables. Is there a way to slow the script down/run each UrlFetchApp one at a time to ensure each returns content?
i've tried this and still getting an error. Have tried up to 10000ms. I've added some more detail to the original question, hope this makes sense as to the error. It is always in a different log that I get nulls, hence why I think it's a timing or cache issue.
In this case, how about the following sample script?
Sample script:
In this sample script, when the value cannot be retrieved from the URL, the value is tried to retrieve again as the retry. This sample script uses the 2 times as the retry. So when the value cannot be retrieved by 2 retries, the empty value is returned.
function sample() {
var range = "A2:A10"; // Please set the range of "handle".
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
var sheet = SpreadsheetApp.getActiveSheet();
var r = sheet.getRange(range);
var values = r.getValues();
var res = values.map(([handle]) => {
if (handle != "") {
var web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
var match_text = raw_data.exec(web_content);
if (!match_text || match_text.length != 3) {
var retry = 2; // Number of retry.
for (var i = 0; i < retry; i++) {
Utilities.sleep(3000);
web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
match_text = raw_data.exec(web_content);
if (match_text || match_text.length == 3) break;
}
}
return [match_text && match_text.length == 3 ? match_text[2] : ""];
}
return [""];
});
r.offset(0, 1).setValues(res);
}
Please adjust the value of retry and Utilities.sleep(3000).
This works for me as a Custom Function:
function MYFUNK(n=2) {
const url = 'my website url'
const re = new RegExp(`<p id="un${n}.*\/p>`,'g')
const r = UrlFetchApp.fetch(url).getContentText();
const v = r.match(re);
Logger.log(v);
return v;
}
I used my own website and I have several paragraphs with ids from un1 to un7 and I'm taking the value of A1 for the only parameter. It returns the correct string each time I change it.
I have this google script to import api calls into google sheet cells.
/**
* Imports JSON data to your spreadsheet
* #param url URL of your JSON data as string
* #param xpath simplified xpath as string
* #customfunction
*/
function IMPORTJSON(url,xpath){
try{
var res = UrlFetchApp.fetch(url);
var content = res.getContentText();
var json = JSON.parse(content);
var patharray = xpath.split(".");
//Logger.log(patharray);
for(var i=0;i<patharray.length;i++){
json = json[patharray[i]];
Utilities.sleep(1000);
}
//Logger.log(typeof(json));
if(typeof(json) === "undefined"){
return "Node Not Available";
} else if(typeof(json) === "object"){
var tempArr = [];
for(var obj in json){
tempArr.push([obj,json[obj]]);
}
return tempArr;
} else if(typeof(json) !== "object") {
return json;
}
}
catch(err){
return "Error getting data";
}
}
And I have various cells into google sheet that calls this function.
To update the cache and get updated values from the same api calls, i added a random number parameter at the end of each string that chance simultaneously for all the calls
=IF(C6>0;IMPORTJSON(concatenate("https://axieinfinity.com/graphql-server-v2/graphql?operationName=GetAxieBriefList&query=query%20GetAxieBriefList%20%7B%20axies(auctionType:All,owner:%22";H6;"%22,%20from:%200,%20sort:%20PriceAsc,%20size:%20100)%20%7B%20total%20%7D%20%7D");"data.axies.total";doNotDelete!$A$1);0)
My problem is that I want to put a delay into the script to avoid updating all the api calls in the same time.
How can I achieve this result? Is it possible just adding a utilities.sleep(500) into the script? Because I didn't have any success by doing that
It's very likely that using a custom function is not a good idea because they have several limitations, instead consider to use other means to update the cells like using Range.setValue, Range.setValues or the Advanced Sheet Service (more specifically batchUpdate). Google Apps Script functions having these methods could be called from a custom menu, a simple or installable trigger, from client-side code by using google.script.run, etc.
Some of the relevant limitations of custom functions are
30 seconds maximum execution time
all the formulas (including those having custom functions) are recalculated when the spreadsheet is opened.
I am trying to develop a program that automatically fills out a google form using the data provided in google sheets.
This is my code.
function auto_data_entry() {
var formURL = "(URL of the form would be put here)";
var workbook = SpreadsheetApp.getActiveSpreadsheet();
var worksheet = workbook.getSheetByName("Sheet1");
var full_name = worksheet.getRange("A2").getValue();
var year = worksheet.getRange("B2").getValue();
var month = worksheet.getRange("C2").getValue();
var day = worksheet.getRange("D2").getValue();
var period = worksheet.getRange("E2").getValue();
var datamap =
{
"entry.1901360617": full_name,
"entry.43103907_year": year,
"entry.43103907_month": month,
"entry.43103907_day": day,
"entry.1047848587": period
};
var options =
{
"method": "post",
"payload": datamap
};
UrlFetchApp.fetch(formURL, options); //Line 27
}
However, it returns...
Exception: Request failed for https://docs.google.com returned code 401.
Truncated server response: <!DOCTYPE html><html lang="en"><head><meta name="description"
content="Web word processing, presentations and spreadsheets"><meta name="viewport" c...
(use muteHttpExceptions option to examine full response) (line 27, file "Code")
Is the problem that I am using a school owned google account or that there is an error with my code.
I am very lost and would appreciate it if someone could help out.
There is no need to use UrlFetchApp because you can use the Class FormResponse and the Class ItemResponse. This code will help you with your issue:
function autoDataEntry() {
// Get the desire form with its questions and create
// a response to later be submitted
var form = FormApp.openById("YOUR-FORM-ID");
var formResponse = form.createResponse();
var formQuestions = form.getItems();
var workbook = SpreadsheetApp.getActiveSpreadsheet();
var worksheet = workbook.getSheetByName("Sheet1");
// Get all the needed values in the second row
var answers = worksheet.getRange("A2:E2").getValues();
answers[0].forEach((answer, answerNumber) => {
// Get the question depending of its type
var question = getQuestion(formQuestions, answerNumber);
// Create the response to your question with the value obtained in the sheet
var formAnswer = question.createResponse(answer);
// Add the answer to the response
formResponse.withItemResponse(formAnswer);
});
// submit the form response
formResponse.submit();
}
What I did was to get the form where you want to send your response and the sheet where the answers are. Then I iterated through those answers to add them to the respective question, which would be added to the form response. When that process is finished, then you only need to submit the form response.
Edit
I modified my code by adding the following function and calling it inside the forEach in my autoDataEntry function:
// This function will return the question as the requiered type
function getQuestion(formQuestions, answerNumber){
var questionType = formQuestions[answerNumber].getType();
switch(questionType){
case FormApp.ItemType.TEXT:
return formQuestions[answerNumber].asTextItem();
case FormApp.ItemType.MULTIPLE_CHOICE:
return formQuestions[answerNumber].asMultipleChoiceItem();
case FormApp.ItemType.DATE:
return formQuestions[answerNumber].asDateItem();
}
}
In that way, you will get the proper question type as the situation requires as long you have set it as a condition in the switch statement. You can see all types in Enum ItemType.
I am trying to get the following to work: A JSON is sent to my Google Script - Now I want to get specific values from that JSON Message and store them into specific rows of one specific Google Spreadsheet. So far so good, this is what I have :
function doPost(response) {
var sheets = SpreadsheetApp.openById('MY SHEET ID');
var dataAll = JSON.parse(response.getContentText());
var nR = getNextRow(sheets) + 1;
// RECORD DATA IN SPREADSHEET
sheets.getRangeByName('timestamp').getCell(nR,1).setValue(new Date());
sheets.getRangeByName('ticket_id').getCell(nR,1).setValue(dataAll);
}
function getNextRow(sheets) {
var timestamps = sheets.getRangeByName("timestamp").getValues();
for (i in timestamps) {
if(timestamps[i][0] == "") {
return Number(i);
break;
}}}
It should store the response and put it into a blank cell of the range "timestamp". But nothing happens at this point.
This is the JSON ( Body ) from JIRA:
{"timestamp":1483576902984,"webhookEvent":"jira:issue_created","issue_event_type_name":"issue_created","user":{"self":"https://xxx.atlassian.net/rest/api/2/user?username=admin","name":"admin","key":"admin","emailAddress":"test#mail.at","avatarUrls":{"48x48":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=48","24x24":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=24","16x16":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=16","32x32":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=32"},"displayName":"Max Mustermann [Administrator]","active":true,"timeZone":"Europe/Berlin"},"issue":{"id":"10057","self":"https://xxx.atlassian.net/rest/api/2/issue/10057","key":"TA-58","fields":{"issuetype":{"self":"https://xxx.atlassian.net/rest/api/2/issuetype/10104","id":"10104","description":"A problem which impairs or prevents the functions of the product.","iconUrl":"https://xxx.atlassian.net/secure/viewavatar?size=xsmall&avatarId=10303&avatarType=issuetype","name":"Bug","subtask":false,"avatarId":10303},"timespent":null,"project":{"self":"https://xxx.atlassian.net/rest/api/2/project/10000","id":"10000","key":"TA","name":"Test Area","avatarUrls":{"48x48":"https://xxx.atlassian.net/secure/projectavatar?avatarId=10324","24x24":"https://xxx.atlassian.net/secure/projectavatar?size=small&avatarId=10324","16x16":"https://xxx.atlassian.net/secure/projectavatar?size=xsmall&avatarId=10324","32x32":"https://xxx.atlassian.net/secure/projectavatar?size=medium&avatarId=10324"}},"customfield_10110":null,"fixVersions":[],"customfield_10111":null,"aggregatetimespent":null,"customfield_10112":"Not started","resolution":null,"customfield_10113":null,"customfield_10114":null,"customfield_10104":null,"customfield_10105":null,"customfield_10106":null,"customfield_10107":null,"customfield_10108":null,"customfield_10109":null,"resolutiondate":null,"workratio":-1,"lastViewed":null,"watches":{"self":"https://xxx.atlassian.net/rest/api/2/issue/TA-58/watchers","watchCount":0,"isWatching":false},"created":"2017-01-05T01:41:42.903+0100","priority":{"self":"https://xxx.atlassian.net/rest/api/2/priority/3","iconUrl":"https://xxx.atlassian.net/images/icons/priorities/medium.svg","name":"Medium","id":"3"},"customfield_10100":null,"customfield_10101":null,"customfield_10102":null,"customfield_10103":null,"labels":[],"timeestimate":null,"aggregatetimeoriginalestimate":null,"versions":[],"issuelinks":[],"assignee":null,"updated":"2017-01-05T01:41:42.903+0100","status":{"self":"https://xxx.atlassian.net/rest/api/2/status/10000","description":"","iconUrl":"https://xxx.atlassian.net/","name":"To Do","id":"10000","statusCategory":{"self":"https://xxx.atlassian.net/rest/api/2/statuscategory/2","id":2,"key":"new","colorName":"blue-gray","name":"To Do"}},"components":[],"timeoriginalestimate":null,"description":"super alles neu","timetracking":{},"customfield_10005":null,"attachment":[],"aggregatetimeestimate":null,"summary":"super alles neu","creator":{"self":"https://xxx.atlassian.net/rest/api/2/user?username=admin","name":"admin","key":"admin","emailAddress":"test#mail.at","avatarUrls":{"48x48":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=48","24x24":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=24","16x16":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=16","32x32":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=32"},"displayName":"Max Mustermann [Administrator]","active":true,"timeZone":"Europe/Berlin"},"subtasks":[],"reporter":{"self":"https://xxx.atlassian.net/rest/api/2/user?username=admin","name":"admin","key":"admin","emailAddress":"test#mail.at","avatarUrls":{"48x48":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=48","24x24":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=24","16x16":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=16","32x32":"https://secure.gravatar.com/avatar/3d238d8be45bd26982fa09ae2f891c3f?d=mm&s=32"},"displayName":"Max Mustermann [Administrator]","active":true,"timeZone":"Europe/Berlin"},"customfield_10000":"{}","aggregateprogress":{"progress":0,"total":0},"customfield_10001":null,"customfield_10115":null,"customfield_10116":"0|i0005r:","environment":null,"duedate":null,"progress":{"progress":0,"total":0},"comment":{"comments":[],"maxResults":0,"total":0,"startAt":0},"votes":{"self":"https://xxx.atlassian.net/rest/api/2/issue/TA-58/votes","votes":0,"hasVoted":false},"worklog":{"startAt":0,"maxResults":20,"total":0,"worklogs":[]}}}}
However, I don't want to have the whole JSON in my cell, I only want to have specific obejcts/id from within the JSON. How do I call them ?
After tons of research, this is a solution that works for me (in my case):
function doPost(response) {
var sheets = SpreadsheetApp.openById('SHEET_ID');
// retrieve data from JIRA Payload and store them into "data"
var json = response.postData.contents;
var data = JSON.parse(json);
// index values from "data" and store them into seperate variables
// for example:
var ticket_id = data.issue.key;
var priority_name = data.issue.fields.priority.name;
var summary = data.issue.fields.summary;
This two lines:
var json = response.postData.contents;
var data = JSON.parse(json);
Made it possible to read the body and index all the specific parameters I want.
Here is an example:
/*
* webhookHandler: JIRA webhook callback function
*/
function webhookHandler(response) {
var data = response.getAs("application/json");
//logs out data in dev console
console.log(data);
var spreadsheet = SpreadsheetApp.openById("<spreadsheet id>");
var cellRange = spreadsheet.getRangeByName("<some range name>");
var cell = cellRange.getCell(0 /*row index*/, 0/*column index*/);
cell.setValue(data.ticket_id/*index the JSON object returned by response*/);
}
UrlFetchApp Documentation
SpreadsheetApp Documentation
I was a user of the deprecated ScriptDB. The use I made of ScriptDB was fairly simple: to store a certain amount of information contained on a panel options, this way:
var db = ScriptDb.getMyDb();
function showList(folderID) {
var folder = DocsList.getFolderById(folderID);
var files = folder.getFiles();
var arrayList = [];
for (var file in files) {
file = files[file];
var thesesName = file.getName();
var thesesId = file.getId();
var thesesDoc = DocumentApp.openById(thesesId);
for (var child = 0; child < thesesDoc.getNumChildren(); child++){
var thesesFirstParagraph = thesesDoc.getChild(child);
var thesesType = thesesFirstParagraph.getText();
if (thesesType != ''){
var newArray = [thesesName, thesesType, thesesId];
arrayList.push(newArray);
break;
}
}
}
arrayList.sort();
var result = db.query({arrayName: 'savedArray'});
if (result.hasNext()) {
var savedArray = result.next();
savedArray.arrayValue = arrayList;
db.save(savedArray);
}
else {
var record = db.save({arrayName: "savedArray", arrayValue:arrayList});
}
var mydoc = SpreadsheetApp.getActiveSpreadsheet();
var app = UiApp.createApplication().setWidth(550).setHeight(450);
var panel = app.createVerticalPanel()
.setId('panel');
var label = app.createLabel("Choose the options").setStyleAttribute("fontSize", 18);
app.add(label);
panel.add(app.createHidden('checkbox_total', arrayList.length));
for(var i = 0; i < arrayList.length; i++){
var checkbox = app.createCheckBox().setName('checkbox_isChecked_'+i).setText(arrayList[i][0]);
panel.add(checkbox);
}
var handler = app.createServerHandler('submit').addCallbackElement(panel);
panel.add(app.createButton('Submit', handler));
var scroll = app.createScrollPanel().setPixelSize(500, 400);
scroll.add(panel);
app.add(scroll);
mydoc.show(app);
}
function include(arr, obj) {
for(var i=0; i<arr.length; i++) {
if (arr[i] == obj) // if we find a match, return true
return true; }
return false; // if we got here, there was no match, so return false
}
function submit(e){
var scriptDbObject = db.query({arrayName: "savedArray"});
var result = scriptDbObject.next();
var arrayList = result.arrayValue;
db.remove(result);
// continues...
}
I thought I could simply replace the ScriptDB by userProperties (using JSON to turn the array into string). However, an error warns me that my piece of information is too large to be stored in userProperties.
I did not want to use external databases (parse or MongoDB), because I think it isn't necessary for my (simple) purpose.
So, what solution I could use as a replacement to ScriptDB?
You could store a string using the HtmlOutput Class.
var output = HtmlService.createHtmlOutput('<b>Hello, world!</b>');
output.append('<p>Hello again, world.</p>');
Logger.log(output.getContent());
Google Documentation - HtmlOutput
There are methods to append, clear and get the content out of the HtmlOutput object.
OR
Maybe create a Blob:
Google Documentation - Utilities Class - newBlob Method
Then you can get the data out of the blob as a string.
getDataAsString
Then if you need to you can convert the string to an object if it's in the right JSON format.
Firstly, if you're hitting the limits on the Properties service, I would recommend you look at an alternative external store, as you're manipulating a large amount of data, and any workaround given here is possibly going to be slower and less efficient then simply using a dedicated service.
Alternatively of course, you could look at making your data come under the limits for the properties service by splitting it up and using multiple properties etc.
One other alternative would be to use a Google Doc or Sheet to store the string. When you're required to pull the data again, you can simply access the sheet and get the string, but this might be slow depending on the size of the string. At a glance it looks like you're just pulling Data on the folders in your drive, so you could consider writing it to a sheet, which would allow you to even display the information in a user friendly way. Given your use of arrays already, you can write them to a sheet easily using .setValues() if you convert them to a 2D array.
Bruce McPherson has done a lot of work on abstracting databases. Take a look at his cDbAbstraction library then you could easily chop and change which DB you use and compare performance. Maybe even create a cDbAbstraction library to use HTMLOutput (I like that idea Sandy, Bruce does some funky stuff with parallel processes via HTMLService)