I am having a spreadsheet which is having more than 1000+ video IDs of youTube in one column. I am trying to retrieve the details of video duration and viewcount and other parameters. However, i am able to get only 50 records not more than that. The error comes up in google script, as soon as i try to get more than 50 records.
The error: "GoogleJsonResponseException: API call to youtube.videos.list failed with error: The request specifies an invalid filter parameter. getYoutubeVideoStatistics" This is for the line "var statsVideo = YouTube.Videos.list("contentDetails, statistics",{id: videoIDArray, maxResults:50, pageToken: pageToken})
The code i am using is given below is taken from one of the youTube Channel, which have adapted to my needs.
Kindly help, as i am not conversant with the coding.!
function getYoutubeVideoStatistics(){
var activeSheet = SpreadsheetApp.getActiveSheet();
var videoIDRange = activeSheet.getRange(4,4,activeSheet.getLastRow()-3)
var videoIDRangeValues = videoIDRange.getValues();
var videoIDArray = videoIDRangeValues.map(function(videoId){return videoId[0];}).join(",");
var pageToken = "";
do {
var statsVideo = YouTube.Videos.list("contentDetails, statistics",{id: videoIDArray, maxResults:50, pageToken: pageToken})
//Logger.log(statsVideo);
pageToken = statsVideo.nextPageToken;
} while (pageToken);
var vidDuration = statsVideo.items.map(function(dur){return [dur.contentDetails.duration] })
var details = statsVideo.items.map(function(stats){return [stats.statistics.viewCount,stats.statistics.commentCount, stats.statistics.likeCount] })
activeSheet.getRange(4,6,vidDuration.length,vidDuration[0].length).setValues(vidDuration);
activeSheet.getRange(4,7,details.length,details[0].length).setValues(details);
}
Related
I have built a simple custom function in Apps Script using URLFetchApp to get the follower count for TikTok accounts.
function tiktok_fans() {
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
var handle = '#charlidamelio';
var web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
var match_text = raw_data.exec(web_content);
var result = (match_text[2]);
Logger.log(result)
return result
}
The Log comes back with the correct number for followers.
However, when I change the code to;
function tiktok_fans(handle) {
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
//var handle = '#charlidamelio';
var web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
var match_text = raw_data.exec(web_content);
var result = (match_text[2]);
Logger.log(result)
return result
}
and use it in a spreadsheet for example =tiktok_fans(A1), where A1 has #charlidamelio I get an #ERROR response in the cell
TypeError: Cannot read property '2' of null (line 6).
Why does it work in the logs but not in the spreadsheet?
--additional info--
Still getting the same error after testing #Tanaike answer below, "TypeError: Cannot read property '2' of null (line 6)."
Have mapped out manually to see the error, each time the below runs, a different log returns "null". I believe this is to do with the ContentText size/in the cache. I have tried utilising Utilities.sleep() in between functions with no luck, I still get null's.
code
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
//tiktok urls
var qld = UrlFetchApp.fetch('https://www.tiktok.com/#thisisqueensland?lang=en').getContentText();
var nsw = UrlFetchApp.fetch('https://www.tiktok.com/#visitnsw?lang=en').getContentText();
var syd = UrlFetchApp.fetch('https://www.tiktok.com/#sydney?lang=en').getContentText();
var tas = UrlFetchApp.fetch('https://www.tiktok.com/#tasmania?lang=en').getContentText();
var nt = UrlFetchApp.fetch('https://www.tiktok.com/#ntaustralia?lang=en').getContentText();
var nz = UrlFetchApp.fetch('https://www.tiktok.com/#purenz?lang=en').getContentText();
var aus = UrlFetchApp.fetch('https://www.tiktok.com/#australia?lang=en').getContentText();
var vic = UrlFetchApp.fetch('https://www.tiktok.com/#visitmelbourne?lang=en').getContentText();
//find folowers with regex
var match_qld = raw_data.exec(qld);
var match_nsw = raw_data.exec(nsw);
var match_syd = raw_data.exec(syd);
var match_tas = raw_data.exec(tas);
var match_nt = raw_data.exec(nt);
var match_nz = raw_data.exec(nz);
var match_aus = raw_data.exec(aus);
var match_vic = raw_data.exec(vic);
Logger.log(match_qld);
Logger.log(match_nsw);
Logger.log(match_syd);
Logger.log(match_tas);
Logger.log(match_nt);
Logger.log(match_nz);
Logger.log(match_aus);
Logger.log(match_vic);
Issue:
From your situation, I remembered that the request of UrlFetchApp with the custom function is different from the request of UrlFetchApp with the script editor. So I thought that the reason for your issue might be related to this thread. https://stackoverflow.com/a/63024816 In your situation, your situation seems to be the opposite of this thread. But, it is considered that this issue is due to the specification of the site.
In order to check this difference, I checked the file size of the retrieved HTML data.
The file size of HTML data retrieved by UrlFetchApp executing with the script editor is 518k bytes.
The file size of HTML data retrieved by UrlFetchApp executing with the custom function is 9k bytes.
It seems that the request of UrlFetchApp executing with the custom function is the same as that of UrlFetchApp executing withWeb Apps. The data of 9k bytes are retrieved by using this.
From the above result, it is found that the retrieved HTML is different between the script editor and the custom function. Namely, the HTML data retrieved by the custom function doesn't include the regex of ("followerCount":)([0-9]+). By this, such an error occurs. I thought that this might be the reason for your issue.
Workaround:
When I tested your situation with Web Apps and triggers, the same issue occurs. By this, in the current stage, I thought that the method for automatically executing the script might not be able to be used. So, as a workaround, how about using a button and the custom menu? When the script is run by the button and the custom menu, the script works. It seems that this method is the same as that of the script editor.
The sample script is as follows.
Sample script:
Before you run the script, please set range. For example, please assign this function to a button on Spreadsheet. When you click the button, the script is run. In this sample, it supposes that the values like #charlidamelio are put to the column "A".
function sample() {
var range = "A2:A10"; // Please set the range of "handle".
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
var sheet = SpreadsheetApp.getActiveSheet();
var r = sheet.getRange(range);
var values = r.getValues();
var res = values.map(([handle]) => {
if (handle != "") {
var web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
var match_text = raw_data.exec(web_content);
return [match_text[2]];
}
return [""];
});
r.offset(0, 1).setValues(res);
}
When this script is run, the values are retrieved from the URL and put to the column "B".
Note:
This is a simple script. So please modify it for your actual situation.
Reference:
Related thread.
UrlFetchApp request fails in Menu Functions but not in Custom Functions (connecting to external REST API)
Added:
About the following additional question,
whilst this works for 1 TikTok handle, when trying to run a list of multiple it fails each time, with the error TypeError: Cannot read property '2' of null. After doing some investigating and manually mapping out 8 handles, I can see that each time it runs, it returns "null" for one or more of the web_content variables. Is there a way to slow the script down/run each UrlFetchApp one at a time to ensure each returns content?
i've tried this and still getting an error. Have tried up to 10000ms. I've added some more detail to the original question, hope this makes sense as to the error. It is always in a different log that I get nulls, hence why I think it's a timing or cache issue.
In this case, how about the following sample script?
Sample script:
In this sample script, when the value cannot be retrieved from the URL, the value is tried to retrieve again as the retry. This sample script uses the 2 times as the retry. So when the value cannot be retrieved by 2 retries, the empty value is returned.
function sample() {
var range = "A2:A10"; // Please set the range of "handle".
var raw_data = new RegExp(/("followerCount":)([0-9]+)/g);
var sheet = SpreadsheetApp.getActiveSheet();
var r = sheet.getRange(range);
var values = r.getValues();
var res = values.map(([handle]) => {
if (handle != "") {
var web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
var match_text = raw_data.exec(web_content);
if (!match_text || match_text.length != 3) {
var retry = 2; // Number of retry.
for (var i = 0; i < retry; i++) {
Utilities.sleep(3000);
web_content = UrlFetchApp.fetch('https://www.tiktok.com/'+ handle + '?lang=en').getContentText();
match_text = raw_data.exec(web_content);
if (match_text || match_text.length == 3) break;
}
}
return [match_text && match_text.length == 3 ? match_text[2] : ""];
}
return [""];
});
r.offset(0, 1).setValues(res);
}
Please adjust the value of retry and Utilities.sleep(3000).
This works for me as a Custom Function:
function MYFUNK(n=2) {
const url = 'my website url'
const re = new RegExp(`<p id="un${n}.*\/p>`,'g')
const r = UrlFetchApp.fetch(url).getContentText();
const v = r.match(re);
Logger.log(v);
return v;
}
I used my own website and I have several paragraphs with ids from un1 to un7 and I'm taking the value of A1 for the only parameter. It returns the correct string each time I change it.
I am trying to develop a program that automatically fills out a google form using the data provided in google sheets.
This is my code.
function auto_data_entry() {
var formURL = "(URL of the form would be put here)";
var workbook = SpreadsheetApp.getActiveSpreadsheet();
var worksheet = workbook.getSheetByName("Sheet1");
var full_name = worksheet.getRange("A2").getValue();
var year = worksheet.getRange("B2").getValue();
var month = worksheet.getRange("C2").getValue();
var day = worksheet.getRange("D2").getValue();
var period = worksheet.getRange("E2").getValue();
var datamap =
{
"entry.1901360617": full_name,
"entry.43103907_year": year,
"entry.43103907_month": month,
"entry.43103907_day": day,
"entry.1047848587": period
};
var options =
{
"method": "post",
"payload": datamap
};
UrlFetchApp.fetch(formURL, options); //Line 27
}
However, it returns...
Exception: Request failed for https://docs.google.com returned code 401.
Truncated server response: <!DOCTYPE html><html lang="en"><head><meta name="description"
content="Web word processing, presentations and spreadsheets"><meta name="viewport" c...
(use muteHttpExceptions option to examine full response) (line 27, file "Code")
Is the problem that I am using a school owned google account or that there is an error with my code.
I am very lost and would appreciate it if someone could help out.
There is no need to use UrlFetchApp because you can use the Class FormResponse and the Class ItemResponse. This code will help you with your issue:
function autoDataEntry() {
// Get the desire form with its questions and create
// a response to later be submitted
var form = FormApp.openById("YOUR-FORM-ID");
var formResponse = form.createResponse();
var formQuestions = form.getItems();
var workbook = SpreadsheetApp.getActiveSpreadsheet();
var worksheet = workbook.getSheetByName("Sheet1");
// Get all the needed values in the second row
var answers = worksheet.getRange("A2:E2").getValues();
answers[0].forEach((answer, answerNumber) => {
// Get the question depending of its type
var question = getQuestion(formQuestions, answerNumber);
// Create the response to your question with the value obtained in the sheet
var formAnswer = question.createResponse(answer);
// Add the answer to the response
formResponse.withItemResponse(formAnswer);
});
// submit the form response
formResponse.submit();
}
What I did was to get the form where you want to send your response and the sheet where the answers are. Then I iterated through those answers to add them to the respective question, which would be added to the form response. When that process is finished, then you only need to submit the form response.
Edit
I modified my code by adding the following function and calling it inside the forEach in my autoDataEntry function:
// This function will return the question as the requiered type
function getQuestion(formQuestions, answerNumber){
var questionType = formQuestions[answerNumber].getType();
switch(questionType){
case FormApp.ItemType.TEXT:
return formQuestions[answerNumber].asTextItem();
case FormApp.ItemType.MULTIPLE_CHOICE:
return formQuestions[answerNumber].asMultipleChoiceItem();
case FormApp.ItemType.DATE:
return formQuestions[answerNumber].asDateItem();
}
}
In that way, you will get the proper question type as the situation requires as long you have set it as a condition in the switch statement. You can see all types in Enum ItemType.
I created a script that runs onSubmit on a Google Form. It should get the ID of the image uploaded to the form, get the Image as Blob and then forward it to some email adress. The thing is, is that sometimes (about 1 in 10), the script gives the following error:
Exception: No item with the given ID could be found, or you do not
have permission to access it.
at on_Submit(Code:6:24)
I figured it would have to do with the time it takes Google to Upload/Move the file into Drive, so I set a timeout to give it some time. The error still appears, a little less frequent. Does anyone understand where this could go wrong?
The code:
function on_Submit(e) {
Utilities.sleep(30 * 1000)
var res = e.response
var image_id = res.getItemResponses()[0].getResponse()
var image = DriveApp.getFileById(image_id).getBlob()}
The on_Submit(e) function is linked to a manual trigger to enable the use of DriveApp.
Thanks
Some of the responses turned out to have multiple file uploads. The response to that question was an array of ID's. Here's the correct code:
Utilities.sleep(30 * 1000)
var res = e.response
var image_id = res.getItemResponses()[0].getResponse()
console.log(image_id)
if(Array.isArray(image_id)){
var images = [];
for(var i in image_id){
var id = image_id[i]
var image = DriveApp.getFileById(id).getBlob()
images.push(image)
}
console.log(images)
GmailApp.sendEmail(SOME_EMAIL, SUBJECT, BODY, {attachments: images})
}
else{
var image = DriveApp.getFileById(image_id).getBlob()
GmailApp.sendEmail(SOME_EMAIL, SUBJECT, BODY, {attachments: [image]})
}
I've been quite overwhelmed with all of this as I am completely new to working with APIs. I would like to populate a Google Sheet with a channel's videos so that I can always have its stats updated anytime I need while adding (or replacing the oldest) new videos to the list. I tried following a video guide and feel like I'm close but I'm having troubles.
NOTE: The playlistID I used below is for the Google Developers channel.
The video I watched showed an example where the videos being pulled were the top search results for a specific keyword. I would instead like to pull the last 50 videos from a specific channel.
The original code had the line:
var sr = YouTube.Search.list("snippet,id", { q: "guitars", maxResults: 50});
I changed that to:
var sr = YouTube.PlaylistItems.list("snippet,id", { playlistId: "UU_x5XG1OV2P6uZZ5FSM9Ttw", maxResults: 50});
The problem seems to be in this line:
var srVidsOnly = sr.items.filter(function(res){ return res.resourceId.kind === "youtube#playlistItems"});
I tried going through this and after many trial-and-error attempts, gave up. I'm not sure what this line is supposed to look like at all.
function myFunction() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var activeSheet = ss.getActiveSheet();
var sr = YouTube.PlaylistItems.list("snippet,id", { playlistId: "UU_x5XG1OV2P6uZZ5FSM9Ttw", maxResults: 50});
var srVidsOnly = sr.items.filter(function(res){ return res.resourceId.kind === "youtube#playlistItems"});
var modRes = srVidsOnly.map(function(v){ return [v.resourceId.videoId,v.snippet.title]; });
var ids = modRes.map(function(res){return res[0]; }).join(",");
var stats = YouTube.Videos.list("statistics", {id: ids});
var vidStats = stats.items.map(function(res){return [res.statistics.viewCount, res.statistics.likeCount]; });
activeSheet.getRange(2, 1, modRes.length, modRes[0].length).setValues(modRes);
activeSheet.getRange(2, 3, vidStats.length, vidStats[0].length).setValues(vidStats);
}
The error given is:
TypeError: Cannot read property "kind" from undefined. (line 6, file "Code")
Removing kind from the fifth line, like this:
var srVidsOnly = sr.items.filter(function(res){ return res.resourceId === "youtube#playlistItems"});
gives the error:
TypeError: Cannot read property "length" from undefined. (line 12, file "Code")
Following the initial video guide exactly worked (videos from search result). The sheet was formatted just as I wanted it. I just need videos from a specific channel rather than a search results.
Thanks
How about this modification?
Modification points:
Please modify as follows.
res.resourceId.kind to res.kind
youtube#playlistItems to youtube#playlistItem
v.resourceId.videoId to v.snippet.resourceId.videoId.
So when above modifications are reflected to your script, it becomes as follows.
Modified script:
Please modify myFunction() in your script as follows.
From:
var srVidsOnly = sr.items.filter(function(res){ return res.resourceId.kind === "youtube#playlistItems"});
var modRes = srVidsOnly.map(function(v){ return [v.resourceId.videoId,v.snippet.title]; });
To:
var srVidsOnly = sr.items.filter(function(res){ return res.kind === "youtube#playlistItem"});
var modRes = srVidsOnly.map(function(v){ return [v.snippet.resourceId.videoId,v.snippet.title]});
Note:
From your script, I understood that you are using Google Apps Script and Advanced Google Services.
References:
PlaylistItems
PlaylistItems: list
If I misunderstood your question and this was not the result you want, I apologize.
I have a lot of functions that fetch JSON API data from a website, but if I run them in sequence in this way, I get the exceeding execution time error:
function fetchdata () {
data1();
data2();
data3();
data4();
...
}
I can schedule a trigger to run them at 5 minutes one of the other (cause a single one runs in 3 minutes), but I would like to know if there is any other way around. Thank you
EDIT:
Every "data" function is like this one:
function data1() {
var addresses = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Import");
var baseUrl = 'https://myapiurl';
var address = addresses.getRange(2, 1, 500).getValues();
for(var i=0;i<address.length;i++){
var addrID = address[i][0];
var url = baseUrl.concat(addrID);
var responseAPI = UrlFetchApp.fetch(url);
var json = JSON.parse(responseAPI.getContentText());
var data = [[json.result]];
var dataRange = addresses.getRange(i+2, 2).setValue(data);
}
}
data2 is for rows 502-1001,
data3 is for rows 1002-1501,
and so on...
I just removed the concat because it has performance issues according to MDN but obviously the real problem is the fetch and there's not much we can do about that unless you can get your external api to dump a bigger batch.
You could initiate each function from a webapp and then have it return via withSuccessHandler and then start the next script in the series and daisy chain your way through all of the subfunctions until your done. Each sub function will take it's 3 minutes or so but you only have to worry about keeping each sub function under 6 minutes that way.
function data1()
{
var addresses = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Import");
var baseUrl = 'https://myapiurl';
var address = addresses.getRange(2, 1, 500).getValues();
for(var i=0;i<address.length;i++){
var responseAPI = UrlFetchApp.fetch(baseUrl + address[i][0]);
var json = JSON.parse(responseAPI.getContentText());
var data = [[json.result]];
var dataRange = addresses.getRange(i+2, 2).setValue(data);
}
}