I am trying to import fivethirtyeight data, but when trying to import the full link from the CSV file, it exceeds the limit allowed by google.
So ... I would like to know how I could filter so that only today's games and future dates are imported. From the games of past dates there is no value to me.
=IMPORTDATA("https://projects.fivethirtyeight.com/soccer-api/club/spi_matches.csv")
IMPORTDATA didn't work because it exceeds the spreadsheet limit, because it saves past game data since 2016, so the data list gets too big.
I just need the current date and future date games
How about this answer?
It seems that the size of CSV data is 4,183,375 bytes, and 32,101 rows and 22 columns. So how about putting the data using Google Apps Script? Unfortunately, this CSV cannot be put using the custom function, because of the large size.
From the benchmark for importing CSV data to Spreadsheet, when Sheets API is used, the process cost can be reduced from the method with Utilities.parseCsv() and setValues(). So in this answer, Sheets API is used for putting CSV values to Spreadsheet. Please think of this as just one of several answers.
Sample script 1:
Before you use the following scripts, please enable Sheets API at Advanced Google services.
function myFunction() {
var url = "https://projects.fivethirtyeight.com/soccer-api/club/spi_matches.csv";
var data = UrlFetchApp.fetch(url).getContentText();
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getActiveSheet();
var resource = {requests: [{pasteData: {
data: data,
coordinate: {sheetId: sheet.getSheetId()},
delimiter: ","
}}]};
Sheets.Spreadsheets.batchUpdate(resource, ss.getId());
}
In this script, the CSV data is put from the cell "A1" of the active sheet.
Sample script 2:
If you cannot use Sheets API, you can also use the following script.
function myFunction() {
var url = "https://projects.fivethirtyeight.com/soccer-api/club/spi_matches.csv";
var data = UrlFetchApp.fetch(url).getContentText();
var csv = Utilities.parseCsv(data);
var sheet = SpreadsheetApp.getActiveSheet();
sheet.getRange(1, 1, csv.length, csv[0].length).setValues(csv);
}
References:
Benchmark: Importing CSV Data to Spreadsheet using Google Apps Script
PasteDataRequest
If this was not the direction you want, I apologize.
Related
I'm trying to get the data from this csv file: "https://app.matrixify.app/files/vasasro/99320ce64bfe22f855ce03e71796e4db/MatrixifyAllExport.csv"
This is for product that we want to edit for vasaro.com and we need to export the data to Google Sheets automatically.
Thank you for your help.
When I tried IMPORTDADA, it worked for small files, but not for this larger file.
When I check the CSV data of your URL, it seems that the data size is 2,413,638 bytes and 1158 rows and 93 columns. I think that this is the reason for your current issue. In this case, in order to put the values into Spreadsheet, how about using Google Apps Script?
Sample script 1:
Please copy and paste the following script to the script editor of Spreadsheet, and save the script. When you use this script, please put a custom function of =SAMPLE("https://app.matrixify.app/files/vasasro/99320ce64bfe22f855ce03e71796e4db/MatrixifyAllExport.csv") to a cell. By this, the CSV data is retrieved and put the values to the Spreadsheet. This can be used like IMPORTDATA.
const SAMPLE = url => Utilities.parseCsv(UrlFetchApp.fetch(url).getContentText());
Sample script 2:
Please copy and paste the following script to the script editor of Spreadsheet, and save the script. And, please set the sheet name. In this case, please run the function myFunction with the script editor. By this, the CSV data is downloaded and put to the Spreadsheet.
function myFunction() {
const sheetName = "Sheet1"; // Please set your sheet name.
const url = "https://app.matrixify.app/files/vasasro/99320ce64bfe22f855ce03e71796e4db/MatrixifyAllExport.csv"; // This is from your URL.
const res = UrlFetchApp.fetch(url);
const values = Utilities.parseCsv(res.getContentText());
const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(sheetName);
sheet.getRange(1, 1, values.length, values[0].length).setValues(values);
}
Sample script 3:
If the CSV data is large, when Sheets API is used, the process cost can be reduced a little. In that case, please enable Sheets API at Advanced Google services. And, please set the sheet name. And, please run the function myFunction with the script editor. By this, the CSV data is downloaded and put to the Spreadsheet.
function myFunction() {
const sheetName = "Sheet1"; // Please set your sheet name.
const url = "https://app.matrixify.app/files/vasasro/99320ce64bfe22f855ce03e71796e4db/MatrixifyAllExport.csv"; // This is from your URL.
const res = UrlFetchApp.fetch(url);
const ss = SpreadsheetApp.getActiveSpreadsheet();
const sheetId = ss.getSheetByName(sheetName).getSheetId();
const requests = [{ pasteData: { data: res.getContentText(), delimiter: ",", coordinate: { sheetId } } }];
Sheets.Spreadsheets.batchUpdate({ requests }, ss.getId());
}
References:
Custom Functions in Google Sheets
parseCsv(csv)
setValues(values)
PasteDataRequest
I need your help to improve my apps script code in order to copy a dataset from 1 file to another file in Google Sheets. Currently it takes nearly 6 mins to finish this execution but our data is bigger by days.
My details case is
In the source file, sheet Database with the range is A1:U11834. But the row will be increased days by days.
In the destination file, I also have a sheet name Database, and I want to clear the old data then copy the above source data into this sheet.
Here is my code.
function getdata(){
let ss = SpreadsheetApp
// open source file and sheet Database
let source_file = ss.openById("id_source_file")
let source_sht_copy = source_file.getSheetByName("Database")
// Get full range of data
let lr = source_sht_copy.getRange("A1").getDataRegion().getLastRow()
let actual_range = `A1:U${lr}`
Logger.log(actual_range)
let source_data = source_sht_copy.getRange(actual_range).getValues()
Logger.log("Copy Done")
// Open destination file
let dest_file = ss.openById("id_dest_file")
let dest_sht = dest_file.getSheetByName("Database")
// //clear content sheet database of destination file
dest_sht.clearContents()
Logger.log("Delete Old Data Done")
// // paste data from source file to destination file using method 'setValues'
dest_sht.getRange(actual_range).setValues(source_data)
Logger.log("Paste Done")
}
And this is the image show the time of processing.
In your situation, in order to reduce the process cost, how about using Sheets API? When Sheets API is reflected in your script, it becomes as follows. When Sheets API is used, the process cost can be reduced than that of Spreadsheet services (SpreadsheetApp). Ref
Modified script:
Before you use this script, please enable Sheets API at Advanced Google services.
function getdata2() {
const srcSpreadsheetId = "###"; // Please set source Spreadsheet ID.
const dstSpreadsheetId = "###"; // Please set destination Spreadsheet ID.
const srcRange = "'Database'!A1:U";
const dstRange = "Database";
const values = Sheets.Spreadsheets.Values.get(srcSpreadsheetId, srcRange).values;
const sheetId = SpreadsheetApp.openById(dstSpreadsheetId).getSheetByName(dstRange).getSheetId();
Sheets.Spreadsheets.batchUpdate({requests:[{repeatCell:{range:{sheetId},fields:"userEnteredValue"}}]}, dstSpreadsheetId);
Sheets.Spreadsheets.Values.update({values}, dstSpreadsheetId, dstRange, {valueInputOption: "USER_ENTERED"});
}
References:
Benchmark: Reading and Writing Spreadsheet using Google Apps Script
Method: spreadsheets.values.get
Method: spreadsheets.values.update
Maybe someone can help me with my Problem.
First some background: I have copied some WhatsApp chats to Google Sheets. I use Latex to generate a book containing the chats. Google Sheet is able to display all Emojis from WhatsApp, Latex, of course, isn't. So I downloaded the Emojis as png files and defined Latex commands to include those emojis as graphic.
For Example: Too display the regular smiling emoji I type \grin. I have a list with hundreds of emojis in one row and the corresponding command in the next row.
Until now I used search and replace -> replace all in the same sheet for every single type of emoji. But as this takes hours I wondered if there is any way to make this more effective.
Here is a spreadsheet with a small example: Google Sheet
Thanks in advance!
You want to replace the Emojis to the defined Latex commands.
For example, you want to replace as follows.
From 03.01.19, 00:29 - me: Hi 😊 to 03.01.19, 00:29 - me: Hi \nettnett.
You want to achieve this using Google Apps Script.
If my understanding is correct, how about this answer? Please think of this as just one of several possible answers.
Flow:
The flow of this sample script is as follows. This sample script uses your shared Spreadsheet.
Retrieve data from the sheet of Codes for Smileys.
Create the request body for the findReplace request of batchUpdate method of Sheets API.
Run the method of batchUpdate.
Sample script:
This script used Sheets API. So, before you run the script, please enable Sheets API at Advanced Google services.
function myFunction() {
var dataSheet = "Codes for Smileys";
var sourceSheet = "unedited Chats with Smileys";
var ss = SpreadsheetApp.getActiveSpreadsheet();
var data = ss.getSheetByName(dataSheet).getDataRange().getValues();
data.shift();
var sheetId = ss.getSheetByName(sourceSheet).getSheetId();
var requests = data.map(function(row) {return {findReplace: {sheetId: sheetId, find: row[0], replacement: row[1]}}});
Sheets.Spreadsheets.batchUpdate({requests: requests}, ss.getId());
}
This sample script uses your shared Spreadsheet. So in this case, the data sheet is Codes for Smileys. And the source sheet for converting is unedited Chats with Smileys.
Note:
If you change the sheet name, also please modify above script. Please be careful this.
When you run the script for the first time, the authorization screen is opened. So please authorize the scopes for using the script.
References:
Advanced Google services
spreadsheets.batchUpdate
FindReplaceRequest
Added:
You want to put the converted values to the sheet of Chat with Latex Code.
The sample script for achieving above is as follows.
Sample script:
function myFunction2() {
var dataSheet = "Codes for Smileys";
var sourceSheet = "unedited Chats with Smileys";
var destinationSheet = "Chats with LaTeX Codes";
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName(dataSheet);
var data = ss.getSheetByName(dataSheet).getDataRange().getValues();
data.shift();
var srcSheet = ss.getSheetByName(sourceSheet);
var tempSheet = srcSheet.copyTo(ss);
var sheetId = tempSheet.getSheetId();
var requests = data.map(function(row) {return {findReplace: {sheetId: sheetId, find: row[0], replacement: row[1]}}});
Sheets.Spreadsheets.batchUpdate({requests: requests}, ss.getId());
var tempValues = tempSheet.getDataRange().getValues();
var destSheet = ss.getSheetByName(destinationSheet);
destSheet.getRange(destSheet.getLastRow() + 1, 1, tempValues.length, tempValues[0].length).setValues(tempValues);
ss.deleteSheet(tempSheet);
}
In this sample script, the following flow is run.
Copy the source sheet unedited Chats with Smileys as a temporal sheet.
Create request body for the batchUpdate method to the temporal sheet.
Run the batchUpdate.
Copy the converted values from the temporal sheet to the destination sheet Chats with LaTeX Codes.
In this case, the converted values are put to the last row of the sheet.
Delete the temporal sheet.
I am importing data from a .CSV file
function myFunction() {
var url = "https://projects.fivethirtyeight.com/soccer-api/club/spi_matches.csv";
var data = UrlFetchApp.fetch(url).getContentText();
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getActiveSheet();
var resource = {requests: [{pasteData: {
data: data,
coordinate: {sheetId: sheet.getSheetId()},
delimiter: ","
}}]};
Sheets.Spreadsheets.batchUpdate(resource, ss.getId());
}
The data from past dates does not interest me and I would like to filter to make the sheet smaller and also to make my work easier, via script I could not filter directly from the data source and when I try to filter by a formula in another page, the filter does not work.
=FILTER('Page1'!A2:A,'Page1'!A2:A>=TEXT(now(),"dd/mm/yyyy"))
I would like to know what options I have to be able to filter this data, because I will update it daily, and this filter is very important for the study I am going to do.
If there is any way to filter already at import time, it would be even better, but if i need to use functions on another spreadsheet page, no problem either.
You want to retrieve CSV data and put it to Spreadsheet.
When the CSV data is put, you want to remove the data of before today.
The date values are the column "A".
You want to achieve this using Google Apps Script.
If my understanding is correct, how about this modification? Please think of this as just one of several answers.
In this modification, the following flow is used.
Retrieve CSV data with UrlFetchApp.fetch().
Parse CSV data with Utilities.parseCsv().
Remove the values before today from the parsed data with filter().
Put the values to Spreadsheet with setValues().
Modified script:
function myFunction() {
var url = "https://projects.fivethirtyeight.com/soccer-api/club/spi_matches.csv";
var data = UrlFetchApp.fetch(url).getContentText();
var csv = Utilities.parseCsv(data);
var d = new Date();
d.setHours(0, 0, 0, 0);
var today = d.getTime();
var values = csv.filter(function(e, i) {
var temp = new Date(e[0]);
temp.setHours(0, 0, 0, 0);
return i > 0 && temp.getTime() >= today;
});
values.unshift(csv[0]);
var sheet = SpreadsheetApp.getActiveSheet();
sheet.getRange(1, 1, values.length, values[0].length).setValues(values);
}
In this script, the values are put to the active sheet.
References:
filter()
If I misunderstood your question and this was not the result you want, I apologize.
I receive an XLSX file from our client on regular basis, and I would like to automate the process of importing it from Gmail (it's automatically labeled) into Google Sheets. So far I managed to make it work for CSV files, but XLSX files seem to be trickier. Can someone help to adjust this code I have for CSV files?
function getCSV()
{
var thread = GmailApp.getUserLabelByName(‘Reconciliation’).getThreads(0,1);
var messages = thread[0].getMessages();
var len = messages.length;
var message=messages[len-1] //get last message
var attachments = message.getAttachments(); // Get attachment of first message
var csv = attachments[0].getDataAsString();
var data = Utilities.parseCsv(csv);
var sheet = SpreadsheetApp.openById("some id").getSheetByName(‘Data’);
sheet.clearContents();
var range = sheet.getRange(1, 1, data.length, data[0].length);
range.setValues(data);
}
You want to put the data from xlsx file attached to an email to the existing Spreadsheet.
If my understanding is correct, how about this modification? Please think of this as just one of several answers.
When you use this script, please enable Drive API at Advanced Google Services and API console. You can see about this at here.
Flow of modified script:
Retrieve a blob of xlsx file.
Convert xlsx format to Google Spreadsheet.
Retrieve values from the converted Spreadsheet.
Remove the converted file.
Put the values to the sheet of Data in the existing Spreadsheet.
Modified script:
Please modify as follows.
From:
var csv = attachments[0].getDataAsString();
var data = Utilities.parseCsv(csv);
To:
var xlsxBlob = attachments[0]; // Is supposes that attachments[0] is the blob of xlsx file.
var convertedSpreadsheetId = Drive.Files.insert({mimeType: MimeType.GOOGLE_SHEETS}, xlsxBlob).id;
var sheet = SpreadsheetApp.openById(convertedSpreadsheetId).getSheets()[0]; // There is the data in 1st tab.
var data = sheet.getDataRange().getValues();
Drive.Files.remove(convertedSpreadsheetId); // Remove the converted file.
Note:
In this modification, it supposes the following points. If your situation is different from the following points, please modify it.
attachments[0] is the blob of xlsx file.
About the xlsx file, the data you want to put is in a 1st tab.
Reference:
Files: insert of Drive API v2
If I misunderstood your question and this didn't work, I apologzize.