I am trying to use Google Apps Script to scrape the first ten results of this page: https://www.tenderboard.gov.bh/Tenders/Opened%20Bids/.
Currently, I have written the POST request to the website. However, the response is coming back in a format that I am unsure how to process. The code I have written so far looks like this:
function fetchData() {
var url = "https://www.tenderboard.gov.bh/Templates/TenderBoardWebService.aspx/GetOpenedTenderF";
var formData = {
"tenderNumber":"",
"ministry":"0",
"category":"0",
"openedDate_filter":"",
"sortingType":"0",
"pageIndex":"2"
};
var payload = JSON.stringify(formData);
var params = {
"method":"post",
"contentType":"application/json; charset=UTF-8",
"payload":payload
};
var response = UrlFetchApp.fetch(url,params);
To illustrate its format, a sample of the response is:
{"d":"\u003cNewDataSet\u003e\r\n \u003cPager\u003e\r\n \u003cPageIndex\u003e1\u003c/PageIndex\u003e\r\n \u003cPageSize\u003e10\u003c/PageSize\u003e\r\n \u003cRecordCount\u003e3799\u003c/RecordCount\u003e\r\n \u003cContent\u003e\u0026lt;div class=\u0027table-head\u0027 style=\u0027display: table-header-group; opacity: 1;\u0027\u0026gt;\u0026lt;div class=\"column\" data-label=\"No.\"\u0026gt;No.\u0026lt;/div\u0026gt;\u0026lt;div class=\"column\"
How would I go about extracting the contents of this response to build a 2D array containing the same information as the web page?
Many thanks and apologies for what I am sure is a dense question.
Related
I've tried to find some topic about it but without results
I would need only send Form from copied prefilled link in apps script but i am not able to find out where doing misstakes.
Here are two options what i've found but without good results.
Form doesn't send any response
I've replace in prefilled Link "viewform" for "formResponse" according to some instructions in web, but it does't work.
Do you can help me anyone?
function send() {
var config = {
muteHttpExceptions: true,
method: "get"
};
var url = "https://docs.google.com/forms/d/e/<form_id>/formResponse?usp=pp_url&entry.231322462=ANO&entry.133135319=1&entry.1743053173=2023-01-15";
var response = UrlFetchApp.fetch(url, config);
//Logger.log(response)
}
or:
function send() {
var url = "https://docs.google.com/forms/d/e/<form_id>/formResponse?usp=pp_url&entry.231322462=ANO&entry.133135319=1&entry.1743053173=2023-01-15";
var response = UrlFetchApp.fetch(url);
//Logger.log(response)
}
You have an existing Google Form and you want to write a script in order to edit existing Form Responses in in the Form.
At the time of writing, this isn't possible.
Refer:
Edit Form Responses as soon as it is submitted
considering the website: https://www.coindesk.com/price/bitcoin/
I'm trying to extract the value $23,073.15 (that changes continuosly) of the following span class.
span class="typography__StyledTypography-owin6q-0 fZpnIj">$23,073.15</span
I tried the following code with Google script using Cheerio without success:
var URL = "https://www.coindesk.com/price/bitcoin/";
var response = UrlFetchApp.fetch(URL);
var $ = Cheerio.load(response.getContentText());
var itemsOfInterest = $('span.typography__StyledTypography-owin6q-0 fZpnIj').text();```
any help about?
When I saw the HTML data from your URL using Google Apps Script, unfortunately, it seems that the data is retrieved from an API by Javascript. By this, in this case, your expected value cannot be retrieved from your URL with cheerio. So, in this answer, I would like to propose retrieving your expected value using the API. The sample script is as follows.
Sample script:
function sample() {
const url = "https://production.api.coindesk.com/v2/tb/price/ticker?assets=BTC";
const res = UrlFetchApp.fetch(url);
const obj = JSON.parse(res.getContentText());
const value = obj.data.BTC.ohlc.l;
console.log(value)
}
I think that when this script is run, you can see your expected value in the log.
I'm trying to fetch some data to a google sheet. The code goes like this:
function ls() {
var url = "https://api.loanscan.io/v1/interest-rates?tokenFilter=BTC";
var params = {
"contentType": "application/json",
"headers":{"x-api-key": "KEY",
},
};
var json = UrlFetchApp.fetch(url, params);
var jsondata=json.getContentText();
var page = JSON.parse(jsondata);
Logger.log page
return page;
}
The Logger.log gives the correct Data, as you can see in the next link.
Log
However when I run the function in the google sheet, it returns a blank cell.
Please help, thank you.
You should write the Logger.log(page); with parentheses, because sheets is probably hanging on your Logger statement.
I don't have an API key for that site, but I do get the forbidden error response from the API when using =ls() in a cell in sheets.
I want to call the Mandrill API in order to have tagged template data populate in my google sheet labeled test, but I can't seem to get it working (spent hours I am a beginner). Below is my code
function mandrill() {
var from = "https://mandrillapp.com/api/1.0/tags/list.json";
var key = "*************";
var params = {
"return" :[
"time",
"sent",
"opens",
"clicks",
],
"sheet": "test"
}}
I followed the official docs but I can't seem to get it working. Any help at all would be greatly appreciated.
https://mandrillapp.com/api/docs/tags.JSON.html#method=all-time-series
The Apps Script reference specifies how to use the fetch() method to retrieve your request from Mandrill.
https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app
Please use
UrlFetchApp.fetch(url, options);
You need to JSON.stringify() the JSON request for the Mandrill /urls/list.json and pass it as a payload into the options of the fetch request. According to the documentation you provided, the correct JSON request is:
{
"key": "example key"
}
Subsequently, you can proceed your response and use it to populate the sheet. Please refer to:
https://developers.google.com/apps-script/reference/url-fetch/http-response
Then you need to parse the answer to JavaScript object, you need to parse it before accessing the content with the method getContentText(). You can then access the entries of interest and populate your sheet with it:
In summary, your code to retrieve the response from Mandrill should look like this:
function mandrill() {
var from = "https://mandrillapp.com/api/1.0/tags/list.json";
var params = {
"key": "*************"
};
var payload = JSON.stringify(params);
var options = {
'method': 'post',
'payload': payload,
'contentType' : 'application/json'
};
var response = UrlFetchApp.fetch(url, options);
var data = JSON.parse(response).getContentText());
var ss=SpreadsheetApp.getActiveSpreadsheet().getSheetByName("test");
sheet.getRange("A1").setValue(data['sent'])
}
I'm trying to pull data from the following sample web page using Google Apps Script:
url = http://www.premierleague.com/players/2064/Wayne-Rooney/stats?se=54
using, UrlFetchApp.Fetch(url)
The problem is when I use UrlFetchApp.Fetch(url) to do that, I don't get the page information defined by the 'se' parameter in the url. Instead, I get the information on the following URL because it looks like the 'se=54' page is asynchronously loaded:
http://www.premierleague.com/players/2064/Wayne-Rooney/stats
Is there any way to pass the parameter 'se' some other way? I was looking at the function and it allows the specification of 'options', as they are referred to, but the documentation on the topic is very limited.
Any help would be most appreciated. Many thanks
Tommy
Go to that website in your browser and open the developer tools (F12 or ctr-shift-i). Click on the network tab and reload the page with F5.
A list of requests will appear. At the bottom of the list you should see the asynchronous requests made to fetch the information. Those requests get the data in json form from footballapi.pulselive.com.
You can do the same thing in apps script. But you have to send a correct "origin" header line or your request gets rejected.
Here is an example.
function fetchData() {
var url = "http://footballapi.pulselive.com/football/stats/player/2064?comps=1";
var options = {
"headers": {
"Origin": "http://www.premierleague.com"
}
}
var json = JSON.parse(UrlFetchApp.fetch(url, options).getContentText());
for(var i = 0; i < json.stats.length; i++) {
if(json.stats[i].name === "goals") Logger.log(json.stats[i]);
}
}
Please try the following solution:
var options =
{
"method" : "GET",
"followRedirects" : true,
"muteHttpExceptions": true
};
var result = UrlFetchApp.fetch(url, options);