UrlFetchApp.fetch() doesn't seem to change user agent - google-apps-script

Trying to grab data from a website using Google Apps Script to put it directly into a spreadsheet. The fetch does not seem to be working, where the Python requests equivalent works just fine.
Python code:
page = requests.get("someurl?as_data_structure", headers={'user-agent':'testagent'})
GAS code:
var page = UrlFetchApp.fetch("someurl?as_data_structure", headers={'user-agent':'testagent'});
The only required header is the user-agent, and the error I am getting from the GAS code is what I would usually get from the Python code if I hadn't included the header. I am new to js but as far as I know this is the proper way to do it..?
EDIT:
Now got the headers in the right place but the issue persists, exactly the same error as before.
var options = {"headers": {"User-Agent": "testagent"}};
var page = UrlFetchApp.fetch("someurl?as_data_structure", options);

Star ★(on top left) the issue here for Google developers to prioritize the issue.
Google doesn't always document it's restrictions(Annoying?). One such restriction is changing the user agent. It's fixed to
"User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script)"
You can't change it.
Sample Test:
function testUrlFetchAppHeaders() {
var options = {
headers: {
'User-Agent':
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
},
};
var fakeRequest = UrlFetchApp.getRequest(
'https://www.httpbin.org/headers',
options
);//providing fake assurance
var realRequest = UrlFetchApp.fetch(
'https://www.httpbin.org/headers',
options
);//like a wrecking ball
Logger.log({ fake: fakeRequest, real: realRequest });
}
Sample Response:
{
"fake": {
"headers": {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
},
"method": "get",
"payload": "",
"followRedirects": true,
"validateHttpsCertificates": true,
"useIntranet": false,
"contentType": null,
"url": "https://www.httpbin.org/headers"
},
"real": {
"headers": {
"Accept-Encoding": "gzip,deflate,br",
"Host": "www.httpbin.org",
"User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script)"
}
}
}
getRequest(url)
Returns the request that would be made if the operation was invoked.
This method does not actually issue the request.
Neither does it accurately return the request that would be made.

the headers belong into the options:
var options = {"headers": {"User-Agent": "testagent"}};
var page = UrlFetchApp.fetch("someurl?as_data_structure", options);

Related

Google App Script: GAS return 403 but browser requests still work

I'm using google app script to monitoring URL is working then writing log in excel after every 30 mins:
try {
var options = {
"method" : "GET",
'headers': {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36'
},
'muteHttpExceptions': true
};
var url_trimmed = url.trim();
var response = UrlFetchApp.fetch(url_trimmed, options);
Logger.log(response.getContentText());
var result = response.getResponseCode();
return result;
}
catch (err) {
return [EXCEPT_ERR_CD, Utilities.formatString('statusCode: %s、Error: %s)', EXCEPT_ERR_CD, err)];
}
}
But when I check Excel result file, result sometimes return responseCode 200, and sometimes return 403. Immediately when responseCode returns 403, I check it in browser but site is working fine.
I tried add header parameter but it doesn't work.
I dont have any idea, I think maybe the error from the site but dont really know causes and how to fix it.

signin-oidc flow throws 500 in chrome but not IE

I have a netcoreapp2.1 webapp which is presenting an issue in Google Chrome, but not Edge or IE, during the OIDC flow redirect. It simply stops at http://localhost:5000/signin-oidc with this response received:
General:
Request URL: http://localhost:5000/signin-oidc
Request Method: POST
Status Code: 500 Internal Server Error
Remote Address: [::1]:5000
Referrer Policy: no-referrer-when-downgrade
Response Headers:
Content-Length: 0
Date: Tue, 04 Aug 2020 09:49:32 GMT
Server: Kestrel
Request Headers:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 2148
Content-Type: application/x-www-form-urlencoded
Host: localhost:5000
Origin: null
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: cross-site
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36
Form Data:
code: Ry9Pec...rdG1U0SB
scopes: resource.WRITE+openid+resource.READ
state: CfDJ8JZ...oESbUvQ
id_token: eyJhbG...9sXMEhs
When I visit localhost:5000 in chrome (incognito or not) the code never reaches the OnTicketReceived event, whereas when fired from IE, Edge etc it does, and proceeds just fine.
This is the startup class:
public class Startup
{
private AppSettings _appSettings;
private IConfiguration _config;
public Startup(IConfiguration configuration)
{
_config = configuration;
_appSettings = _config.Get<AppSettings>();
}
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
services.Configure<AppSettings>(_config);
services.AddSingleton<IAPIRepository, APIRepository>();
services.AddSingleton<IUserRepository, UserRepository>();
services.AddSingleton<INavigationRepository, NavigationRepository>();
services.UseOpenIDConnectMiddleware(new OpenIDConnectMiddlewareOptions
{
BaseUrl = _appSettings.API.BaseUrl,
AppName = _appSettings.AppName,
ClientId = _appSettings.API.ClientId,
ClientSecret = _appSettings.API.ClientSecret,
Secure = !_appSettings.Local
});
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
app.UseAuthentication();
if (!_appSettings.Local)
{
app.UseGlobalLoginMiddleware();
}
if (_appSettings.FeatureFlags["ProcessRedirectionRules"])
{
app.UseRedirectUserMiddleware(_appSettings.RedirectionRules);
}
app.UseMvc();
app.UseProtectHTMLRouteMiddleware();
if (_appSettings.Local)
{
app.UseDeveloperExceptionPage();
app.UseWebpackDevMiddleware(new WebpackDevMiddlewareOptions {
ProjectPath = Path.Join(Directory.GetCurrentDirectory(), "../UI"),
ConfigFile = "node_modules/#vue/cli-service/webpack.config.js",
HotModuleReplacement = true
});
}
else
{
app.UseDefaultFiles();
app.UseStaticFiles();
}
app.UseJavascriptVarMiddleware(new JavascriptSettingsMiddlewareOptions
{
FileName = "env.js",
ObjectName = "__env",
Settings = new Dictionary<string, string>
{
{ "insightsKey", _appSettings.ApplicationInsights.InstrumentationKey },
{ "environment", _appSettings.Environment },
{ "gatekeeperBaseUrl", _appSettings.Gatekeeper.BaseUrl }
}
});
app.UseGlobalSignoutMiddleware(new GlobalSignoutMiddlewareOptions
{
GatekeeperLogoutUrl = _appSettings.API.LogoutUrl
});
app.UseHTMLRouteFallback(new HTMLRouteFallbackMiddlewareOptions
{
Local = _appSettings.Local,
Path = Path.Join(Directory.GetCurrentDirectory(), "../UI/dist")
});
}
}
I have cleaned the localhost and localhost:5000 cookies from the browser.
A quick guess would be that it is a SameSite cookie issue that different browsers handle differently due to various bugs.
This article might give you a starting point:
How To Prepare Your IdentityServer For Chrome's SameSite Cookie Changes - And How To Deal With Safari, Nevertheless
Also, your Configure method looks very odd. Because the ordering of the App.Use statements matter and usually you would put the app.UseMvc(); last in that method. Each incoming request will pass through those middlewares that the App.UseXXX adds and if the ordering is not correct then you will have issues.

Is there a JavaScript function to figure out what type of device you're on?

I have a site that I want to behave slightly differently on a computer than on a mobile device (phone or tablet) and I want a function that I can call to figure out what type of device I'm on.
Is there like window.getDevice or document.getDevice or something like that?
I think that there isn't a way to get the device name.
You can use:
window.navigator.userAgent to get info about platform in use and browser.
and
window.screen to get an object with available width and available height that tell you the real width and height of device in use.
I have a suggestion, use this website's api https://www.iplocinfo.com/, it will help you to figure out the visitor information (the browser type with version, device type and version, mobile, desktop, tablet or bot .... and the ip address and location and privacy in the same time):
example:
const getApiData = async () => {
let ip_address = "155.45.44.44";
let api_key = "uicVcaMqPM_zumdStwSQrA"; //Add your api for testing
let user_agent = "Mozilla/5.0 (Linux; Android 12; SM-S906N Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.119 Mobile Safari/537.36";
let endPoint = `https://www.iplocinfo.com/api/v1/${ip_address}?apiKey=${api_key}&user-agent=${user_agent}`
let response = await fetch(endPoint, {
method: 'GET',
headers: {
"Content-type": "application/json",
},
})
let data = await response.json();
console.log(data);
};
getApiData();
and you can get the ip address easy by fetch it from this api example:
const getIpAddress = async () => {
let endPoint = `https://www.iplocinfo.com/ip`;
let response = await fetch(endPoint, {
method: 'GET',
headers: {
"Content-type": "application/json",
},
})
let data = await response.json();
console.log(data);
};
getIpAddress();

Can not post JSON to server in titanium appcelerator

Below is my specifications I am running:
Operating System Name = Mac OS X Version
= 10.9.5 Architecture = 64bit
Titanium CLI CLI Version = 3.4.1 Titanium SDK SDK
Version = 3.5.0.GA Target Platform =
android
Ex:-> Node.js Node.js Version = 0.10.24 npm Version
= 1.3.21
Here is my code:
var url = "http://192.168.1.100/test.php";
var xhr = Titanium.Network.createHTTPClient();
xhr.onerror = function(e) {
swim.lib.err('' + e.error);
alert('HTTP ERR...');
};
xhr.onload = function(e) {
Ti.API.info(this.responseText);
alert('success');
};
xhr.open("POST", url);
//post/get and your URL
xhr.setRequestHeader("content-type", "application/json");
var param = {
"user" : {
"email" : "someone#email.com",
"password" : "secure"
}
};
xhr.send(JSON.stringify(param));
What does your server expect? I just tested your code replacing the url with http://requestb.in/157oi2x1 and that seems fine, as you can see on http://requestb.in/157oi2x1?inspect:
Headers
Via: 1.1 vegur
Connect-Time: 1
User-Agent: Appcelerator Titanium/3.5.0 (iPhone Simulator/8.2; iPhone OS; en_US;)
Host: requestb.in
Total-Route-Time: 0
X-Requested-With: XMLHttpRequest
Content-Type: application/json
Accept-Language: en-us
X-Titanium-Id: d6caf6df-6a27-49b0-ad38-1c0d64356393
X-Request-Id: 257ddc46-066d-4c1d-95d6-cf6a642dbeab
Content-Length: 58
Accept: */*
Connection: close
Accept-Encoding: gzip, deflate
Body
{"user":{"email":"someone#email.com","password":"secure"}}

Response status : 404 using HttpClient in wp8.0

I had this code working but now suddenly it does not return
"Response status code does not indicate success:404()"..
what ever may be the URL it doesn't matter..
below is the code:
protected async void getstring(string url)
{
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0");
try
{
var html = await client.GetStringAsync(url);
extractData(html);
}
catch (HttpRequestException e)
{
MessageBox.Show(e.Message);
arrivalTime.Text = e.Message;
}
//time.Text = html;
}