How to read MediaWiki API JSON response - json

I am trying to search images on Wikimedia Commons, using MediaWiki API. Here is my requested URL with search params:
https://commons.wikimedia.org/w/api.php?action=query&list=allimages&format=json&aifrom=Dada
I am succeed to get response in JSON format, but I could not read it programmatically because:
No 'Access-Control-Allow-Origin' header is present on the requested
resource.
Any advice?
UPDATE:
I have added one more param to the url: callback=JSON_CALLBACK, which transforms response to jsonp format. Now it possible to use angular $http.jsonp() method also.

use jsonp1 as dataType to prevent the "No 'Access-Control-Allow-Origin' header is present on the requested resource." error. Then it works :
$.ajax({
dataType: 'jsonp',
url : 'https://commons.wikimedia.org/w/api.php?action=query&list=allimages&format=json&aifrom=Dada',
success : function(json) {
json.query.allimages.forEach(function(item) {
$('<img/>', { src : item.url }).appendTo('#images');
})
}
})
Here I add each image to a #images <div> just for demonstration purposes.
demo -> http://jsfiddle.net/52g2Lazw/
1) JSONP stands for “JSON with Padding” and it is a workaround for loading data from different domains. It loads the script into the head of the DOM and thus you can access the information as if it were loaded on your own domain, thus by-passing the cross domain issue cite.

If you are accessing the Wikimedia Commons API from a Wikimedia wiki, you can use the origin parameter of the API and this way make the API setting CORS headers. E.g. on en.wikipedia.org, you could access the Commons API this way:
$.get('https://commons.wikimedia.org/w/api.php?' +
$.param({
format: 'json',
action: 'query',
list: 'allimages',
aifrom: 'Dada',
origin: location.protocol + '//' + location.host
})).done(function() { /*...*/ });
It is generally safer to use JSON (a pure data format) than loading and executing JavaScript (JSONP) file that could, in theory, do evil with your visitors. I would probably set up a proxy server for this purpose, instead of using JSONP. A simple web search for set up a proxy json may result in plenty of useful results.

Related

How do i allow a CORS requests in my google script?

I want to post my contact form to my google script that will send an e-mail to me. I use the following code:
var TO_ADDRESS = "example#gmail.com"; // where to send form data
function doPost(e) {
var callback = e.parameter.callback;
try {
Logger.log(e); // the Google Script version of console.log
MailApp.sendEmail(TO_ADDRESS, "Contact Form Submitted",
JSON.stringify(e.parameters));
// return json success results
return ContentService
.createTextOutput(callback+
JSON.stringify({"result":"success",
"data": JSON.stringify(e.parameters) }))
.setMimeType(ContentService.MimeType.JSON);
} catch(error) { // if error return this
Logger.log(error);
return ContentService
.createTextOutput(callback+JSON.stringify({"result":"error",
"error": e}))
.setMimeType(ContentService.MimeType.JSON);
}
}
When i try to post to the google script url, i get the following error:
Access to XMLHttpRequest at
'https://script.google.com/macros/s/~~myscriptid~~/exec' from origin
'http://localhost:4200' has been blocked by CORS policy: Response to
preflight request doesn't pass access control check: No
'Access-Control-Allow-Origin' header is present on the requested
resource.
I have no clue how to add the CORS-filter to my google script.
I know the script is working i have tested it with this plugin:
https://chrome.google.com/webstore/detail/allow-control-allow-origi/nlfbmbojpeacfghkpbjhddihlkkiljbi
Late answer, but totally working...
To pass data from appscripts to another website, just use mime type JAVASCRIPT on appscripts side, like so:
doGet(e){
return ContentService
.createTextOutput(e.parameter.callback + "(" + JSON.stringify(YOUR OBJECT DATA HERE)+ ")")
.setMimeType(ContentService.MimeType.JAVASCRIPT);
}
And on the front end access it as:
<script>
var url = "https://script.google.com/macros/s/AKfy*****ACeR/exec?callback=loadData";
// Make an AJAX call to Google Script
jQuery.ajax({
crossDomain: true,
url: url,
method: "GET",
dataType: "jsonp"
});
// log the returned data
function loadData(e) {
console.log(e);
}
</script>
This works without any CROB/ CROS headache
After a lot of hard work, the only solution which worked for me:
In Google Apps Script
function doPost(e) {
return ContentService.createTextOutput(JSON.stringify({status: "success", "data": "my-data"})).setMimeType(ContentService.MimeType.JSON);
}
In JavaScript
fetch(URL, {
redirect: "follow",
method: "POST",
body: JSON.stringify(DATA),
headers: {
"Content-Type": "text/plain;charset=utf-8",
},
})
Note the attribute redirect: "follow" that is very important;
Quick answer
You (frontend developer) can't fix cors error from remote server. Only the owner of the remote server (google app script server) could do it.
Workaround 1 (GET)
Use only GET method in app script. Get method will not throw CORS errors, no matter where you consume it from: csr, spa, frontend, react, angular, vue, jquery, pure javascript, etc
Workaround 2 (Backend)
If you are in the backend server (java, php, c#, node, ruby, curl, etc) not in the frontend (browser, react, angular, vue), you could consume any method published on google apps script.
CORS don't affect when the consumption is at the backend layer
So if only use get endpoints are not an option for you, you could use another server language (java, nodejs, php, etc) to consume the Post google app script, and return that information to your web
Explanation
Let's imagine this script with 02 methods deployed as web in google app script
function doGet(e) {
var response = {
"code": 200,
"message": "I'm the get"
};
return ContentService.createTextOutput(JSON.stringify(response)).setMimeType(ContentService.MimeType.JSON);
}
function doPost(e) {
var response = {
"code": 200,
"message": "I'm the post"
};
return ContentService.createTextOutput(JSON.stringify(response)).setMimeType(ContentService.MimeType.JSON);
}
and url like this after the deployment:
https://script.google.com/a/utec.edu.pe/macros/s/AKfy\*\*\*\*\*\*eo/exec
In the backend
You could consume the POST and GET methods without any problems with any language: java, nodejs, python, php, c#, go , etc and/or with any http client like postman, insomnia, soapui, curl, etc
In the frontend (js in the browser)
I was not able to consume the POST method. I tried with jsonp and other crazy attempts and the error was the same:
Cross-Origin Request Blocked: The Same Origin Policy disallows
reading the remote resource at
https://script.google.com/a/utec.edu.pe/macros/s/AKfy***A4B***eo/exec?foo=bar
(Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
So for any reason, the google server don't allow us to use POST operations from javascript side (2021)
In the frontend : GET Method
Only GET method worked for me. I will assume that google configuration at server layer has some CORS permission only for GET method.
The following ways worked for me, from a simple js to an advanced frameworks like react, vue or angular:
axios
const axios = require('axios');
axios.get('https://script.google.com/a/acme.org/macros/s/AKfy***A4B***eo/exec').then(resp => {
console.log(resp.data);
});
$.getJSON
$.getJSON('https://script.google.com/a/acme.org/macros/s/AKfy***A4B***eo/exec?foo=bar', function(result) {
console.log(result);
});
XMLHttpRequest
var xmlhttp = new XMLHttpRequest();
var theUrl = "https://script.google.com/a/acme.org/macros/s/AKfy***A4B***eo/exec?foo=bar";
xmlhttp.open("GET", theUrl);
xmlhttp.setRequestHeader("Content-Type", "application/json;charset=UTF-8");
xmlhttp.send();
CORS : Cross-origin resource sharing
A lot of developers don't understand what is CORS. It is not easy to understand. Commonly the developer fix the error at the server layer and don't invest time (or don't let him) to understand what CORS is:
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
https://portswigger.net/web-security/cors
https://youtu.be/4KHiSt0oLJ0
If you don't have time, check my definition, extreme summary bordering on wrong:
CORS is a protection offered by trusted browsers to avoid that a web acme.com can load in the background(ajax/js) an http resource from another domain like hacker-api.com/foo/bar
But if acme.com and hacker-api.com/foo/bar are developed by you and/or hacker-api.com/foo/bar is designed to be consumed by any web of the world, you could fix it at server layer
How to fix CORS errors?
Are very common and simple to control with a few lines in the server if the server belongs to us, but since we don't have control over the server(google), we can not do anything at this layer.
Here some samples of CORS configuration to allow consumption from webs is the backend server belongs to you:
java sample:
//only http://acme.com could consume my api
#CrossOrigin("http://acme.com")
#RequestMapping(method = RequestMethod.GET, "/{id}")
public Account retrieve(#PathVariable Long id)
nodejs sample:
//only http://localhost:8080 could consume my api
var corsOptions = {
origin: 'http://localhost:8080',
optionsSuccessStatus: 200 // For legacy browser support
}
app.use(cors(corsOptions));
//any web could consume my api
origin : "*"
I ran into the same issue while trying to create an application that logs data and retrieves log sections to/from a google sheet through Google Apps Script using Get and Post requests.
I did find a solution that may or may not be helpful to some people.
From the Google Docs:
There are two types of CORS requests: simple and preflighted. A simple
request can be initiated directly. A preflighted request must send a
preliminary, "preflight" request to the server to get permission
before the primary request can proceed. A request is preflighted if
any of the following circumstances are true:
It uses methods other than GET, HEAD or POST. It uses the POST method
with a Content-Type other than text/plain,
application/x-www-form-urlencoded, or multipart/form-data. It sets
custom headers. For example, X-PINGOTHER.
All I did was change the content type of my Get and Post requests
var request = new window.XMLHttpRequest();
request.open(opts.method, opts.url, true);
request.setRequestHeader("Content-Type", "text/plain");
And within the google script, parse to JSON to be used
function doPost(e) {
const d = JSON.parse(e);
...
As far as I understood you have application to be run on custom domain. And it should access script on google cloud.
The bad news: there are no way to skip CORS check on your application side(until request is simple that I believe is not your case).
You should specify Access-Control-Allow-Origin on Google Cloud side:
Cloud Storage allows you to set CORS configuration at the bucket level only. You can set the CORS configuration for a bucket using the gsutil command-line tool, the XML API, or the JSON API. For more information about setting CORS configuration on a bucket, see Configuring Cross-Origin Resource Sharing (CORS). For more information about CORS configuration elements, see Set Bucket CORS.
You can use either of the following XML API request URLs to obtain a response from Cloud Storage that contains the CORS headers:
storage.googleapis.com/[BUCKET_NAME]
[BUCKET_NAME].storage.googleapis.com
If this does not help for any reason you will need to get your own server working as a proxy:
your client application <-> your backend that returns Access-Control-Allow-Origin <-> google cloud
Well after several attempts, I was able to send the data through a web app form in angular 8.
The solution is simple, within "HttpClient.post" you can enter a third parameter to establish an HTTP connection header this for "https://script.google.com" may not be correct and will end with an http connection failed by CORS security.
Just don't add the HTTP connection header as the third parameter of HttpClient.post
const object = {
title: 'Prices',
phone: '999999999',
full_name: 'Jerson Antonio',
email: 'test#example.com',
message: 'Hello, .......'
};
return this.http.post(this.API_REST_FORM, JSON.stringify(object));
In App script always use New deployment to deploy the script.
Otherwise it will use old script and you will get CORS error
The CORS error is most probably caused by a fatal error in your Google Apps Web App script. In this case the Google error handling system displays a human-readable HTML page that does not contain CORS headers.
In my case I got the following error page:

How can I stop Google crawling webservice URLs?

I am finding that GoogleBot is crawling webservice URLs which are referenced in JavaScript/AJAX code. The URL is already in robots.txt as an exclusion, but Google no longer seems to obey robots.txt when determining what to crawl - it only seems to use it to know what not to index.
Thankfully these service URLs only return data rather than performing actions but it's messing up the statistics we collect which is highly undesirable. I cannot personally see how Google is even finding out the URL of the webservice unless it crawls arbitrary strings in Javascript code (which seems unlikely?).
For some URLs this also results in me getting LOTS of Elmah error messages from the website which say:
System.InvalidOperationException: Request format is unrecognized for URL unexpectedly ending in '/GetShortlists'." ... as Google tries to GET the URL when it only supports POST.
The code it's finding the URLs in is as follows:
function GetShortlistsForUser() {
$.ajax({
type: "POST", url: "/WebService/WebService.asmx/GetShortlists",
contentType: "application/json; charset=utf-8",
dataType: "json",
success: function (data) { /*--CUT--*/ });
}
});
So should I obfuscate the URL somehow by perhaps replacing the slashes, or is there a better way to stop these getting crawled?
(1) Try break the url format in your javascript codes, e.g,
var breaker="x/G";
......
url: "/WebServic"+"e/WebService."+"asm"+breaker+"etshortlists",
since Google may use regex to determine which part is url... (I am not sure if this could prevent crawlers, but if it works, you don't need to break it to this extend as it also breaks code reading experience.)
(2) On your server, Google crawler typically use customized agent string, so you can deny it (or ignore it).

Cross-domain $http request AngularJS

I have simple website I'm building with AngularJS which calls an API for json data.
However I am getting Cross domain origin problem is there anyway around this to allow for cross domain requests?
Error:
XMLHttpRequest cannot load http://api.nestoria.co.uk/api?country=uk&pretty=1&action=search_listings&place_name=soho&encoding=json&listing_type=rent. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http:// localhost' is therefore not allowed access.
searchByPlaceName: function() {
var url = baseurl+'country=uk&pretty=1&action=search_listings&place_name=london'+encoding+type;
return $http.get(url);
}
It seems that api.nestoria.co.uk does not allow CORS. It has to set the Access-Control-Allow-Origin header itself -- you have no direct control over that.
However, you can use JSONP. That site allows it via the callback query parameter.
$http.jsonp(baseurl+'country=uk&pretty=1&action=search_listings&place_name=london'
+encoding+type + "&callback=JSON_CALLBACK")
Install Fiddler. Add a custom rule to it:
static function OnBeforeResponse(oSession: Session)
{
oSession.oResponse.headers.Add("Access-Control-Allow-Origin", "*");
}
This would allow you to make cross domain requests from localhost. If the API is HTTPS make sure you enable 'decrypt HTTPS traffic' in fiddler.
Reference
-------------------- UPDATE
The response you are getting is JSON. Specifying JSONP as datatype would not work. When you do specify JSONP the return should be a function not JSON object.
JSONP

Sendgrid API - JSON call

I'm trying to receive data from SendGrid API
$.ajax({
type:'GET',
url:"https://sendgrid.com/api/bounces.get.json",
data: {api_user:'username',api_key:'userkey',date:1},
success: function(data){
console.log(data)
},
crossDomain: true,
dataType: 'jsonp',
error:function(a,b,c){
console.log(a);
}
});
Console shows:
Object { readyState=4, status=200, statusText="success"}
parsererror
Error: jQuery17208301184673423685_1374648217666 was not called
Where is the bug or issue ?
The issue is that SendGrid does not support jsonp.
Unfortunately, switching to plain JSON will not work either, as SendGrid has no CORS headers and browsers will not allow you to access the pages. In short you cannot make AJAX requests dorectly to SendGrid.
However, generally this is for the better as all SendGrid endpoints require authentication and having your username and password in an AJAX request would allow users to take them and then use them to send email.
To get these stats on the frontend, you'll need a server to get them and output them on your domain or a domain with CORS allowances.
Here comes a one click solution!
Deploy your instance of SendGrid Proxy to Heroku
Use {your-sendgrid-proxy}.herokuapp.com instead of api.sendgrid.com
Done (Really)
How it works:
It creates a node powered http proxy using express-http-proxy
It adds needed headers such as Authorization and Content-Type
It overrides Access-Control-Allow-Origin to * to make your browser CORS warning free
See how the magic is working. Feedback is welcome!

Play Framework Cross Domain Web Service

I've been trying to create a Web Service with play framework. I've searched some tutorial and resource about this and end up using the renderJSON() method to provide JSON RESTful web service.
But, there seems to be a problem with that when I try to consume the web service with JQuery. If I use JSON, it fails with this error Origin http://localhost is not allowed by Access-Control-Allow-Origin. Which seems to be a cross-domain problem. Then I try to use JSONP as the datatype when JQuery trying to consume the web service, again there's an parseError problem.
After some research, I found that to provide a web service to the JSONP request. We need to provide something like a callback function. So it's not enough to return only the JSON object to the client. If I'm not mistaken, The format should look something like {#callback-function}{#JSON-data}
How to make a Cross-Domain web service in Play Framework? Is there any workaround for this?
Is there anyway to keep the JSON format of a/a list of object to a string instead of render the object as JSON directly as done by calling renderJSON()?
Here is what I did. In my routes I set the format of the response to javascript with:
GET /api/projects.json Api.projects(format:'json')
Then for the controller in Api I used flexjson and did this:
public static void projects(String p) {
JSONSerializer json = new JSONSerializer();
List<Project> projects = Project.all().fetch(3);
renderJSON(p + '(' + json.serialize(projects) + ')');
}
So now jsonp calls can be made by hitting /api/projects.json?p=whatever and the return will be:
whatever([{"your":"json"}])
You can also use flexjson to only serialize the parts of the object that you want to expose.
Testing it with jquery:
$.ajax({
url: "http://api/projects.json",
dataType: "jsonp",
jsonp : 'p',
crossDomain : true,
success: function(data){
console.log(data);
}
});
This is working great but you can use Gson object directly instead of the flexjson object.