I am using .ajax function to read data from an external page .
I am not able to read the page and I get the exception as Access Control Allow origin.
i want to show the data of another page in my page I even set crossdomain:true but it dint help
$.ajax({
type: "GET", url: myurl,
dataType: "html",
crossDomain:true,
success:parsehtml});
How to read an external html page using jquery ?
The underlying Javascript engines in most modern browsers don't permit Cross-Domain transactions. You will have to use a server-side language such as PHP to do this. Mozilla has a fairly decent description of this issue.
There is one solution without using any server side technologies. Please see this
http://net.tutsplus.com/tutorials/javascript-ajax/quick-tip-cross-domain-ajax-request-with-yql-and-jquery/
You may have to process the returned output to suit your need.
Related
I'm trying to write a spider that will automatically log in to this website. However, when I try using scrapy.FormRequest.from_response in the shell I get the error:
No <form> element found in <200 https://www.athletic.net/account/login/?ReturnUrl=%2Fdefault.aspx>
I can definitely see the form when I inspect element on the site, but it just did not show up in Scrapy when I tried finding it using response.xpath() either. Is it possible for the form content to be hidden from my spider somehow? If so, how do I fix it?
The form is created using Javascript, it's not part of the static HTML source code. Scrapy does not parse Javascript, thus it cannot be found.
The relevant part of the static HTML (where they inject the form using Javascript) is:
<div ng-controller="AppCtrl as appC" class="m-auto pt-3 pb-5 container" style="max-width: 425px;">
<section ui-view></section>
</div>
To find issues like this, I would either:
compare the source code from "View Source Code" and "Inspect" to each other
browse the web page with a browser without Javascript (when I develop scrapers I usually have one browser with Javascript for research and documentations and another one for checking web pages without Javascript)
In this case, you have to manually create your FormRequest for this web page. I was not able to spot any form of CSRF protection on their form, so it might be as simple as:
FormRequest(url='https://www.athletic.net/account/auth.ashx',
formdata={"e": "foo#example.com", "pw": "secret"})
However, I think you cannot use formdata, but instead they expect you to send JSON. Not sure if FormRequest can handle this, I guess you just want to use a standard Request.
Since they heavily use Javascript on their front end, you cannot use the source code of the page to find these parameters either. Instead, I used the developer console of my browser and checked the request/response that happened when I tried to login with invalid credentials.
This gave me:
General:
Request URL: https://www.athletic.net/account/auth.ashx
[...]
Request Payload:
{e: "foo#example.com", pw: "secret"}
Scrapy has a JsonRequest class to help with posting JSON. See here [https://docs.scrapy.org/en/latest/topics/request-response.html]
So something like the below should work
data = {"password": "pword", "username": "user"}
# JSON POST to API login URL
return JsonRequest(
url=url,
callback=self.after_login,
data=data,
)
My company is launching a Marketo campaign landing page to promote a microsite/testing tool I have built. I have a basic understanding of Marketo, but that is it.
We want to include some stats on the page using live data from my app, I can easily build an API to get this data, and based on what I have read I can show it in the Marketo landing page using a custom code filed.
I am trying to find proof that the code field can handle JSONP, but I can't seem to, I am hoping someone here could validate that it does.
Essentially I would want to put something like this in the code block:
<script>
// this is pseudo-code
function show_stats(json){
$("#holder").text("stuff from json")
}
$( document ).ready(function() {
$.ajax({
url: "https://myapp.mycompany.com",
dataType: "jsonp",
jsonpCallback: "show_stats"
});
});
</script>
<p id="holder"></p>
If by code field you mean the custom HTML element in the landing page editor - the docs indicate that you can put any scripts in there. I haven't tried exactly your pseudo-code, but anything I've tried putting in one has rendered as expected.
Also, at the template editor level of a Marketo landing page template, it's just a normal HTML document with some extra classes thrown in, so you can put whatever code you want in there - so I don't see any reason why that wouldn't work.
The only thing that you might run into trouble with JSONP stuff, is possibly cross-domain CORS issues?
Lets say I have example.html and inside that i have a code like
<iframe src="x.com" id="x"></iframe>
from x.com, I would like to get everything inside
<div class="content">...</div>
into example.html inside
<div class="xCodes">ONTO HERE</div>
So I tried to get the elements inside x.com to show up on example.html and I heard it's not possible to access them for cross domain problems.
I was wondering if there was another way to retrieve HTML tags from x.html into example.html
Maybe without using <iframe />??
Sourced from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/
$('.xCodes').load('http://x.com/x.html');
OR
$.ajax({
url: 'http://x.com/x.html',
type: 'GET',
success: function(res) {
var data = $(res.responseText).find('.content').text();
$('.xCodes').html(data);
}
});
If I understand correctly you want to rip the content from a DIV on one site and display it on another. There are several issues with this, but I'll focus on the technological aspect and assume you are acting in good faith with pulling the content.
The real issue you're running up against here is that you don't have access to DOM elements of pages that haven't loaded yet. As such you need to tell the browser to load the data for that page so that you can access the elements that should have loaded on the page and then pull the information out. JQuery has a nice little method to help with that called .load() (http://api.jquery.com/load/).
As an important side note however you can't do this as all modern broswers forbid cross site access in such a manner:
From the JQuery .load() page:
Additional Notes:
Due to browser security restrictions, most "Ajax" requests are subject to the same origin policy; the request can not successfully retrieve data from a different domain, subdomain, or protocol.
And check out:
http://en.wikipedia.org/wiki/Same_origin_policy
One more bit of warning. If you don't control the code on the other site you are potentially exposing yourself to serious security issues so only do this in situations where you have control of the other site or for some reason have absolute faith in that site. Alternatively you should try to, if available, use APIs for the sites/services you are trying to get data from.
I have a problem with sending some custom headers to with a jQuery ajax JSON (or JSONP) request.
Code looks like this:
$.ajax({
beforeSend: function(xhr) {
xhr.setRequestHeader("X-VER", VER);
xhr.setRequestHeader("X-TIMESTAMP", now);
},
type: 'GET',
data: null,
url: site_uri,
dataType: 'jsonp',
success: function(msg){$(selector).html(msg);}
});
Firebug shows no custom headers. In addition the url changes with jsonp (can I change this?)
Edit:
I found that it probably violates http://en.wikipedia.org/wiki/Same_origin_policy but have no idea how to get it fixed (the script can't be on the same domain).
Thanks for any help
If you have server access, a solution would be HTTP access control modification. Otherwise, probably the best idea is some kind of reverse proxy solution, I have no idea how good of support access control has outside of Firefox.
jsonp isn't XHR (you are simply adding script elements to the head), emulating XHR. Hence, you can't set custom headers.
I'm using the flickr api to create a simple gallery based on images pulled in by their tag.
The gallery is working fine in every browser except ie6. When you navigate to the page (by clicking a link) in ie6, the $.ajax success/error code blocks refuse to fire, however when the page is reloaded, or navigated to directly (by entering a url) there are no problems.
$.ajax({
type: "GET",
url: "http://api.flickr.com/services/rest/?&method=flickr.photos.search&api_key=" + api_key + "&user_id=" + user_id + "&tags=" + tags + "&format=json&jsoncallback=?",
cache: false,
dataType: "jsonp",
contentType: "application/json; charset=utf-8",
success: function(data) {
alert('good');
},
timeout: 2000,
error: function(XMLHttpRequest, textStatus, errorThrown) {
alert(textStatus);
}
});
The above code is obviously a simplified version of the actual javascript, but the bug is still present.
Any help would be greatly appreciated. This seems to be a very weird bug. Perhaps it's a caching issue?
This problem is pretty recurrent. IE caches AJAX calls. You need to append a random number or string to your call each time so that it doesn't run a cached call.
url: "http://[your url]&rand=" + Math.Random(),
Two thoughts:
Is your call within a
$(document).ready() block?
Try getting rid of the ampersand
right at the beginning of the query
string:
/?method=flickr.photos.search&...
How is the "link" triggering the AJAX load?
inline onclick="doSomething();"
inline href="javascript:doSomething();"
via jQuery $('#somelink').bind('click', doSomething);
something else?
I ask because IE6 has a known bug whereby if you have/use the javascript: protocol on a link... then use JavaScript to load a different page it requests the page but never renders it. I wonder if (depending on how your AJAX is attached) if the above bug extends to AJAX requests not just location.href requests.
Four things related to your code but not to your question:
timeout is not taken into account by $.ajax when dealing with jsonp
nor is contentType
the error callback is never called in the context of a jsonp request
You should really use the data parameter (the code would be far cleaner than with the custom url building it currently shows)
Point 1 & 3 are due to limitations in the way $.ajax is implemented. I recommend http://code.google.com/p/jquery-jsonp/ if you really need those features.
Now, apart from what karim79 pointed out, I see nothing wrong with your code. So my guess is you have something else going wrong prior to it in the function you feed to $(document).ready(). IE is far less lenient than other browsers when it comes to javascript syntax. Try putting the $.ajax call as early as possible.
Also:
Does it work in IE7?
What version of jQuery are you using?