I am writing test scripts for a website using selenium webdriver's perl bindings. The problem I am facing is mentioned below:
I am not able to find some of the web elements (tried every approach: xpath, id, css, etc.) which exist on the web page, although I can click on them as a user.
After debugging through firebug I found that there is a property called base URI. And this has a different value from one web element to another. So whenever BASE URI = URL, I am able find elements and work on them.
But if BASE URI IS NOT EQUAL TO URL I am unable to find web elements.
Here is a sample code for successfull case:
#!/usr/bin/perl -w
use strict;
use warnings;
use Time::HiRes qw(sleep);
use Test::More "no_plan";
use Test::Exception;
use TAP::Harness;
use Data::Dumper;
use Selenium::Remote::Driver;
use Selenium::Remote::WDKeys;
my $url = 'https://www.xyz.com/';
my $sel = new Selenium::Remote::Driver('browser_name' => 'firefox',
'platform' => 'VISTA');
ok(defined $sel, 'Object loaded fine...');
ok($sel->isa('Selenium::Remote::Driver'), '...and of right type');
$sel->get("$url");
is ($sel->get_title(), 'xyz','Got the right title');
my $elem = $sel->find_element('html/body/div[1]/div/header/div[1]/div/center/div/ul/li[1]/a');
ok(defined $elem, 'packageLink web element defined');
$elem->click();
In this case the DOM structure shows that base uri is the same as url.
Now for unsuccessful the case, the DOM shows the base URI as https://www.xyz.com/epg.php
In the above mentioned code, if I use the $sel->navigate(https://www.xyz.com/epg.php) command just before the find element command then the script will be able to find the elements. But it will be just a static page click and will give no response.
I am a first time user of HTML related tasks. I would appreciate any kind of help.
Thanks,
Abhishek
Related
Im trying to scrape a div class but everything I have tried has failed so far :(
Im trying to scrape the element(s):
<a href="http://www.bellator.com/events/d306b5/bellator-newcastle-pitbull-vs-
scope"><div class="s_buttons_button s_buttons_buttonAlt
s_buttons_buttonSlashBack">More info</div></a>
from the website: http://www.bellator.com/events
I tried accessing the list of elements by doing
Elements elements = document.select("div[class=s_container] > li");
but that didnt return anything.
Then i tried accessing just the parent with
Elements elements = document.select("div[class=s_container]");
and that returned two div with classname "s_container", non of which is the one I needed :<
then i tried accessing that ones parent with
Elements elements = document.select("div[class=ent_m152_bellator module
ent_m152_bellator_V1_1_0 ent_m152]");
And that didnt return anything
I also tried
Elements elements = document.select("div[class=ent_m152_bellator]");
because I wasnt sure about the white spaces but it didnt return anything either
Then I tried accessing its parent by
Elements elements = document.select("div#t3_lc");
and that worked, but it returned an element containing
<div id="t3_lc">
<div class="triforce-module" id="t3_lc_promo1"></div>
</div>
which is kinda weird because i cant see that it has that child when i inspect the website in chrome :S
Anyone knows whats going on? I feel kinda lost..
What you see in your web browser is not what Jsoup sees. Disable JavaScript and refresh page to get what Jsoup gets OR press CTRL+U ("Show source", not "Inspect"!) in your browser to see original HTML document before JavaScript modifications. When you use your browser's debugger it shows final document after modifications so it's not not suitable for your needs.
It seems like whole "UPCOMING EVENTS" section is dynamically loaded by JavaScript.
Even more, this section is asynchronously loaded with AJAX. You can use your browsers debugger (Network tab) to see every possible request and response.
I found it but unfortunately all the data you need is returned as JSON so you're going to need another library to parse JSON.
That's not the end of the bad news and this case is more complicated. You could make direct request for the data:
http://www.bellator.com/feeds/ent_m152_bellator/V1_1_0/d10a728c-547e-4a6f-b140-7eecb67cff6b
but the URL seems random and few of these URLs (one per upcoming event?) are included inside JavaScript code in HTML.
My approach would be to get the URLs of these feeds with something like:
List<String> feedUrls = new ArrayList<>();
//select all the scripts
Elements scripts = document.select("script");
for(Element script: scripts){
if(script.text().contains("http://www.bellator.com/feeds/")){
// here use regexp to get all URLs from script.text() and add them to feedUrls
}
}
for(String feedUrl : feedUrls){
// iterate over feed URLs, download each of them
String json = Jsoup.connect(feedUrl).ignoreContentType(true).get().body().toString();
// here use JSON parsing library to get the data you need
}
ALTERNATIVE approach would be to stop using Jsoup because of its limitations and use Selenium Webdriver as it supports dynamic page modifications by JavaScript so you'd get the HTML of the final result - exactly what you see in web browser and Inspector.
If anyone finds this in the future; I managed to solve it with Selenium, dont know if its a good/correct solution but it seems to be working.
System.setProperty("webdriver.chrome.driver", "C:\\Users\\PC\\Desktop\\Chromedriver\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.get("http://www.bellator.com/events");
String html = driver.getPageSource();
Document doc = Jsoup.parse(html);
Elements elements = doc.select("ul.s_layouts_lineListAlt > li > a");
for(Element element : elements) {
System.out.println(element.attr("href"));
}
Output:
http://www.bellator.com/events/d306b5/bellator-newcastle-pitbull-vs-scope
http://www.bellator.com/events/ylcu8d/bellator-215-mitrione-vs-kharitonov
http://www.bellator.com/events/yk2djw/bellator-216-mvp-vs-daley
http://www.bellator.com/events/e8rdqs/bellator-217-gallagher-vs-graham
http://www.bellator.com/events/281wxq/bellator-218-sanchez-vs-grimshaw
http://www.bellator.com/events/8lcbdi/bellator-219-koreshkov-vs-larkin
http://www.bellator.com/events/9rqguc/bellator-macdonald-vs-fitch
I need to scrape some URLs from some retailer product pages, but the specific URLs I need to get aren't in the html part of the page. The html looks like this for each of the items on which one would click to get to the page with the URL I need to grab:
<div id="name" class="hand bold" onclick="AVON.productcontrol.Go(45714);">ADVANCE TECHNIQUES Color Protection Conditioner Bonus Size</div>
I wrote the following to get URLs from the page, but since the actual URLs I need don’t seem to be stored in the page, it doesn’t get what I need:
def getUrls(URL):
"""input: product page url
output: list of urls to products
"""
connection = urllib.urlopen(URL)
dom = lxml.html.fromstring(connection.read())
selAnchor = CSSSelector('a')
foundElements = selAnchor(dom)
urlList = [e.get('href') for e in foundElements]
return urlList
Is there a way to get the link that the function after ‘onclick’ (I guess AVON.productcontrol.Go(#);) takes you to? I don’t fully understand html, and while I’ve read a bit about onclick, I can’t figure out how the function after 'onclick' works.
In order to find the URL that you are taken to on click, you need to find the JavaScript source code of the 'Go' function and read and understand it. It's buried somewhere within a tag or some JavaScript .js file that is referenced directly or indirectly by the HTML page. Happy digging!
Or: you automate the interaction with the web page with a tool like Selenium (http://docs.seleniumhq.org/) and just check where it takes you if you click.
Im trying to scrape data from this website: http://www.bundesliga.de/de/liga/tabelle/
In the source code i can see the tables but there's no content, just things like:
<td>[no content]</td>
<td>[no content]</td>
<td>[no content]</td>
<td>[no content]</td>
....
With firebug (F12 in Firefox) i wont see any content too but i can select the table and then copy the innerHTML via firebug option. In that case i get all the informations about the teams, but i dont know how to get the table with the content in Jsoup.
To get the value of an attribute, use the Node.attr(String key) method
For the text on an element (and its combined children), use Element.text()
For HTML, use Element.html(), or Node.outerHtml() as appropriate
For example:
String html = "<p>An <a href='http://example.com/'><b>example</b></a> link.</p>";
Document doc = Jsoup.parse(html);
Element link = doc.select("a").first();
String text = doc.body().text(); // "An example link"
String linkHref = link.attr("href"); // "http://example.com/"
String linkText = link.text(); // "example""
String linkOuterH = link.outerHtml();
// "<b>example</b>"
String linkInnerH = link.html(); // "<b>example</b>"
reference:
http://jsoup.org/cookbook/extracting-data/attributes-text-html
The table is not rendered on the server directly, but build by the client side JavaScript of the page and constructed with data that is getting to the client via AJAX. So what you get with the naive Jsoup approach is expected.
I see two possible solutions:
You analyze the network traffic and identify the ajax calls that the site is making. Then you try to reconstruct the format and fire the same requests as the JavaScript would. Then you can reconstruct the table.
you don't use Jsoup but a real browser, that loads the page and runs the JavaScript including all AJAX calls. You could use Selenium webdriver for that. There is a headless browser called phantomjs which has a relatively small footprint that you can use in combination with selenium webdriver.
both options have their (dis)advantages:
This takes more time, since you need to understand the network traffic pretty good. The reward will be a very fast and memory efficient scraper.
The programming of selenium is very easy and you should not have any difficulties achieving your goal. You don't need to understand the inner workings of the site you want to scrape. However, the price is a further dependency in your project. Memory consumption is high. Another process runs. The scraping will be slow.
Maybe you find another source with the soccer table that is holding the infos you want? That might be the easiest. For example http://www.fussballdaten.de/bundesliga/
That title is probably a bit confusing so let me elaborate.
I'm using HTML_purifier to clean up user input, although in this case the only user who will be using it will be myself (its in password protected folders). A long story short I would like to be able to add in image tag code to a web form, then on the page that it sends too use the code to display said image.
However i need the image tag to have css attributes added to it, one of which is
display:block
Anyway by default HTML_purifier removes this, detailed here because of the CSS.allowTricky option. As i understand it if you set the CSS.allowTricky option to True, then it should allow
display:block
However after doing this its still removing it, just wondering if anybody has done this before as i can't find much documentation about it on the web? Its not generating any errors in syslog, so im assuming that its the correct implementation but isn't working as expected.
My code at the moment.
include('HTMLPurifier.standalone.php');
$config = HTMLPurifier_Config::createDefault();
$config->set('CSS.AllowTricky', true);
* UPDATE **
The code should pass the config object (which the code already set) to the html purifier object. Putting it together it should look something like this.
include('HTMLPurifier.standalone.php');
$config = HTMLPurifier_Config::createDefault();
$config->set('CSS.AllowTricky', true);
$purifier = new HTMLPurifier($config);
Duplicate of http://htmlpurifier.org/phorum/read.php?3,6724 (solution was passing the config object to the HTML Purifier object so that the config actually got applied.)
I am using Text::MultiMarkdown to create HTML files from MultiMarkdown documents.
I would like all links to open in a new tab.
Is there a way to configure this behavior using a CSS template, or directly in the MultiMarkdown document (without explicitly writing HTML around each link in the MultiMarkdown document)?
Definitely not in CSS - that is only concerned with the way the elements appear, not how they behave.
It should be possible to add <base target="_blank"> to the head of the HTML document (using XSLT), but that's on par with adding it to each link.
In HTML and/or JavaScript you can only initialize the opening of a new window. The user is in some UAs able to force the opening of a new window as a new tab instead. But you can not control this behaviour.
In theory, you could do this with CSS3: http://www.w3.org/TR/css3-hyperlinks/ - however no common browser ever implemented this. The reason might be that it is a common believe that the choice of when a new window or tab is opened should be left to the user alone.
You can't do this in CSS but you can use the source.
You could subclass Text::MultiMarkdown and provide your own implementation of _GenerateAnchor, something similar to this might work:
sub _GenerateAnchor {
my ($self, $whole_match, $link_text, $link_id, $url, $title, $attributes) = #_;
if($url
&& index($url, '#') != 0) {
$attributes = $attributes ? $attributes . ' target="_blank"' : 'target="_blank"';
}
return $self->SUPER::_GenerateAnchor($whole_match, $link_text, $link_id, $url, $title, $attributes);
}
This is a bit kludgey as _GenerateAnchor isn't part of the public interface. You'd also need to use the OO interface rather than just the markdown function.
You could also contact the Text::MultiMarkdown author and see if he'll add a flag for this sort of thing. Maybe you could provide a patch to get things started.
You can also use HTML::Parser and friends to parse the HTML that comes out of Text::MultiMarkdown and add the target attributes yourself.