What's the best way to scrape specific content from multiple HTML files? - html

Question

I have quite a few HTML files of webpages with many pieces of information. I am trying to extract some of the content and place it into an xml file or possible an excel spreadsheet. All webpages are quite similar by design and the information is placed in the same locations across all pages. Does anybody know of any way to do this?

Answer 1

there are many scraper library which can help you to extract data from html pages
Web scraping and crawling is not always so straightforward, so it depends on what you’re trying to achieve. Different products, SDK, libraries, etc., focus on different aspects of scraping or crawling. Here are a few you can check out:
Apify - (formerly Apifier) is a cloud-based web scraper that extracts structured data from any website using a few simple lines of JavaScript.
Diffbot - which extracts data from web pages automatically and returns structured JSON.
`
Espion
- a headless browser that enables you to inject JavaScript code directly into your target web pages.
Also if you have knowledge of Node Js then node-osmosis is realy cool and easy to use library

Answer 2

I strong recommend you this library:
http://sourceforge.net/projects/simplehtmldom/
/**
* Website: http://sourceforge.net/projects/simplehtmldom/
* Acknowledge: Jose Solorzano (https://sourceforge.net/projects/php-html/)
* Contributions by:
* Yousuke Kumakura (Attribute filters)
* Vadim Voituk (Negative indexes supports of "find" method)
* Antcs (Constructor with automatically load contents either text or file/url)
*
* all affected sections have comments starting with "PaperG"
*
* Paperg - Added case insensitive testing of the value of the selector.
* Paperg - Added tag_start for the starting index of tags - NOTE: This works but not accurately.
* This tag_start gets counted AFTER \r\n have been crushed out, and after the remove_noice calls so it will not reflect the REAL position of the tag in the source,
* it will almost always be smaller by some amount.
* We use this to determine how far into the file the tag in question is. This "percentage will never be accurate as the $dom->size is the "real" number of bytes the dom was created from.
* but for most purposes, it's a really good estimation.
* Paperg - Added the forceTagsClosed to the dom constructor. Forcing tags closed is great for malformed html, but it CAN lead to parsing errors.
* Allow the user to tell us how much they trust the html.
* Paperg add the text and plaintext to the selectors for the find syntax. plaintext implies text in the innertext of a node. text implies that the tag is a text node.
* This allows for us to find tags based on the text they contain.
* Create find_ancestor_tag to see if a tag is - at any level - inside of another specific tag.
* Paperg: added parse_charset so that we know about the character set of the source document.
* NOTE: If the user's system has a routine called get_last_retrieve_url_contents_content_type availalbe, we will assume it's returning the content-type header from the
* last transfer or curl_exec, and we will parse that and use it in preference to any other method of charset detection.
*
* Found infinite loop in the case of broken html in restore_noise. Rewrote to protect from that.
* PaperG (John Schlick) Added get_display_size for "IMG" tags.
*
* Licensed under The MIT License
* Redistributions of files must retain the above copyright notice.
*
* #author S.C. Chen <me578022#gmail.com>
* #author John Schlick
* #author Rus Carroll
* #version 1.5 ($Rev: 196 $)
* #package PlaceLocalInclude
* #subpackage simple_html_dom
*/
/**
* All of the Defines for the classes below.
* #author S.C. Chen <me578022#gmail.com>
*/
here's an example
$html = file_get_html($ad_bachecubano_url);
//Proceder a capturar el texto
$anuncio['header'] = $html->find('.headingText', 0)->plaintext;
$anuncio['body'] = $html->find('.showAdText', 0)->plaintext;
$precio = $html->find('#lineBlock');
foreach ($precio as $possibleprice) {
$item = $possibleprice->find('.headingText2', 0)->plaintext;
$precio = 0;
if ($item == "Precio: ") {
$precio = $possibleprice->find('.normalText', 0)->plaintext;
$anuncio['price'] = $this->getFinalPrice($precio);
} else {
continue;
}
}
$contactbox = $html->find('#contact');
foreach ($contactbox as $contact) {
$boxes = $contact->find('#lineBlock');
foreach ($boxes as $box) {
$key = $box->find('.headingText2', 0)->plaintext;
$value = $box->find('.normalText', 0)->plaintext;
if ($key == "Nombre: ") {
$anuncio['nombre'] = $value;
}
if ($key == "Teléfono: ") {
$anuncio['phone'] = $value;
}
}
}
$anuncio['email'] = scrapeemail($anuncio['body'])[0][0];
if (!isset($anuncio['email']) || $anuncio['email'] == '') {
$anuncio['email'] = "";
}