I am trying to extract the text only from the html page and want to ignore or bypass the html escape signs "<" and ">". I am copying the part the html page that i used for extraction of text:
<table class="reference">
<tr>
<th align="left" width="25%">Tag</th>
<th align="left" width="75%">Description</th>
</tr>
<tr>
<td><!--...--></td>
<td>Defines a comment</td>
</tr>
<tr>
<td><!DOCTYPE> </td>
<td>Defines the document type</td>
</tr>
<tr>
<td><a></td>
<td>Defines a hyperlink</td>
</tr>
<tr>
<td><abbr></td>
<td>Defines an abbreviation</td>
</tr>
<tr>
...
My perl code is:
my $urlToScrape = "http://www.w3schools.com/tags/";
# prepare data
my $teamsdata = scraper {
process "table.reference > tr > td > a ", 'tags[]' => 'TEXT';
process "table.reference > tr > td > a ", 'urls[]' => '#href';
};
# scrape the data
my $res = $teamsdata->scrape(URI->new($urlToScrape));
print "<HTML_tags>\n";
for my $i ( 0 .. $#{$res->{urls}}) {
print FILE " <tag_Name> $res->{tags}[$i] </tag_Name>\n ";
}
print "</HTML_tags>\n";
The output I get is the following:
<HTML_tags>
<tag_Name> <!--...--> </tag_Name>
<tag_Name> <!DOCTYPE> </tag_Name>
<tag_Name> <a> </tag_Name>
<tag_Name> <abbr> </tag_Name>
</HTML_tags>
whereas I want output as:
<HTML_tags>
<tag_Name> !--...-- </tag_Name>
<tag_Name> !DOCTYPE </tag_Name>
<tag_Name> a </tag_Name>
<tag_Name> abbr </tag_Name>
</HTML_tags>
Can anyone tell what do I have to change inorder to get the above output?
Many Thanks.
Brute Force:
$res->{tags}[$i] =~ s/[\<\>]//gs; ## Added line
print FILE " <tag_Name> $res->{tags}[$i] </tag_Name>\n ";
Related
I have a table where i'm trying to add more rows with powershell then export it as a new HTML file.
Here's the body of the HTML i'm trying to add rows to.
<BODY>
<TABLE style="WIDTH: 100%" cellPadding=5>
<TBODY>
<TR>
<TH>Bruger</TH>
<TH>Windows</TH>
<TH>Installations dato</TH>
<TH>Model</TH>
<TH>Sidst slukket</TH></TR>
<TR>
<TD>Users name</TD>
<TD>Windows 10 Pro</TD>
<TD>23-01-2020</TD>
<TD>ThinkPad</TD>
<TD>7 dage</TD></TR></TBODY></TABLE>
<TABLE>
<TBODY></TBODY></TABLE></BODY>
I figured i'd need to change the inner html of an object but it's just throwing an error.
Here's my code
$src = [IO.File]::ReadAllText($outPath)
$doc = New-Object -com "HTMLFILE"
$doc.IHTMLDocument2_write($src)
$elm = $doc.getElementsByTagName('tr')[0]
$elm.innerHTML = "<TR>New row!</TR>"
When I check the inner html variable I get the HTML output that I would expect, so it's grabbing the correct object, but I can't assign anything to it for whatever reason.
Here's the error
Exception from HRESULT: 0x800A0258
At line:1 char:1
+ $elm.innerHTML = "<TH>User</TH>"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException
Instead of modifying the innerHTML contents of an existing <tr> element, you'll want to:
Create a new <tr> element
Create any requisite <td> child element(s)
Append <td> element(s) to your new row
Append the new row to the existing <tbody>
Try something like this:
$html = #'
<BODY>
<TABLE style="WIDTH: 100%" cellPadding=5>
<TBODY>
<TR>
<TH>Bruger</TH>
<TH>Windows</TH>
<TH>Installations dato</TH>
<TH>Model</TH>
<TH>Sidst slukket</TH></TR>
<TR>
<TD>Users name</TD>
<TD>Windows 10 Pro</TD>
<TD>23-01-2020</TD>
<TD>ThinkPad</TD>
<TD>7 dage</TD></TR></TBODY></TABLE>
<TABLE>
<TBODY></TBODY></TABLE></BODY>
'#
# Create HTML document object
$doc = New-Object -ComObject HTMLFile
# Load existing HTML
$doc.IHTMLDocument2_write($html)
# Create new row element
$newRow = $doc.createElement('tr')
# Create new cell element
$newCell = $doc.createElement('td')
$newCell.innerHTML = "New row!"
$newCell.colSpan = 5
# Append cell to row
$newRow.appendChild($newCell)
# Append row to table body
$tbody = $doc.getElementsByTagName('tbody')[0]
$tbody.appendChild($newRow)
# Inspect resulting HTML
$tbody.outerHtml
You should expect to see the new row appended to the table body:
<TBODY><TR>
<TH>Bruger</TH>
<TH>Windows</TH>
<TH>Installations dato</TH>
<TH>Model</TH>
<TH>Sidst slukket</TH></TR>
<TR>
<TD>Users name</TD>
<TD>Windows 10 Pro</TD>
<TD>23-01-2020</TD>
<TD>ThinkPad</TD>
<TD>7 dage</TD></TR>
<TR>
<TD colSpan=5>New row!</TD></TR></TBODY>
You could create a nice little helper function for adding new rows:
function New-HTMLFileTableRow {
param(
[Parameter(Mandatory)]
[mshtml.HTMLDocumentClass]$Document,
[Parameter(Mandatory)]
[string[]]$Property,
[Parameter(Mandatory, ValueFromPipeline)]
$InputObject
)
process {
$newRow = $Document.createElement('tr')
foreach($propName in $Property){
$newCell = $Document.createElement('td')
$newCell.innerHtml = $InputObject.$propName
[void]$newRow.appendChild($newCell)
}
return $newRow
}
}
Then use like:
Import-Csv .\path\to\user-os-list.csv |New-HTMLFileTableRow -Property User,OSVersion,InstallDate,Model,LastActive -Document $doc |ForEach-Object {
[void]$tbody.appendChild($_)
}
I am trying to parse this page and pull the date that begins after
>p>From Date:
I get the error
Invalid predicate: //b[text() = '<p>From Date: ' (Nokogiri::XML::XPath::SyntaxError)
The xpath from "inspect element" is
/html/body/div#timelineItems/table/tbody/tr/td/table.resultsTypes/tbody/tr/td/p
This is an example of the code:
#/usr/bin/ruby
require 'Nokogiri'
noko = Nokogiri::HTML('china.html')
noko.xpath("//b[text() = '<p>From Date: ").each do |b|
puts b.next_sibling.content.strip
end
This is file://china.html
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>File </title>
</head>
<body>
<div id ="timelineItems">
<H2 id="telegram1"> Title </H2>
<p><table cellspacing="0">
<tr>
<td width="2%"> </td>
<td width="75%">
<table cellspacing="0" cellpadding="0" class="resultsTypes">
<tr>
<td width="5%" class="hide"> </td>
<td width="70%">
<p>Template: <span class="bidi">ארכיון בן גוריון - מסמך</span></p>
<p>Title: <span class="bidi">Meeting in China</span></p>
<p>recipient: David Ben Gurion</p>
<p>sender: Prime Minister of Union of Burma, Rangoon</p>
<p> Sub collection: <span class="bidi">התכתבות > תת-חטיבה מכתב</span></p>
<p>From Date: 02/14/1936</p>
<p>Link to file: <span class="bidi">תיק התכתבות 1956 ינואר</span></p>
</td>
</tr>
<tr>
<td colspan="2">
</td>
</tr>
</table></td>
<td class="actions"> </td>
</tr>
</table>
</p>
</div>
</body></html>
Amadan's answer
original.rb
#/usr/bin/ruby
require 'Nokogiri'
noko = Nokogiri::HTML('china.html')
date = noko.at_xpath("//p[starts-with(text(),'From Date: ')]").text()
puts date
formatted = date[/From Date: (.*)/, 1]
puts formatted
gives an error original.rb:5:in '<main>': undefined method 'text' for nil:NilClass (NoMethodError)
You can't use
noko = Nokogiri::HTML('china.html')
Nokogiri::HTML is a shortcut to Nokogiri::HTML::Document.parse. The documentation says:
.parse(string_or_io, url = nil, encoding = nil, options = XML::ParseOptions::DEFAULT_HTML) {|options| ... } ⇒ Object`
... string_or_io may be a String, or any object that responds to read and close such as an IO, or StringIO. ...
While 'china.html' is a String, it's not HTML. It appears you're thinking that a filename will suffice, however Nokogiri doesn't open anything, it only understands strings containing markup, either HTML or XML, or an IO-type object that responds to the read method. Compare these:
require 'nokogiri'
doc = Nokogiri::HTML('china.html')
doc.to_html
# => "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\" \"http://www.w3.org/TR/REC-html40/loose.dtd\">\n<html><body><p>china.html</p></body></html>\n"
versus:
doc = Nokogiri::HTML('<html><body><p>foo</p></body></html>')
doc.to_html
# => "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\" \"http://www.w3.org/TR/REC-html40/loose.dtd\">\n<html><body><p>foo</p></body></html>\n"
and:
doc = Nokogiri::HTML(open('http://www.example.org'))
doc.to_html[0..99]
# => "<!DOCTYPE html>\n<html>\n<head>\n <title>Example Domain</title>\n\n <meta charset=\"utf-8\">\n <met"
The last works because OpenURI adds the ability to read URLs to open, which responds to read:
open('http://www.example.org').respond_to?(:read) # => true
Moving on to the question:
require 'nokogiri'
require 'open-uri'
html = <<EOT
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>File </title>
</head>
<body>
<div id ="timelineItems">
<H2 id="telegram1"> Title </H2>
<p><table cellspacing="0">
<tr>
<td width="2%"> </td>
<td width="75%">
<table cellspacing="0" cellpadding="0" class="resultsTypes">
<tr>
<td width="5%" class="hide"> </td>
<td width="70%">
<p>Template: <span class="bidi">ארכיון בן גוריון - מסמך</span></p>
<p>Title: <span class="bidi">Meeting in China</span></p>
<p>recipient: David Ben Gurion</p>
<p>sender: Prime Minister of Union of Burma, Rangoon</p>
<p> Sub collection: <span class="bidi">התכתבות > תת-חטיבה מכתב</span></p>
<p>From Date: 02/14/1936</p>
<p>Link to file: <span class="bidi">תיק התכתבות 1956 ינואר</span></p>
</td>
</tr>
<tr>
<td colspan="2">
</td>
</tr>
</table></td>
<td class="actions"> </td>
</tr>
</table>
</p>
</div>
</body></html>
EOT
doc = Nokogiri::HTML(html)
Once the document is parsed, it's easy to find a particular <p> tag using the
<table cellspacing="0" cellpadding="0" class="resultsTypes">
as a placemarker:
from_date = doc.at('table.resultsTypes p[6]').text
# => "From Date: 02/14/1936"
It looks like its going to be tougher pulling the title = "Meeting in China" and link = "bing.com"; since they are on the same line.
I'm using CSS selectors to define the path to the desired text. CSS is more easily read than XPath, though XPath is more powerful and descriptive. Nokogiri allows us to use either, and lets us use search or at with either. at is equivalent to search('some selector').first. There are also CSS and XPath specific versions of search and at, described in Nokogiri::XML::Node.
title_link = doc.at('table.resultsTypes p[2] a')['href'] # => "http://www.bing.com"
title = doc.at('table.resultsTypes p[2] span').text # => "Meeting in China"
You're trying to use the XPath:
/html/body/div#timelineItems/table/tbody/tr/td/table.resultsTypes/tbody/tr/td/p
however, it's not valid for the HTML you're working with.
Notice tbody in the selector. Look at the HTML, immediately after either of the <table> tags, neither occurrence has a <tbody> tag, so the XPath is wrong. I suspect that was generated by your browser, which is doing a fix-up of the HTML to add <tbody> according to the specification, however Nokogiri doesn't do a fix-up to add <tbody> and the HTML doesn't match, causing the search to fail. So, don't rely on the selector defined by the browser, nor should you trust the browser's idea of the actual HTML source.
Instead of using an explicit selector, it's better, easier, and smarter, to look for specific way-points in the markup, and use those to navigate to the node(s) you want. Here's an example of doing everything above, only using a placeholder, and a mix of XPath and CSS:
doc.at('//p[starts-with(., "Title:")]').text # => "Title: Meeting in China"
title_node = doc.at('//p[starts-with(., "Title:")]')
title_url = title_node.at('a')['href'] # => "http://www.bing.com"
title = title_node.at('span').text # => "Meeting in China"
So, it's fine to mix and match CSS and XPath.
from_date = noko.at_xpath('//p[starts-with(text(), "From Date:")]').text()
date = from_date[/From Date: (.*)/, 1]
# => "02/14/1936"
EDIT:
Explanation: Get the first node (#at_xpath) anywhere in the document (//) such that ([...]) text content (text()) starts with (starts-with(string, stringStart)) "From Date" ("From Date:"), and take its text content (#text()), storing it (=) into the variable from_date (from_date). Then, extract the first group (#[regexp, 1]) from that text (from_date) by using the regular expression (/.../) that matches the literal characters "From Date: ", followed by any number (*) of any characters (.), that will be captured ((...)) in the first capture group to be extracted by #[regexp, 1].
Also,
Amadan's answer [...] gives an error
I did not notice that your Nokogiri construction is broken, as explained by the Tin Man. The line noko = Nokogiri::HTML('china.html') (which was not a part of my answer) will give you a single node document that only has the text "china.html" in it, and no <p> nodes at all.
I am trying to get values from a database and place them in a dropdown list within an HTML <select> tag.
I'm able to get the values in a long string and display all of them within a single option but I want to put each value in a separate <option> tag. I just don't know what logic I could use to do this.
Here's what I have so far:
#!c:\perl\bin\perl.exe
use CGI;
require ("data_eXchangeSubs.pm");
$query = new CGI;
print $query->header(-expires=>'-1d');
print $query->start_html(
-title=>'Dex Vendor Testing',
-bgcolor=>'white'
);
$user = $query->param("user");
my $dataX = ${ConnectToDatabase($main::DBone, $main::dataENV)};
$resultSet = $dataX->Execute("select vendor from dex_vendor_info group by vendor");
while(!$resultSet->EOF) {
$vendors .= $resultSet->Fields("vendor")->Value."\n";
$resultSet->MoveNext;
}
print <<ONE;
<table width=75% border=0>
<th colspan=2 align=left><strong><font size=5pt color=#FF6633 face=garamond>Vendor Information</strong</font><hr size=4pt color=midnightblue></th>
<tr>
<td align=left nowrap><font size=4pt face=garamond><label id=lVendor for=vendor><strong>Company Name</strong></font></label></td>
<td align=left nowrap><font size=4pt face=garamond><label id=lVendor for=vendor><strong>Contact's Name</strong></font></label></td>
</tr>
<tr>
<td align=left nowrap><select id="vendors">
<option>$vendors</option>
</td>
</td>
<td align=left nowrap><input type=text name="contact" id=contact value="" size=25></td>
</tr>
</table>
<br>
ONE
print $vendors;
print $query->end_html;
If you're using CGI, then use CGI.
print $query->popup_menu(
-name => 'vendors'
, -values => \#list_of_vendors
, -default => $default_vendor
);
And you get #list_of_vendors in your row processing loop:
my #list_of_vendors;
while(!$resultSet->EOF) {
push #list_of_vendors, $resultSet->Fields("vendor")->Value;
$resultSet->MoveNext;
}
If you want labels to be a different text value from values include -labels tag in the call and point it to an array ref containing the text you want visible.
I have document with the following two formats:
<p><b>Referral Description:</b></p>
<p>
This is the body of the referral's detailed description.
I want to get this text out of the document.
</p>
and
<table>
<tr>
<td><b>FieldName:</b></td>
<td>field value</td>
</tr>
<tr>
<td><b>Field2Name:</b></td>
<td>field value</td>
</tr>
<tr>
<td><b>Field3Name:</b></td>
<td>field value</td>
</tr>
</table>
In both cases, you can see that I need a value sitting in an un-named element, and its adjacent neighbor is a matching tag with a <b>FieldName:</b> body.
My question is, how can I use the neighbor tags to get the values I need? I can target the neighbor with
doc.xpath('//p/b[content(text(), "Referral Description:")]')
but how do I take that and say "Give me your neighbor"?
I would do as below using Axis - following-sibling:::
require 'nokogiri'
doc = Nokogiri::HTML.parse <<-html
<p><b>Referral Description:</b></p>
<p>
This is the body of the referral's detailed description.
I want to get this text out of the document.
</p>
html
node = doc.xpath('//p[./b[contains(text(), "Referral Description:")]]/following-sibling::p')
puts node.text
# >>
# >> This is the body of the referral's detailed description.
# >> I want to get this text out of the document.
Or, using wild-card character * :
require 'nokogiri'
doc = Nokogiri::HTML.parse <<-html
<p><b>Referral Description:</b></p>
<p>
This is the body of the referral's detailed description.
I want to get this text out of the document.
</p>
html
["Referral Description:", "FieldName:", "Field1Name:"].map |header|
doc.xpath("//*[./b[contains(text(), '#{header}')]]/following-sibling::*')
end
# >>
# >> ["This is the body of the referral's detailed description.\nI want to get this text out of the document.", "field value", "field value"]
For the second part of HTML table :
require 'nokogiri'
doc = Nokogiri::HTML.parse <<-html
<table>
<tr>
<td><b>FieldName:</b></td>
<td>field value</td>
</tr>
<tr>
<td><b>Field2Name:</b></td>
<td>field value</td>
</tr>
<tr>
<td><b>Field3Name:</b></td>
<td>field value</td>
</tr>
</table>
html
field_ary = %w(FieldName Field2Name Field3Name)
nodeset = field_ary.map{|n| doc.xpath("//td[./b[contains(.,'#{n}')]]/following-sibling::*")}
nodeset.map{|n| n.text }
# => ["field value", "field value", "field value"]
or(another approach)
nodeset = field_ary.map{|n| doc.xpath("//*[./b[contains(.,'#{n}')]]/following-sibling::*")}
nodeset.map{|n| n.text }
# => ["field value", "field value", "field value"]
In css, the next adjacent sibling selector is +:
doc.at('p:has(b[text()="Referral Description:"]) + p').text
I have this input text:
<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body><table cellspacing="0" cellpadding="0" border="0" align="center" width="603"> <tbody><tr> <td><table cellspacing="0" cellpadding="0" border="0" width="603"> <tbody><tr> <td width="314"><img height="61" width="330" src="/Elearning_Platform/dp_templates/dp-template-images/awards-title.jpg" alt="" /></td> <td width="273"><img height="61" width="273" src="/Elearning_Platform/dp_templates/dp-template-images/awards.jpg" alt="" /></td> </tr> </tbody></table></td> </tr> <tr> <td><table cellspacing="0" cellpadding="0" border="0" align="center" width="603"> <tbody><tr> <td colspan="3"><img height="45" width="603" src="/Elearning_Platform/dp_templates/dp-template-images/top-bar.gif" alt="" /></td> </tr> <tr> <td background="/Elearning_Platform/dp_templates/dp-template-images/left-bar-bg.gif" width="12"><img height="1" width="12" src="/Elearning_Platform/dp_templates/dp-template-images/left-bar-bg.gif" alt="" /></td> <td width="580"><p> what y all heard?</p><p>i'm shark oysters.</p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p></td> <td background="/Elearning_Platform/dp_templates/dp-template-images/right-bar-bg.gif" width="11"><img height="1" width="11" src="/Elearning_Platform/dp_templates/dp-template-images/right-bar-bg.gif" alt="" /></td> </tr> <tr> <td colspan="3"><img height="31" width="603" src="/Elearning_Platform/dp_templates/dp-template-images/bottom-bar.gif" alt="" /></td> </tr> </tbody></table></td> </tr> </tbody></table> <p> </p></body></html>
As you can see, there's no newline in this chunk of HTML text, and I need to look for all image links inside, copy them out to a directory, and change the line inside the text to something like ./images/file_name.
Currently, the Perl code that I'm using looks like this:
my ($old_src,$new_src,$folder_name);
foreach my $record (#readfile) {
## so the if else case for the url replacement block below will be correct
$old_src = "";
$new_src = "";
if ($record =~ /\<img(.+)/){
if($1=~/src=\"((\w|_|\\|-|\/|\.|:)+)\"/){
$old_src = $1;
my #tmp = split(/\/Elearning/,$old_src);
$new_src = "/media/www/vprimary/Elearning".$tmp[-1];
push (#images, $new_src);
$folder_name = "images";
}## end if
}
elsif($record =~ /background=\"(.+\.jpg)/){
$old_src = $1;
my #tmp = split(/\/Elearning/,$old_src);
$new_src = "/media/www/vprimary/Elearning".$tmp[-1];
push (#images, $new_src);
$folder_name = "images";
}
elsif($record=~/\<iframe(.+)/){
if($1=~/src=\"((\w|_|\\|\?|=|-|\/|\.|:)+)\"/){
$old_src = $1;
my #tmp = split(/\/Elearning/,$old_src);
$new_src = "/media/www/vprimary/Elearning".$tmp[-1];
## remove the ?rand behind the html file name
if($new_src=~/\?rand/){
my ($fname,$rand) = split(/\?/,$new_src);
$new_src = $fname;
my ($fname,$rand) = split(/\?/,$old_src);
$old_src = $fname."\\?".$rand;
}
print "old_src::$old_src\n"; ##s7test
print "new_src::$new_src\n\n"; ##s7test
push (#iframes, $new_src);
$folder_name = "iframes";
}## end if
}## end if
my $new_record = $record;
if($old_src && $new_src){
$new_record =~ s/$old_src/$new_src/ ;
print "new_record:$new_record\n"; ##s7test
my #tmp = split(/\//,$new_src);
$new_record =~ s/$new_src/\.\\$folder_name\\$tmp[-1]/;
## print "new_record2:$new_record\n\n"; ##s7test
}## end if
print WRITEFILE $new_record;
} # foreach
This is only sufficient to handle HTML text with newlines in them.
I thought only looping the regex statement,
but then i would have to change the matching line to some other text.
Do you have any idea if there an elegant Perl way to do this?
Or maybe I'm just too dumb to see the obvious way of doing it, plus I know putting global option doesn't work.
thanks.
~steve
There are excellent HTML parsers for Perl, learn to use them and stick with that. HTML is complex, allows > in attributes, heavily use nesting, etc. Using regexes to parse it, beyond very simple tasks (or machine generated code), is prone to problems.
I think you want my HTML::SimpleLinkExtor module:
use HTML::SimpleLinkExtor;
my $extor = HTML::SimpleLinkExtor->new;
$extor->parse_file( $file );
my #imgs = $extor->img;
I'm not sure what exactly you're trying to do, but it surely sounds like one of the HTML parsing modules should do the trick if mine doesn't.
If you must avoid any additional module, like an HTML parser, you could try:
while ($string =~ m/(?:\<\s*(?:img|iframe)[^\>]+src\s*=\s*\"((?:\w|_|\\|-|\/|\.|:)+)\"|background\s*=\s*\"([^\>]+\.jpg)|\<\s*iframe)/g) {
$old_src = $1;
my #tmp = split(/\/Elearning/,$old_src);
$new_src = "/media/www/vprimary/Elearning".$tmp[-1];
if($new_src=~/\?rand/){
// remove rand and push in #iframes
else
{
// push into #images
}
}
That way, you would apply this regex on all the source (newlines included), and have a more compact code (plus, you would take into account any extra space between attributes and their values)