Given the following HTML code snippet:
<div class="item">
large
<span class="some-class">size</span>
</div>
I'm looking for the best way to extract the string "large" using Symfony's Crawler.
$crawler = new Crawler($html);
Here I could use $crawler->html() then apply a regex search. Is there a better solution?
Or how would you do it exactly?
I've just found a solution that looks the cleanest to me:
$crawler = new Crawler($html);
$result = $crawler->filterXPath('//text()')->text();
This is a bit tricky as the text that you're trying to get is a text node that the DOMCrawler component doesn't (as far as I know) allow you to extract. Thankfully DOMCrawler is just a layer over the top of PHP's DOM classes which means you could probably do something like:
$crawler = new Crawler($html);
$crawler = $crawler->filterXPath('//div[#class="item"]');
$domNode = $crawler->getNode(0);
$text = null;
foreach ($domNode->children as $domChild) {
if ($domChild instanceof \DOMText) {
$text = $domChild->wholeText;
break;
}
}
This wouldn't help with HTML like:
<div>
text
<span>hello</span>
other text
</div>
So you would only get "text", not "text other text" in this instance. Take a look at the DOMText documentation for more details.
$crawler = new Crawler($html);
$node = $crawler->filterXPath('//div[#class="item"]');
$domElement = $node->getNode(0);
foreach ($node->children() as $child) {
$domElement->removeChild($child);
}
dump($node->text()); die();
After you have to trim whitespace.
Related
I have this regular expression:
(\S+)=[""']?((?:.(?![""']?\s+(?:\S+)=|[>""']))+.)[""']?
This regex expression will extract the name of the tag and the value from HTML string, everything is working fine, but, when I have a single char the regex will trap the left side quote and the character.
This is my string:
<select title="Campo" id="6:7" style="width: auto; cursor: pointer;" runat="server" controltype="DropDownList" column="Dummy_6"><option value="0">Value:0</option><option selected="selected" value='1'>Value:1Selected!</option></select>
I don't know how to modify this regex expression to capture the char correctly even there is only one character.
You should be using HTML parser for this task, regex cannot handle HTML properly.
To collect all tag names and there attribute names and values, I recommend the following HtmlAgilityPack-based solution:
var tags = new List<string>();
var result = new List<KeyValuePair<string, string>>();
HtmlAgilityPack.HtmlDocument hap;
Uri uriResult;
if (Uri.TryCreate(html, UriKind.Absolute, out uriResult) && uriResult.Scheme == Uri.UriSchemeHttp)
{ // html is a URL
var doc = new HtmlAgilityPack.HtmlWeb();
hap = doc.Load(uriResult.AbsoluteUri);
}
else
{ // html is a string
hap = new HtmlAgilityPack.HtmlDocument();
hap.LoadHtml(html);
}
var nodes = hap.DocumentNode.Descendants().Where(p => p.NodeType == HtmlAgilityPack.HtmlNodeType.Element);
if (nodes != null)
foreach (var node in nodes)
{
tags.Add(node.Name);
foreach (var attribute in node.Attributes)
result.Add(new KeyValuePair<string, string>(attribute.Name, attribute.Value));
}
I think you're trying something overly intricate and, ultimately, incorrect, with your regex.
If you want to naively parse an HTML attribute: this regex should do the trick:
(\S+)=(?:"([^"]+)"|'([^']+)')
Note that it parses single-quoted and double-quoted values in different legs of the regex. Your regex would find that in the following code:
<foo bar='fu"bar'>
the attribute's value is fu when it really is fu"bar.
There are better ways to parse HTML, but here's my take at your question anyway.
(?<attr>(?<=\s).+?(?==['"]))|(?<val>(?<=\s.+?=['"]).+?(?=['"]))
Without capture group names:
((?<=\s).+?(?==['"]))|((?<=\s.+?=['"]).+?(?=['"]))
quotes included:
((?<=\s).+?(?==['"]))|((?<=\s.+?=)['"].+?['"])
Update: For more in-depth usage, do give HTML Agility Pack a try.
I'm a bit of a newbie when it comes to linq and I'm working on a site that parses a json feed using json.net. The problem that I'm having is that I need to be able to pull multiple fields from the json feed and use them for a foreach block. The documentation for json.net only shows how to pull just one field. I've done a few variations after checking out the linq documentation, but I've not found anything that works best. Here's what I've got so far:
WebResponse objResponse;
WebRequest objRequest = HttpWebRequest.Create(url);
objResponse = objRequest.GetResponse();
using (StreamReader reader = new StreamReader(objResponse.GetResponseStream()))
{
string json = reader.ReadToEnd();
JObject rss = JObject.Parse(json);
var postTitles =
from p in rss["feedArray"].Children()
select (string)p["item"],
//These are the fields I need to also query
//(string)p["title"], (string)p["message"];
//I've also tried this with console.write and labeling the field indicies for each pulled field
foreach (var item in postTitles)
{
lbl_slides.Text += "<div class='slide'><div class='slide_inner'><div class='slide_box'><div class='slide_content'></div><!-- slide content --></div><!-- slide box --></div><div class='rotator_photo'><img src='" + item + "' alt='' /></div><!-- rotator photo --></div><!-- slide -->";
}
}
Has anyone seen how to pull multiple fields from a json feed and use them as part of a foreach block (or something similar?
Couldn't you just reference the fields directly in your foreach loop, like this (below)? I'm not sure you really need the linq query here. (Note, I have cut out most of your html for this example for clarity. You'll need to adjust for your actual project, do appropriate HTML escaping, etc.)
foreach (var p in rss["feedArray"].Children())
{
lbl_slides.Text += string.Format(
"<img src='{0}' title='{1}'/><span>{2}</span>",
(string)p["item"],
(string)p["title"],
(string)p["message"]);
}
Same thing using linq would look like this:
var postTitles =
from p in rss["feedArray"].Children()
select new
{
Src = (string)p["item"],
Title = (string)p["title"],
Message = (string)p["message"],
}
foreach (var item in postTitles)
{
lbl_slides.Text += string.Format(
"<img src='{0}' title='{1}'/><span>{2}</span>",
item.Src, item.Title, item.Message);
}
I have a project where users submit many links to external sites and I need to parse the HTML of these submitted links and extract basic information from the page in the same way that Digg and Facebook do when a link is submitted.
I want to retrieve:
main title or heading (could be in title, h1, h2, p etc...)
intro or description text (could be in div, p etc...)
main image
My main problem is that there seem to be too many options to explore here and im getting a little confused to sat the least. Many solutions I have looked so far seem to be inadequate or huge overkill.
You would pick a server side language to do this.
For example, with PHP, you could use get_meta_tags() for the meta tags...
$meta = get_meta_tags('http://google.com');
And you could use DOMDocument to get the title element (some may argue if needing the title element, you may as well use DOMDocument to get the meta tags as well).
$dom = new DOMDocument;
$dom->loadHTML('http://google.com');
$title = $dom
->getElementsByTagName('head')
->item(0)
->getElementsByTagName('title')
->item(0)
->nodeValue;
As for getting main image, that would require some sort of extraction of what may be considered the main image. You could get all img elements and look for the largest one on the page.
$dom = new DOMDocument;
$dom->loadHTML('http://google.com');
$imgs = $dom
->getElementsByTagName('body')
->item(0)
->getElementsByTagName('img');
$imageSizes = array();
foreach($imgs as $img) {
if ( ! $img->hasAttribute('src')) {
continue;
}
$src = $img->getAttribute('src');
// May need to prepend relative path
// Assuming Apache, http and port 80
$relativePath = rtrim($_SERVER['SERVER_NAME'] . $_SERVER['REQUEST_URI'], '/') . '/';
if (substr($src, 0, strlen($relativePath) !== $relativePath) {
$src = $relativePath . $src;
}
$imageInfo = getimageinfo($src);
if ( ! $imageInfo) {
continue;
}
list($width, $height) = $imageInfo;
$imageSizes[$width * $height] = $img;
}
$mainImage = end($imageSizes);
I'm trying to find a regex for VBScript to remove some html tags and their content from a string.
The string is,
<H2>Title</H2><SPAN class=tiny>Some
text here</SPAN><LI>Some list
here</LI><SCRITP>Some script
here</SCRITP><P>Some text here</P>
Now, I'd like to EXCLUDE <SPAN class=tiny>Some text here</SPAN> and <SCRITP>Some script here</SCRITP>
Maybe someone has a simple solution for this, thanks.
This should do the trick in VBScript:
Dim myRegExp, ResultString
Set myRegExp = New RegExp
myRegExp.IgnoreCase = True
myRegExp.Global = True
myRegExp.Pattern = "<span class=tiny>[\s\S]*?</span>|<script>[\s\S]*?</script>"
ResultString = myRegExp.Replace(SubjectString, "")
SubjectString is the variable with your original HTML and ResultString receives the HTML with all occurrences of the two tags removed.
Note: I'm assuming scritp in your sample is a typo for script. If not, adjust my code sample accordingly.
You could do this a lot easier using css:
span.tiny {
display: none;
}
or using jQuery:
$("span.tiny").hide();
I think you want this
$(function(){
$('span.tiny').remove();
$('script').remove();
})
I am working on parsing HTML obtain all the hrefs that match a particular url (let's call it "target url") and then get the anchor text. I have tried LinkExtractor, TokenParser, Mechanize, TreeBuilder modules. For below HTML:
<a href="target_url">
<img src=somepath/nw.gf alt="Open this result in new window">
</a>
all of them give "Open this result in new window" as the anchor text.
Ideally I would like to see blank value or a string like "image" returned so that I know there was no anchor text but the href still matched the target url (http://www.yahoo.com in this case). Is there a way to get the desired result using other module or Perl regex?
Thanks,
You should post some examples that you tried with "LinkExtractor, TokenParser, Mechanize & TreeBuilder" so that we can help you.
Here is something which works for me in pQuery:
use pQuery;
my $data = '
<html>
Not yahoo anchor text
<img src="somepath/nw.gif" alt="Open this result in new window"></img>
just text for yahoo
anchor text only<img src="blah" alt="alt text"/>
</html>
';
pQuery( $data )->find( 'a' )->each(
sub {
say $_->innerHTML
if $_->getAttribute( 'href' ) eq 'http://www.yahoo.com';
}
);
# produces:
#
# => <img alt="Open this result in new window" src="somepath/nw.gif"></img>
# => just text for yahoo
# => anchor text only<img /="/" alt="alt text" src="blah"></img>
#
And if you just want the text:
pQuery( $data )->find( 'a' )->each(
sub {
return unless $_->getAttribute( 'href' ) eq 'http://www.yahoo.com';
if ( my $text = pQuery($_)->text ) { say $text }
}
);
# produces:
#
# => just text for yahoo
# => anchor text only
#
/I3az/
Use a proper parser (like HTML::Parser or HTML::TreeBuilder). Using regular expressions to parse SGML (HTML/XML included) isn't really all that effective because of funny multiline tags and attributes like the one you've run into.
If the HTML you are working with is fairly close to well formed you can usually load it into an XML module that supports HTML and use it to find and extract data from the parts of the document you are interested in.
My method of choice is XML::LibXML and XPath.
use XML::LibXML;
my $parser = XML::LibXML->new();
my $html = ...;
my $doc = $parser->parse_html_string($html);
my #links = $doc->findnodes('//a[#href = "http://example.com"]');
for my $node (#links) {
say $node->textContent();
}
The string passed to findnodes is an XPath expression that looks for all 'a' element descendants of $doc that have an href attribute equal to "http://example.com".