Parsing with Xpath - html

Question

Consider the following HTML:
<div class='data'>
<div class='user_name'>Lankesh</div>
<div class='user_details'>
<div class='country'>Srilanka</div>
<div class='age'>9</div>
</div>
<div class='user_name'>Bob</div>
<div class='user_details'>
<div class='country'>US</div>
<div class='age'>54</div>
</div>
<div class='user_name'>Deiter</div>
<div class='user_details'>
<div class='country'>Germany</div>
<div class='age'>34</div>
</div>
<div class='user_name'>Yakob</div>
<div class='user_details'>
<div class='country'>Syria</div>
<div class='age'>90</div>
</div>
<div class='user_name'>Qureshi</div>
<div class='user_details'>
<div class='country'>Afgan</div>
<div class='age'>56</div>
</div>
<div class='user_name'>Smith George</div>
<div class='user_details'>
<div class='country'>India</div>
<div class='age'>23</div>
</div>
</div>
And the following Ruby code:
require 'nokogiri'
sample_html = File.open("r.htm", "r").read
n = Nokogiri::HTML::parse sample_html
xpaths = {}
xpaths[:name] = "//div[#class = 'user_name']/text()"
xpaths[:country] = "//div[#class = 'country']/text()"
xpaths[:age] = "//div[#class = 'age']/text()"
full_path = xpaths.values.join(" | ")
n.xpath(full_path).each do |i|
puts i
end
This works to extract data, but how can I chunk (name, age and country) so that I can extract the parsed data into a structure more easily.
Since name is outside the user_details block, I am unable to write a query like: //div[#class = 'user_details'] and extract each attribute.
I know I can chunk the array into groups of 3; but I am looking for xpath based solution, because my actual need has varying number of child properties.
Silly, but: anyway to somehow inject characters to the extracted text, during parsing?
Any ideas?

Answer 1

Let me start out by saying it would be better to adjust the HTML to wrap each user block in its own containing div:
<div class='user'>
<div class='name'>John</div>
<div class='details'>
<div class='country'>US</div>
...
</div>
</div>
Then you could simply query each user block separately using "//div[#class = 'user']". You are probably not in control of the HTML, though.
Given the current situation I would propose to simply obtain the user_name divs, as well as the user_details divs and zip them together. Then, you can create a Hash from the user details based on the child divs (.xpath("div")) which will work for any amount of user_details and uses their class attribute as a Hash key and their text as a value. Note this implementation only works on single-level user_details. Of course this will have to be adjusted if not all user_details child divs will have a class attribute. But judging from your example input they do.
require 'pp'
require 'nokogiri'
sample_html = File.open("r.htm", "r").read
n = Nokogiri::HTML::parse sample_html
user_names = n.xpath("//div[#class = 'user_name']")
user_details = n.xpath("//div[#class = 'user_details']")
users = user_names.zip(user_details).map do |name, details|
{
name: name.text,
details: Hash[details.xpath("div").map { |d| [d['class'].to_sym, d.text] }]
}
end
pp users
# [{:name=>"Lankesh", :details=>{:country=>"Srilanka", :age=>"9"}},
# {:name=>"Bob", :details=>{:country=>"US", :age=>"54"}},
# {:name=>"Deiter", :details=>{:country=>"Germany", :age=>"34"}},
# {:name=>"Yakob", :details=>{:country=>"Syria", :age=>"90"}},
# {:name=>"Qureshi", :details=>{:country=>"Afgan", :age=>"56"}},
# {:name=>"Smith George", :details=>{:country=>"India", :age=>"23"}}]