Nokogiri truncating HTML for some pages, not others - html

Question

We are using Nokogiri to parse data from iTunes. On some pages, it works. On others, it fails and truncates the page mysteriously.
Our code:
# Get iTunes HTML for app bundle
itunes_url = 'https://itunes.apple.com/us/app-bundle/id918236019'
uri = URI.parse itunes_url
http = Net::HTTP.new uri.host, uri.port
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
req = Net::HTTP::Get.new uri.request_uri
resp = http.request req
# Covnert HTML into XML for parsing
bundle_xml = Nokogiri.XML resp.body
bundle_xml.remove_namespaces!
#puts "ERRORS: #{bundle_xml.errors}"
puts "ORIGINAL\n=============\n#{resp.body}\n\n\n\n============="
puts "NOKO\n=============\n#{bundle_xml}"
Even though errors are returned for other iTunes pages, Nokogiri is able to parse the page properly. Basically, most of the elements after a certain element are removed mysteriously by Nokogiri.
Resp.body output: https://gist.github.com/anonymous/33ecfe82e3d22a39375a
Nokogiri output: https://gist.github.com/anonymous/7622ef92bf430889b9f4
i18n (0.6.11, 0.6.9, 0.6.5, 0.6.4, 0.6.1)
io-console (0.3)
journey (1.0.4)
jquery-rails (3.1.1, 3.1.0, 3.0.4, 2.2.1)
json (1.8.1, 1.8.0, 1.7.7, 1.5.5)
kgio (2.8.1, 2.8.0)
mail (2.4.4)
mime (0.4.0, 0.2.0, 0.1)
mime-types (1.25.1, 1.25, 1.24, 1.23, 1.21)
mini_portile (0.6.0, 0.5.3, 0.5.2, 0.5.1)
minitest (2.5.1)
mongo (1.10.0, 1.9.2, 1.9.1)
mongo_mapper (0.12.0)
mongoid (3.1.6)
moped (1.5.2, 1.5.1)
multi_json (1.10.1, 1.9.2, 1.9.0, 1.8.4, 1.8.2, 1.8.0, 1.7.9, 1.7.7, 1.6.1)
mysql2 (0.3.16, 0.3.15)
newrelic_rpm (3.9.0.229, 3.7.3.204, 3.7.2.192, 3.6.6.147)
nokogiri (1.6.1, 1.6.0)

Answer 1

By trying this myself, I think I see the problem. You're parsing an HTML response.
Change
bundle_xml = Nokogiri.XML resp.body
to:
bundle_xml = Nokogiri.HTML resp.body
and see if this works for you.
The HTML parser is much more lenient and handles missing closing tags, etc.

Answer 2

The HTML has invalid markup. Nokogiri says:
>> #doc.errors
[
[0] #<Nokogiri::XML::SyntaxError: htmlParseEntityRef: no name>,
[1] #<Nokogiri::XML::SyntaxError: htmlParseEntityRef: no name>,
[2] #<Nokogiri::XML::SyntaxError: Tag nav invalid>
]
That can make content disappear as Nokogiri attempts to fix up the HTML to make it usable.