Scrapy, how to extract s subtext from <b> - html

Question

I got a html like this:
<section id="SECTION_A">
<h4>List</h4>
<a style="text-decoration: none;" href="#list" data-toggle="collapse">
<div class="ITEM">
TEXT
</div>
</a>
<div id="IDENTIFICATION" class="collapse">
</div>
<a style="text-decoration: none;" href="#list" data-toggle="collapse">
<div class="ITEM2">
TEXT2
</div>
</a>
<div id="IDENTIFICATION2" class="collapse">
<div><b>TITLE</b>: CONTENT</div>
<div><b>TITLE2</b>: CONTENT2</div>
</div>
</section>
I've got stored it in a selector XPATH like this, because the html got several sections with similar structure, tags and repeated data:
sectionA = response.xpath('//section[#id="SECTION_A"]')
Now, I want to extract the ITEMS and their IDENTIFICATIONS and write them into a file.
Extracting the ITEM gave no problem with:
item = sectionA.xpath('.//div/#class[contains(.,"ITEM")]').extract()
And it returns:
[u'ITEM', u'ITEM2']
But I cannot extract the TEXT of the ITEMS, I've tried:
item = sectionA.xpath('.//div/#class[contains(.,"ITEM")]/text()').extract()
But returns an empty list.
I'm also unable to extract the IDENTIFICATIONS, one problem with these is that they may not have any content or several, so I've tried to extract a selector of them from the SECTIONA selector like this:
identifications = sectionA.xpath('.//div/#id[contains(.,"IDENTIFICATION")]')
It retunrs me a selector similar to sectionA, but when I try to search in it i got nothing with this:
for id in identifications:
title= signature.xpath('.//div')
I've tried sevelal combinations like .//div/b or .//b or just .// but i got nothing.
Anyone know how I can get the ITEM-TEXT and IDENTIFICATIONS-CONTENT from an html like this?

Answer 1

The problem you are facing is not in the steps applied but is a Logical mistake. The reason why you are not getting the Text inside the 'ITEM' class is due to an extra / that you are using.
In the code that you wrote :
item = sectionA.xpath('.//div/#class[contains(.,"ITEM")]').extract()
Here it returns [u'ITEM', u'ITEM2'] due to the use of / before #class in //div/#class , which basically here means : return me the value used in the class which contains "ITEM" substring in it. And since the attribute #class is being pointed to here, it returns [] for there is no text to be pointed to.
What you instead want to do is :
item = sectionA.xpath('.//div[contains(#class,"ITEM")]/text()').extract()
Here the output of sectionA.xpath('.//div[contains(#class,"ITEM")]') is the selector:
[<Selector xpath='.//div[contains(#class,"ITEM")]' data=u'<div class="ITEM">'>, <Selector xpath='.//div[contains(#class,"ITEM")]' data=u'<div class="ITEM2">'>]
Similar mistake is made in the extraction for "IDENTIFICATIONS", with one more grave Logical Problem. The usage of // in title = signature.xpath('.//div') is not the appropriate method since it will not show the div in just the div IDENTIFICATIONS, but will instead try with all divs preset in the HTML. Again, this may not be a problem unless there was a div with substring "IDENTIFICATION" outside the div we are searching in. So a better way to do it, instead is to do something similar follows as per requirement :
>>> identification=sectionA.xpath('.//div[contains(#id,"IDENTIFICATION")]')
>>> for id in identification:
... print(id.xpath('div/b')).extract()