How to loop through all items in xpath - html

Question

I'm new to both xpath and html so I'm probably missing something fundamental here. I have a html where I want to extract all the items displayed below. (I'm using scrapy to do my requests, I just need the proper xpath to get the data)
enter image description here
Here I just want to loop through all these items and from there and get some data from inside each item.
for item in response.xpath("//ul[#class='feedArticleList XSText']/li[#class='item']"):
yield {'name': item.xpath("//div[#class='intro lhNormal']").get()}
The problem is that this get only gives me the first item for all the loops. If I instead use .getall() I then get all the items for every loop (which in my view shouldn't work since I thought I only selected one item at the time in each iteration). Thanks in advance!

Answer 1

It seems you're missing a . in your XPath expression (to "indicate" you're working from a context node).
Replace :
yield {'name': item.xpath("//div[#class='intro lhNormal']").get()}
For :
yield {'name': item.xpath(".//div[#class='intro lhNormal']").get()}

Answer 2

You do miss smth. fundamental. Python by default does not have xpath() function.
You better use bs4 or lxml libraries.
See an example with lxml:
import lxml.html
import os
doc = lxml.html.parse('http://www.websters-online-dictionary.org')
if doc:
table = []
trs = doc.xpath("/html/body/div[1]/div[1]/table[1]/tbody/tr")
for tr in islice(trs, 3):
for td in tr.xpath('td'):
table += td.xpath("/b/text() | /text()")
buffer = ''
for i in range(len(table)):
buffer += table[i]
the full explanation is here.