Trying to determine why my xpath is failing in Scrapy - html

Question

I'm trying to run a Scrapy spider on pages like this:
https://careers.mitre.org/us/en/job/R104514/Chief-Engineer-Technical-Analysis-Department
And I'd like the spider to retrieve the bullet points with qualifications and responsibilities. I can write an xpath expression that gets exactly that, and it works in my browsers:
//*/section/div/ul/li
But when I try to use the Scrapy shell:
response.xpath("//*/section/div/ul/li")
It returns an empty list. Based on copying the response.text and loading it in a browser, it seems like the text is accessible, but I still can't access those bullets.
Any help would be much appreciated!

Answer 1

Looking at the page you have linked, the list items you are targeting are not actually in the document response itself but later loaded into the DOM by JavaScript.
To access these I'd recommend looking at scrapy's documentation on Selecting dynamically-loaded content. The section that applies here in particuler is the Parsing JavaScript code section.
Following the second example, we can use chompjs (you'll need to first install it with pip) to extract the JavaScript data, unescape the html string, and then load it into scrapy for parsing. e.g.:
scrapy shell https://careers.mitre.org/us/en/job/R104514/Chief-Engineer-Technical-Analysis-Department
Then:
import html # Used to unescape the HTML stored in JS
import chompjs # Used to parse the JS
javascript = response.css('script::text').get()
data = chompjs.parse_js_object(javascript)
description_html = html.unescape(data['description'])
description = scrapy.Selector(text=description_html, type="html")
description.xpath("//*/ul/li")
This should output your desired list items:
[<Selector xpath='//*/ul/li' data='<li>Ensure the strength ...