Scraping HTML with Rvest: no text - html

Question

I am trying to scrape information from HTML webpages, I have the direct links but cannot for some reason get to the relevant text.
These are two examples of the webpages:
http://151.12.58.148:8080/CPC/CPC.detail.html?A00002
http://151.12.58.148:8080/CPC/CPC.detail.html?A00003
After I read the html, I am left with all the source code aside from the relevant text (which should change from page to page).
For example, the first link gives a page with this:
data di nascita 1872
which is coded, when I inspect it on my browser, as:
<p y:role="datasubset" y:arg="DATA_NASCITA" class="smalltitle">
<span class="celllabel">data di nascita</span>
<span y:role="multivaluedcontent" y:arg="DATA_NASCITA">1872</span>
</p>
however, when I read it with my code:
link <- 'http://151.12.58.148:8080/CPC/CPC.detail.html?A00002'
page <- read_html(link)
write.table(as.character(page), "page.txt")
and I print "page", to check what I am getting, the same part of the code is:
<p y:role=\"datasubset\" y:arg=\"NASCITA\" class=\"smalltitle\">
<span class=\"celllabel\">luogo di nascita</span> 
<span y:role=\"multivaluedcontent\" y:arg=\"NASCITA\"></span>
</p>
without 1872, which is the piece of information I am interested in.
(and also without not sure if that is indicative of anything).
I can't seem to get around it, would anyone have suggestions?
Thank you very much!

Answer 1

To expand a bit further, the site's HTML code loads a bunch of javascript and then has a template which is filled in after the document loads and also uses the query parameter as some type of value that get computed. I tried to just read in the target javascript file and parse it with V8 but there are too many external dependencies.
To read this, you'll need to use something like splashr or seleniumPipes. I'm partial to the former as I wrote it 😎.
Using either requires running an external program. I will not go into how to install Splash or Selenium in this answer. That's leg work you have to do but splashr makes it pretty easy to use Splash if you are comfortable with Docker.
This bit sets up the necessary packages and starts the Splash server (it will auto-download it first if Docker is available on your system:
library(rvest)
library(splashr)
library(purrr)
start_splash()
This next bit tells Splash to fetch & render the page and then retrieves the page content after javascript has done its work:
splash_local %>%
splash_response_body(TRUE) %>%
splash_user_agent(ua_macos_chrome) %>%
splash_go("http://151.12.58.148:8080/CPC/CPC.detail.html?A00002") %>%
splash_wait(2) %>%
splash_html() -> pg
Unfortunately, it's still a mess. They used namespaces and they are fine in XML docs but somewhat problematic the way they've used them here. But we can work around that with some clever XPath:
html_nodes(pg, "body") %>%
html_nodes(xpath=".//*[local-name()='h4' or local-name()='p' or local-name()='span']/text()") %>%
html_text(trim=TRUE) %>%
discard(`==`, "")
## [1] "Abachisti Vittorio" "data di nascita" "1872"
## [4] "luogo di nascita" "Mirandola, Modena, Emilia Romagna, Italia" "luogo di residenza"
## [7] "Mirandola, Modena, Emilia Romagna, Italia" "colore politico" "socialista"
## [10] "condizione/mestiere/professione" "falegname" "annotazioni riportate sul fascicolo"
## [13] "radiato" "Unità archivistica" "busta"
## [16] "1" "estremi cronologici" "1905-1942"
## [19] "nel fascicolo è presente" "scheda biografica" "A00002"
Do this after you're all done with Splash/splashr to remove the running Docker container:
killall_splash()