Extract text between two html horizontal lines with rvest package - html

Question

Here is a big web document : https://gallica.bnf.fr/ark:/12148/bpt6k5619759j.texteBrut . I know how to extract the text with
library(rvest)
library(magrittr)
page_url<- "https://gallica.bnf.fr/ark:/12148/bpt6k5619759j.texteBrut"
page_html<- read_html(page_url)
document <- page_html %>%
html_nodes("hr") %>%
html_text()
document
[1] "Rappel de votre demande:"
[2] "Format de téléchargement: : Texte"
[3] "Vues 1 à 544 sur 544"
[4] "Nombre de pages: 544"
[5] "Notice complète:"
[6] "Titre : Oeuvres complètes de Molière : accompagnées de notes tirées de tous les commentateurs avec des remarques nouvelles. Monsieur de Pourceaugnac / par M. Félix Lemaistre"
[7] "Auteur : Molière (1622-1673). Auteur du texte"
[8] "Auteur : Voltaire (1694-1778). Auteur du texte"
[9] "Auteur : La Harpe, Jean François de (1739-1803). Auteur du texte"
[10] "Auteur : Auger, Louis-Simon (1772-1829). Auteur du texte"
However, it's important for me to track the page from which the text was extracted. The start and the end of a page is actually represented by an horizontal line as you can see here https://gallica.bnf.fr/ark:/12148/bpt6k5619759j.texteBrut. So instead of retrieving a vector in which each element represent a row of the document, I want to have a list in which each element is a page, and each page is a vector in which each element is a row of the document. Something like
[[1]]
[1] "avurrbbihevyupsexvgymphjhdiqtfxzlwrbzpuqqpcxtlyrmyfxewydqnwqpinafaajvhylgaerlqilsvlwnscbiwoyinwjoudu"
[2] "gcgyuizpzznacdnrucvcjajjkbfahvlqqcoudbhpvuuvgrefpglnweznrimuzuydbzjzvhqezmjqtndzdhvvvbnhyipujusjmbhf"
[3] "caugvpyabksaqgktlrcoghkgjaqglpicgcngovvecesasevcdsmimysvrojvpwhbewxfwhdysvdcwmgxlziajwhilclecnkobmnc"
[4] "vuskqpyfqvqexilxqbhviqbdhhldprgdhifwzvhhvcclmljdgqmzsjrvlosftjshpuhxyjfsmfkqsxhaafysgesxwtoechrtekhy"
[[2]]
[1] "muvahkvftgglaphbzfehpnzvemhzixawlvadoxncmtmtzhqjlciozhgspnrusbkycgoqovxslusonmgqehbajbwpcldjquxchsvx"
[2] "pnhpzpbhjvqhehmlchncmgnhapaoqncvezaphilrpqguetutczpydrqthgdhwjtmlfhgvqvofdcylefrmergbkkwnsxlojgyaagw"
[3] "okjhxdpliykzbmdaghtgnsqftxhgpmkpsmiknuugejnrqmzaxqdljnbroxensegyxpikhzwkfzrqairvdhcvglcelnexvcypjkrx"
[4] "ftrbacjpwgmiuwbprvdkfpplycthukvycsyrjwsrokrrvcylzaxxdsgwlctglqaylegeflnlodttkiincavtncxttegstkgvvqgo"
[[3]]
[1] "ndnsdtqxpatoigobldauekhqdbcgvyqmcwyvmcvaredlrfjafiidwvcczqmufvufwjtdhordkaauukjezkyaodffohbzrnhwvioi"
[2] "ywryphperpsnbuspbfengmlllevavpbebfquiguvahshxdleyutvknsfiqcvrsirajqkzppbutsfbspjoirnqacoipcfxisugrto"
[3] "ivuzuxpflzqyphbnsdwvrqwcblxfagdflhqpgldnxkpuhzlhapueowofcgnakgwajgnaaqcvqxzwmorcmjybljsioulscnnntbmx"
[4] "cpbjxincbyrdasbrgrfdzxdzlmogfjmezgdkswpmcjrrlonsvgsaccrjvpbholodgsdcwslpsylslhoxliarkbighsmffoxprffb"

Answer 1

library(stringi)
library(rvest)
library(tidyverse)
Cache the page since it's big and loads really slowly:
if (!file.exists("~/Data/forso.html")) {
read_html(
"https://gallica.bnf.fr/ark:/12148/bpt6k5619759j.texteBrut"
) -> pg
write_lines(as.character(pg), "~/Data/forso.html")
}
Read it in as lines. This is usually a really bad idea for working with HTML but it's better for this process since the XPath required for dealing with text between sequences of tags is gnarly and slow (even just finding the <hr> elements felt kinda slow using html_nodes():
doc <- read_lines("~/Data/forso.html")
Now, find all the <hr> elements, ignoring the first two since they are after the intro/metadata section:
pos <- which(doc == "<hr>")[-(1:2)]
Create start/end index marker positions the text:
starts <- head(pos, -1)
ends <- tail(pos, -1)
Iterate along the start/end positions, extract the text, split it into lines and make a data frame:
map_df(seq_along(starts), ~{
start <- starts[.x]
end <- ends[.x]
data_frame(
pg = .x,
txt = read_html(paste0(doc[start:end], collapse="\n")) %>%
html_children() %>%
html_text() %>%
stri_split_lines() %>%
flatten_chr() %>%
list()
)
}) -> xdf
Take a look:
xdf
## # A tibble: 542 x 2
## pg txt
## <int> <list>
## 1 1 <chr [4]>
## 2 2 <chr [2]>
## 3 3 <chr [13]>
## 4 4 <chr [1]>
## 5 5 <chr [35]>
## 6 6 <chr [19]>
## 7 7 <chr [22]>
## 8 8 <chr [18]>
## 9 9 <chr [16]>
## 10 10 <chr [36]>
## # ... with 532 more rows
Another look:
glimpse(xdf)
## Observations: 542
## Variables: 2
## $ pg <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, ...
## $ txt <list> [<"OEUVRES COMPLETES ", "DE MOLIERE ", "TOMI: III ", "">, <"PARIS. — I1IP. SIMON RAÇON ET COUP., RUE D...
One more:
str(head(xdf))
## Classes 'tbl_df', 'tbl' and 'data.frame': 6 obs. of 2 variables:
## $ pg : int 1 2 3 4 5 6
## $ txt:List of 6
## ..$ : chr "OEUVRES COMPLETES " "DE MOLIERE " "TOMI: III " ""
## ..$ : chr "PARIS. — I1IP. SIMON RAÇON ET COUP., RUE D'ERFURTH, 1. " ""
## ..$ : chr "OEUVRES COMPLETES " "DE MOLIERE " "NOUVELLE ÉDITION " "ACe-OJIPAfi NEES DE NOTES TIRÉES DE TOUS L, E S COMMENTATEURS AVEC DES REMARQUES NOUVELLES " ...
## ..$ : chr ""
## ..$ : chr "OEUVRES " "COMPLÈTES " "DE MOLIÈRE " "MONSIEUR DE POURCEAUGNAC' " ...
## ..$ : chr "MONSIEUR DE POURCEAUGNAC. " "MATASSINS dansants. DEUX AVOCATS chantants. DEUX PROCUREURS dansants. DEUX SERGENTS dansants. TROUPE DE MASQUES"| __truncated__ "La scène est à Paris. " "ACTE PREMIER " ...
This captures empty lines as well, but I have no idea what you need outside of what you described.

Answer 2

another approach
As #hrbrmstr already mentioned in his answer, xpath is not very friendly if you want to extract nodes between other nodes... Things get very inefficient, very fast...
So, keep in mind that the following code will take up several minutes to complete (or longer, depending on your machine)... (maybe an other user can speed things up using this answer as a base).
Having said that:
library( xml2 )
library( data.table )
#get the contents od the webpage
doc <- read_html( "https://gallica.bnf.fr/ark:/12148/bpt6k5619759j.texteBrut" )
#determine how many hr-tags/nodes are there in the document
hr <- length( xml_nodes( doc, "hr") )
#create an empty list
l <- list()
#fill the list with a loop. This seems to take forever, but is works!
# just be patient (and get a cup of coffe. or two...).
for( i in seq(1, hr, by = 1) ) {
#set up the xpath.
#xpath: get all p-nodes after the i-th hr-nodes, that have exactly i preceding hr-nodes
xpath_ <- paste0 ( ".//hr[", i, "]/following-sibling::p[count(preceding-sibling::hr)=", i, "]" )
#
l[[i]] <- xml_find_all( doc, xpath = xpath_ ) %>% xml_text() %>% data.table()
}
some results
l[1:5]
# [[1]]
# Empty data.table (0 rows) of 1 col: .
#
# [[2]]
# Empty data.table (0 rows) of 1 col: .
#
# [[3]]
# .
# 1: OEUVRES COMPLETES
# 2: DE MOLIERE
# 3: TOMI: III
#
# [[4]]
# .
# 1: PARIS. — I1IP. SIMON RAÇON ET COUP., RUE D'ERFURTH, 1.
#
# [[5]]
# .
# 1: OEUVRES COMPLETES
# 2: DE MOLIERE
# 3: NOUVELLE ÉDITION
# 4: ACe-OJIPAfi NEES DE NOTES TIRÉES DE TOUS L, E S COMMENTATEURS AVEC DES REMARQUES NOUVELLES
# 5: PAR FÉLIX L E M A I T R E
# 6: P R É C É D É E
# 7: DE LA VIE DE MOLIÈRE PAR VOLTAIRE
# 8: TOME TROISIEME
# 9: PARIS
# 10: GARNIER FRÈRES, LIBRAIRES-ÉDITEURS
# 11: G, RUE DES SAINTS-PÈRES, ET P A L A I S-R 0 V A I., 213
# 12: 8 6 7
or bind everything together in a data.table
dt <- rbindlist(l, use.names = TRUE, idcol = "page")
# page .
# 1: 3 OEUVRES COMPLETES
# 2: 3 DE MOLIERE
# 3: 3 TOMI: III
# 4: 4 PARIS. — I1IP. SIMON RAÇON ET COUP., RUE D'ERFURTH, 1.
# 5: 5 OEUVRES COMPLETES
# 6: 5 DE MOLIERE
# 7: 5 NOUVELLE ÉDITION
# 8: 5 ACe-OJIPAfi NEES DE NOTES TIRÉES DE TOUS L, E S COMMENTATEURS AVEC DES REMARQUES NOUVELLES
# 9: 5 PAR FÉLIX L E M A I T R E
# 10: 5 P R É C É D É E
# 11: 5 DE LA VIE DE MOLIÈRE PAR VOLTAIRE
# 12: 5 TOME TROISIEME
# 13: 5 PARIS
# 14: 5 GARNIER FRÈRES, LIBRAIRES-ÉDITEURS
# 15: 5 G, RUE DES SAINTS-PÈRES, ET P A L A I S-R 0 V A I., 213
# 16: 5 8 6 7
# 17: 7 OEUVRES
# 18: 7 COMPLÈTES
# 19: 7 DE MOLIÈRE
# 20: 7 MONSIEUR DE POURCEAUGNAC'

Answer 3

Finding an index of all the hr nodes is a straightforward way to go about it. The mutate section is the most notable part which uses %in% and cumsum.
# set up and read
library(rvest)
library(xml2)
library(dplyr)
page_url<- "https://gallica.bnf.fr/ark:/12148/bpt6k5619759j.texteBrut"
page_html<- read_html(page_url)
# filter to body only, so no need to deal with child nodes
allbodynodes <- page_html %>%
xml_node('body')
# get all nodes and all hr nodes to compare later
# the first could be put into the pipeline, but it's more clear to me here
allnodes <- allbodynodes %>%
xml_nodes('*')
allhr <- allbodynodes %>%
xml_nodes('hr')
alltext <- allnodes %>%
html_text(trim = T) %>% # convert to text only
as.data.frame(stringsAsFactors = F) %>% # put into dataframe
select(maintext = '.') %>% # give the text a variable name
mutate(
ishr = allnodes %in% allhr, # check which nodes were <hr> (now blank)
page = cumsum(ishr) + 1 # add page number by running across the hr
) %>%
filter(!ishr) %>% # get rid of blank hr lines
select(-ishr) # get rid of all false ishr column
# split into a list of sorts if desired
alltextlist <- split(alltext$maintext,alltext$page)
I hope there's a more succinct way to create the index (preferably within the dplyr pipeline), but I haven't found it yet.