rvest emails scraping on page with complex node structure (html nodes) - html

Question

I am trying to collect names and email addresses from this page "https://www.gu.se/en/about/find-staff?affiliation_types=Teaching%20staff&hits=2744". I am having hard times to figure out the correct way to select the nodes. For example I am doing the following to select people names, but it selects the wrong node.
Thank you in advance for your help
library(rvest)
library(tidyverse)
r<-read_html("https://www.gu.se/en/about/find-staff?affiliation_types=Teaching%20staff&hits=2744")
people_name <- r %>%
html_nodes("a span") %>%
html_text()

Answer 1

As #QHarr mentioned in the comments, the data in the webpage is generated dynamically. The html code you get in your read_html does not yet have the data you need. You could use RSelenium, but in this case I think rvest is better.
If you look at the Developer Tools in chrome (see image below), you will see that when you load the webpage, it makes several subsequent requests. One of them is to the url #QHarr mentioned that returns a json string with all the data the then populates the website using javascript.
So, you can make a request directly to this url, get the json string and parse the
json string so you can get the data directly (this is much lighter than using RSelenium). Sometimes this does not work, because you may need to set state variables in the request to the server or make a complicated POST request. But in this case it is a simpler GET request and it worked!
The json response is a nested list so you need to look at it and identify where is the data you need for each person.
Here is my code:
library(rvest)
library(dplyr)
url.1 <- 'https://www.gu.se/api/search/rest/apps/gu/searchers/person_en?q=*&sort=relevance&affiliation_types=Teaching+staff&hits=2744'
# get json string and parse it to list using jsonlite::fromJSON
json.content <- read_html(url.1) %>% html_node('body') %>% html_text() %>%
jsonlite::fromJSON(simplifyVector = FALSE)
# the list of people is in json.content$documentList$documents (also a nested list)
# use plyr::ldply to get info from each person and combine into a dataframe
df.staff <- plyr::ldply(json.content$documentList$documents,
.fun = {function(x){
name = x$title
aff = ifelse(length(x$affiliations[[1]]$affiliation_name) > 0,
x$affiliations[[1]]$affiliation_name,
NA)
email = ifelse(length(x$affiliations[[1]]$email[[1]]) > 0,
x$affiliations[[1]]$email[[1]],
NA)
dept = ifelse(length(x$affiliations[[1]]$organization) > 0,
x$affiliations[[1]]$organization,
NA)
data.frame(name=name,
affiliation=aff,
email=email,
department=dept)}})
head(df.staff)
# name affiliation email department
#1 Zareen Abbas SENIOR LECTURER zareen.abbas#gu.se Department of Chemistry & Molecular Biology
#2 Yehia Abd Alrahman POSTDOCTOR yehia.abd.alrahman#gu.se Formal Methods
#3 Afrah Abdulla SENIOR LECTURER afrah.abdulla#gu.se Unit for General Didactics and Pedagogic Work
#4 Behjat Omer Abdulla LECTURER behjat.o.a#akademinvaland.gu.se The Crafts and Fine Art Unit
#5 Frida Abel Docent frida.abel#gu.se Department of Laboratory Medicine
#6 Andreas Martin Abel SENIOR LECTURER abela#chalmers.se Computer Science (CS)