R web scraping function getPageNumber error - html

Question

I am building a webscraper and trying to understand why my getPage Number Function does not work. The function worked last night and tonight I have been having an error getting the right output
library(rvest)
library(RCurl)
library(XML)
library(stringr)
getPageNumber <- function(URL) {
parsedDocument <- read_html(URL)
results_per_page <- length(parsedDocument %>% html_nodes(".sr-list"))
total_results <- parsedDocument %>%
toString() %>%
str_match(., 'num_results":"(.*?)"') %>%
.[,2] %>%
as.integer()
pageNumber <- tryCatch(ceiling(total_results / results_per_page), error = function(e) {1})
return(pageNumber)
}
getPageNumber("https://academic.oup.com/dnaresearch/search-results?rg_IssuePublicationDate=01%2F01%2F2010%20TO%2012%2F31%2F2010&fl_SiteID=5275&page=")
The output I am getting is NA, when it should be numeric number

Answer 1

Do this to troubleshoot your problems:
getPageNumber <- function(URL) {
parsedDocument <- read_html(URL)
results_per_page <- length(parsedDocument %>% html_nodes(".sr-list"))
total_results <- parsedDocument %>%
toString() %>%
str_match(., 'num_results":"(.*?)"') %>%
.[,2] %>%
as.integer()
browser() ## <-- inject this here to stop when you get to this point.
pageNumber <- tryCatch(ceiling(total_results / results_per_page), error = function(e) {1})
return(pageNumber)
}
getPageNumber("https://academic.oup.com/dnaresearch/search-results?rg_IssuePublicationDate=01%2F01%2F2010%20TO%2012%2F31%2F2010&fl_SiteID=5275&page=")
This is what my R session looks like (I typed total_results and results_per_page there myself to see what they contained):
> getPageNumber("https://academic.oup.com/dnaresearch/search-results?rg_IssuePublicationDate=01%2F01%2F2010%20TO%2012%2F31%2F2010&fl_SiteID=5275&page=")
Called from: getPageNumber("https://academic.oup.com/dnaresearch/search-results?rg_IssuePublicationDate=01%2F01%2F2010%20TO%2012%2F31%2F2010&fl_SiteID=5275&page=")
Browse[1]> debug at #10: pageNumber <- tryCatch(ceiling(total_results/results_per_page),
error = function(e) {
1
})
Browse[2]> total_results
[1] 176
Browse[2]> results_per_page
[1] 20
Browse[2]>
debug at #11: return(pageNumber)
Browse[2]>
[1] 9
>
Other things you can do while you are inside the running function with browser():
put the html somewhere you can inspect it later: cat( toString(parsedDocument), file="~/foo.html")
look at: parsedDocument %>% toString() %>% str_match('num_results":"(.*?)"')
If you still experience NA as output, try changing your function code to this:
getPageNumber <- function(URL) {
parsedDocument <- read_html(URL)
results_per_page <- length(parsedDocument %>% html_nodes(".sr-list"))
total_results <- parsedDocument %>%
toString() %>%
str_match(., 'num_results":"(.*?)"') %>%
.[,2] %>%
as.integer()
pageNumber <- tryCatch(ceiling(total_results / results_per_page), error = function(e) {1})
if( is.na(pageNumber) ) {
cat( "PAGENUMBER IS NA!\n" )
cat( "total_results is: ", total_results, "\n" )
cat( "results_per_page is: ", results_per_page, "\n" )
cat( toString(parsedDocument), file="~/foo.html")
cat( "The HTML is stored here: ", normalizePath("~/foo.html") )
cat( "Open it in a text editor and investigate why it went wrong\n" )
}
return(pageNumber)
}