R : Two Different Methods of Webscraping Produce Two Different Results? - html

Question

I am trying to scrape the name, address and longitude/latitude coordinates for each name on a website (e.g. www.mywebsite.com). I used the following the code to get the address and name based on this SO post
library(tidyverse)
library(rvest)
library(httr)
library(XML)
# Define function to scrape 1 page
get_info <- function(page_n) {
cat("Scraping page ", page_n, "\n")
page <- paste0("mywebsite.com",
page_n, "?extension") %>% read_html
tibble(title = page %>%
html_elements(".title a") %>%
html_text2(),
adress = page %>%
html_elements(".marker") %>%
html_text2(),
page = page_n)
}
# Apply function to pages 1:10
df_1 <- map_dfr(1:10, get_info)
# Check dimensions
dim(df_1)
[1] 90
Since I did not know how to modify the above code to extract the coordinates, I wrote a separate script to scrape them:
# Recognize pattern in websites
part1 = "www.mywebsite.com"
part2 = c(0:55)
part3 = "?extension"
temp = data.frame(part1, part2, part3)
# Create list of websites
temp$all_websites = paste0(temp$part1, temp$part2, temp$part3)
# Scrape
df_2 <- list()
for (i in 1:10)
{tryCatch({
url_i <-temp$all_websites[i]
page_i <-read_html(url_i)
b_i = page_i %>% html_nodes("head")
listanswer_i <- b_i %>% html_text() %>% strsplit("\\n")
df_2[[i]] <- listanswer_i
print(listanswer_i)
}, error = function(e){})
}
# Extract long/lat from results
lat_long = grep("LatLng", unlist(df_2[]), value = TRUE)
df_2 = data.frame(str_match(lat_long, "LatLng(\\s*(.*?)\\s*);"))
In the end, scraping the first 10 pages for name/address resulted in 90 entries, but scraping the same 10 pages for the longitude/latitude resulted in 96 entries:
dim(df_1)
[1] 90
dim(df_2)
[1] 96 3
Can someone please help me understand why this is happening and what can I do to fix this?
In the end, I would to make a final table (using df_1 and df_2) that looks something like this:
id name address long lat
1 1 name1 address1 long1 lat1
2 2 name2 address2 long2 lat2
3 3 name3 address3 long3 lat3
Thanks!
Note: I understand that its possible that some names might be missing their latitude/longitudes, and it might not be possible to have the dimensions of "df_1" match the dimensions of "df_2". If this is the case, would it be somehow possible to find out which names are missing their latitude/longitudes (e.g. replace the latitude/longitude entries with NULL for those cases)? For example - suppose the latitude/longitude was not available for "name3":
id name address long lat
1 1 name1 address1 long1 lat1
2 2 name2 address2 long2 lat2
3 3 name3 address3 NA NA

Answer 1

The Problem
The problem is that your second code snippet is not filtering out strings that contain "LatLng" but do not provide coordinates.
After your second code snippet finished scaping the pages, you do the following:
lat_long = grep("LatLng", unlist(df_2[]), value = TRUE)
If you look at the output of this with print(lat_long), you would see a bunch of rows with coordinates. In fact, you'd see exactly 90 such rows because that's how many providers appeared on all those pages. However, you'd also see rows with the string "\t\t\t\tvar bounds = new google.maps.LatLngBounds();". If you go back to the raw HTML you grabbed, you'd see this appears occasionally. Accordingly, you need to remove these rows.
I thought that perhaps you accomplished this with the remaining code, but you never actually remove them. For example, the below code just produces an object filled with NA values. I don't think this does what you want:
as.numeric(gsub("([0-9]+).*$", "\\1", lat_long))
Additionally, the below retains those values as well:
data.frame(str_match(lat_long, "LatLng(\\s*(.*?)\\s*);"))
The Solution
You need to drop elements without coordinates. You'll notice that those elements all contain the substring "LatLngBounds();", so you can just filter them out once they're in a data.frame like below, or using regex.
df_2 %>% filter(X1 != "LatLngBounds();")
Note that this will actually produce 86 rows instead of 90. So, now we're actually short 4 rows. This is because you are not actually collecting all of the GPS coordinates for everyone on the provider page. You can know this because every provide has an address in df_1 and the coordinates are simply passing those addresses to the Maps API.
Why aren't you getting all of the coordinates? My guess is two reasons. First, you are scraping coordinate based on the marker substring. This marker indicates markers/pin on the map. Since the number of pins on the map need not equal the number of providers on the page, you will miss some providers. The less likely issue may have to do with the Google Maps API. If you visit the URLs you create to scrape from (example], you'll see in the bottom left that the Google Maps widget contains the error "This page didn't load Google Maps correctly. See the JavaScript console for technical details". If you look at the JS console, you'll see that an invalid Google Maps API Key was provided. This seems like a likely issue since (a) there is one "LatLngBounds" row per page you are scraping and (b) the row after each of those rows contains coordinates that are not necessarily anywhere near the providers (mine initializes in the U.S. West Coast while the providers are in Canada). I don't know if this has any consequence, but it would explain it if the marker issue isn't the driver.
However, all of this is mostly irrelevant since you don't even need to scrape the coordinates in the first place. You have a list of addresses: you can GeoCode them yourself! There are different ways of doing this, but you can replicate what the site is doing by simply passing them to the Google Maps API! For step-by-step instructions on how to do this, see here.
Identifying the Problem
To provide a better idea of how to approach similar problems in the future, I'll show how I worked through this. One way to approach issue like this is to start by ruling out possible explanations.
Why the problem isn't "missing coordinates"
If the issue was that names are missing coordinates, we would expect nrow(df1) > nrow(df2). However, you reported the opposite: nrow(df2) > nrow(df1).
Why the problem isn't the first code snippet
Since each page contains 9 providers (at least until the last page) and you are scraping 10 pages, we expect to return 9*10 = 90 elements. As you noted, the first code snippet returns an object with 90 rows while the second code snippet returns an object with 96 rows. The second code snippet must be the issue.
Why the problem isn't the pages
Looking at your code, I noticed that you're scraping different pages. Your code to produce df1 iterates over the values of page_n in the interval 1:10. In contrast, your code to produce df2 iterates over the values of page_n in the interval 0:9. This is because the latter code extracts the values of all_websites at indices 1:10, which happen to be the value 0:9 since all_websites is simply the vector 0:55. Since page_n == 0 returns the same page as page_n == 1, your first code is scaping pages 1:10 and your latter code is scraping page c(1,1:9). This means that the values contained in df1 and df2 will differ.
However, this cannot explain the discrepancy in the dimensionality of the two objects since they would still be expected to return 90 rows!