Scraping WordPress reviews - html

Question

I am just learning R programming. For an exercise, I want to scrape reviews of a WordPress plugin that seems to have been discontinued here
I start by specifying the URL
> url <- 'https://wordpress.org/plugins/demo-data-creator/#reviews'
Scraping the HTML content from the URL
> url <- read_html('https://wordpress.org/plugins/demo-data-creator/#reviews')
Extract the title of each review using the ID tag
> reviews <- html_nodes(url, 'h3.review-title')
Strip out the HTML tags, leaving only the content of the title
> titletext <- html_text(reviews)
Print the titles scraped
> head(titletext)
> [1] "Good for development"
> [2] "Used it for creating test users"
> [3] "Excelent! negative comments come from people who doesn't read!"
> [4] "Thanks"
> [5] "Does EXACTLY what it says it will – thanks! Very Handy"
> [6] "Dangerous plugin"
I repeat the same for the contents of the reviews
> reviewcontent <- html_nodes(url, 'div.review-content')
> reviewtext <- html_text(reviewcontent)
And prints out
> head(reviewcontent)
> {xml_nodeset (6)} [1] <div class="review-content">Good and handy tool
> for deve ... [2] <div class="review-content">This plugin came in very
> han ... [3] <div class="review-content">Does exactly what it offers!
> ... [4] <div class="review-content">Thanks</div> [5] <div
> class="review-content">Very handy for a test system ... [6] <div
> class="review-content">I have to agree with viesli ...
However, I realized it didn't scrape all the reviews as there are more here
Is there a way to tell R to check each review listed to extract the title and review content and probably populate in a table?

Answer 1

You can use the same approach to extract the reviews from the second link. The main difference is that the content of each review is in its own page. Hence, you need two steps:
Extract the list of review page URLs from the main page.
For each URL, fetch the page, and extract the title and content of the review.
For step 1, it is very similar to what you already did, except that you are now trying to extract the URLs for the review pages. If you inspect that page, you'll see that these links (<a> elements) have CSS class bbp-topic-permalink.
So, we can extract them using:
links <- html_nodes(page, css='a.bbp-topic-permalink')
Now, we don't want the text part of the tag, but rather the href attribute value (where the link is pointing). We can extract that using
reviewurls <- html_attr(links, 'href')
For step 2, we will loop over the list of reviewurls, and for each one, fetch the page using read_html, extract the title and the content using html_node and html_text, then add them to a table/matrix/data.frame.
The loop can be done using:
for (u in reviewurls) {
}
Inside the loop, u is the variable that holds the current review URL. We will use read_html to read the page, then extract the title and content.
Inspecting the review page, the title is in an <h1> tag with CSS class page-title. Simiarly, thThe content of the review are inside a <div> with CSS class bbp-topic-content.
So, inside the loop, you can do this:
page = read_html(u)
reviewT = html_text(html_node(page, css='h1.page-title'))
reviewC = html_text(html_node(page, css='div.bbp-topic-content'))
Now you will have both the title and the content for that particular review. You can add them to a list, so that by the end of the loop, you will have titles and contents of all the reviews.
The final code will look like this:
url <- 'https://wordpress.org/support/plugin/demo-data-creator/reviews/'
page <- read_html(url)
links <- html_nodes(page, css='a.bbp-topic-permalink')
reviewurls <- html_attr(links, 'href')
# Two empty lists, to be populated inside the loop
titles = c()
contents = c()
for (u in reviewurls) {
page = read_html(u)
reviewT = html_text(html_node(page, css='h1.page-title'))
reviewC = html_text(html_node(page, css='div.bbp-topic-content'))
titles = c(titles, reviewT)
contents = c(contents, reviewC)
}
Once it's done, you will get:
> length(titles)
[1] 21
> head(titles)
[1] "Good for development"
[2] "Used it for creating test users"
[3] "Excelent! negative comments come from people who doesn't read!"
...
> head(contents)
[1] "\n\n\t\t\t\t\n\t\t\t\tGood and handy tool for development.\n\n\n\n\t\tThis topic was modified 2 years, 10 months ago by Subrata Sarkar.\n\t\n\n"
[2] "\n\n\t\t\t\t\n\t\t\t\tThis plugin came in very handy during development of my own plugin. I used it to create a lot of users and it did exactly what it should.\nNot sure where all the negativity about wiping the database comes from. Are they users that didn’t read all the warnings? Or did older versions of the plugin not warn about wiping all data? Anyway, now it does \U0001f642\n\n\t\t\t\t\n\t\t\t"
[3] "\n\n\t\t\t\t\n\t\t\t\tDoes exactly what it offers! Nothing less.\nPeople complaining is too lazy to read the SEVERAL warnings about the usage of this plugin.\n\n\t\t\t\t\n\t\t\t"
...