How to follow a link with data-params using rvest - html

Question

I'm trying to web scrape a public data provider but I got stuck when I had to click on a button passing a parameter to the JS. Here's my attempt:
require(rvest)
url <- 'https://myterna.terna.it/SunSet/Public/'
page <- url %>% read_html()
node_link <- page %>% html_node('.sub-item:nth-child(1) .postlink')
In node_link I can easily find the target page as the href of this HTML tag:
<a href="/SunSet/Public/Pubblicazioni"
class="postlink"
data-params="filter.IdSezione=52767620567B3077E053A8829B0A9478">
The point is that I cannot easily retrieve the content of the linked page because there are other buttons that point to the same link. The only difference between the various buttons is the data-params attribute which probably has to be given to the JS in order to retrieve the specific content.
Any ideas on how to solve the issue?

Answer 1

Obligatory heads-up:
It's not really clear if the site allow scraping, the Legal Notice says Authorization is granted for the reproduction of documents published on this website exclusively for personal use and not for commercial purposes, provided the name of source is properly indicated.
Use this respecting their terms of service.
Inspecting the network activity when clicking on that link, we can see the webpage makes a POST request to https://myterna.terna.it/SunSet/Public/Pubblicazioni/List. We can find both the requested headers and the params sent.
par <- '{"draw":1,"columns":[{"data":0,"name":"","searchable":true,"orderable":true,"search":{"value":"","regex":false}},{"data":1,"name":"","searchable":true,"orderable":true,"search":{"value":"","regex":false}},{"data":2,"name":"","searchable":false,"orderable":false,"search":{"value":"","regex":false}},{"data":3,"name":"","searchable":false,"orderable":false,"search":{"value":"","regex":false}},{"data":4,"name":"","searchable":false,"orderable":false,"search":{"value":"","regex":false}},{"data":5,"name":"","searchable":false,"orderable":false,"search":{"value":"","regex":false}},{"data":6,"name":"","searchable":false,"orderable":false,"search":{"value":"","regex":false}},{"data":7,"name":"","searchable":false,"orderable":false,"search":{"value":"","regex":false}}],"order":[],"start":0,"length":10,"search":{"value":"","regex":false},"filter":{"IdSezione":"52767620567B3077E053A8829B0A9478","Titolo":"","Id":"","ExtKey":"","TipoPubblicazione":"","SheetName":"","Anno":"2017","Mese":"7","Giorno":"","DataPubblicazione":"","TipoDatoPubblicazione":""},"details":{}}'
This is json, we can parse and change its values if we want (although I tried a few different filters and it does no respond much)
par <- jsonlite::fromJSON(par)
par$filter$Mese <- '7'
As for headers only X-Requested-With:MLHttpRequest is really needed so we can cut it down to that.
response <- POST('https://myterna.terna.it/SunSet/Public/Pubblicazioni/List',
add_headers('X-Requested-With' = 'XMLHttpRequest'),
body = par,
encode = 'json')
json_data <- content(response)$data
This returns a list, that we can safely transform to a dataframe for convenient use:
df <- data.frame(matrix(unlist(json_data), nrow=length(json_data), byrow=TRUE))
head(df, 2)
#> X1
#> 1 SbilanciamentoAggregatoZonale_SegnoGiornaliero_Orario_20170709
#> 2 SbilanciamentoAggregatoZonale_SegnoGiornaliero_QuartoOrario_20170709
#> X2
#> 1 /Date(1499680800000)/
#> 2 /Date(1499680800000)/
#> X3
#> 1 <div class="actions detail-inline export" data-pk="53F4A57FCB70304EE0532A889B0A7758"></div>
#> 2 <div class="actions detail-inline export" data-pk="53F4A57FCB6D304EE0532A889B0A7758"></div>
#> X4 X5 X6
#> 1 53F4A57FCB70304EE0532A889B0A7758 25 SEGNO_MACROZONALE_ORARIO
#> 2 53F4A57FCB6D304EE0532A889B0A7758 25 SEGNO_MACROZONALE_QUARTO_ORARIO
#> X7 X8
#> 1 Segno Giornaliero Orario
#> 2 Segno Giornaliero Quarto Orario

Answer 2

Ok, basicly I was missing the mechanism of how HTTP works. After some days of study I understood the correct approach is using httr package the way showed below.
First of all I retrieved all the settings needed from the public page:
lnkd_url <- paste0(dirname(dirname(url)),
node_link %>%
html_attr('href'))
lnkd_id <- strsplit(zs_node %>%
html_attr('data-params'), '=')[[1]][2]
Then it is possible to launch the POST request to the target page:
lnkd_page <- POST(lnkd_url,
body = list('filter.IdSezione' = lnkd_id)
That's it!