Get data from URL json in R - json

Question

I'm using R and I would like to get JSON information from url and I have around 5000 user agent to sent to this API (http://www.useragentstring.com/pages/api.php)
I use this code to create the url and concatenate the user-agent:
url_1<-paste(" \"http://www.useragentstring.com/?uas=",uaelenchi[11,1],"&getJSON=all\"",sep = '');
json_data2<-fromJSON(readLines(cat(url_1)))
But I receive this error:
Error in readLines(cat(url_1)) : 'con' is not a connection
Any suggestions would be really appreciated! Thanks

Answer 1

I use rjson::fromJSON(file = paste(your_url)). If you make a reproducible example, I could check if it is working in your case.

Answer 2

library(httr)
library(jsonlite)
library(purrr)
uas <- c("Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0",
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:17.0) Gecko/20100101 Firefox/17.0",
"Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.6 Safari/537.11",
"Mozilla/5.0 (X11; OpenBSD amd64; rv:28.0) Gecko/20100101 Firefox/28.0",
"Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.6 Safari/537.11",
"Mozilla/5.0 (X11; OpenBSD amd64; rv:28.0) Gecko/20100101 Firefox/28.0",
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:14.0) Gecko/20120405 Firefox/14.0a1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:14.0) Gecko/20120405 Firefox/14.0a1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1944.0 Safari/537.36")
parse_uas <- function(uas) {
res <- GET("http://www.useragentstring.com/", query=list(uas=uas, getJSON="all"))
stop_for_status(res)
content(res, as="text", encoding="UTF-8") %>%
fromJSON(res, flatten=TRUE) %>%
as.data.frame(stringsAsFactors=FALSE)
}
map_df(uas, parse_uas)
To save API calls you should add a caching layer to the parse_uas() function, which could be done pretty easily with the memoise package:
library(memoise)
.parse_uas <- function(uas) {
res <- GET("http://www.useragentstring.com/", query=list(uas=uas, getJSON="all"))
stop_for_status(res)
content(res, as="text", encoding="UTF-8") %>%
fromJSON(res, flatten=TRUE) %>%
as.data.frame(stringsAsFactors=FALSE)
}
parse_uas <- memoise(.parse_uas)
Also, if you're on Linux, you can also try this package (it doesn't compile well on macOS and not at all on Windows IIRC) which will do all the processing locally.