I am trying to import a json file in r.
Have installed necessary packages & libraries.
But keep on getting an error.
Error: lexical error: invalid string in json text.
temp.jsonl
(right here) ------^
My code is as below -
library(rjson)
library(jsonlite)
library(RJSONIO)
install.packages("rjson")
install.packages("RJSONIO")
json_data_raw<-fromJSON("temp.jsonl")
Thanks
To read lines of JSON you can use the jsonlite::stream_in function.
df <- jsonlite::stream_in(file("temp.jsonl"))
Reference
The jsonlite stream_in and stream_out functions implement line-by-line processing of JSON data over a connection, such as a socket, url, file or pipe
JSON streaming in R
dat <- fromJSON(sprintf("[%s]", paste(readLines(filepath), collapse=",")))
Was able to fix the issue and create dataframe in r.
Thanks guys.
Related
I have a large .json file and I only want to read in a part of it.
I tried the the following solutions but they didn´t work:
yelp <- stream_in(file("yelp_academic_dataset_review.json"), paigesize = 500)
yelp <- stream_in(file("yelp_academic_dataset_review.json"), nrows = 500)
Anyone know how it works?
First off- always helpful to provide the packages you are using, in your case jsonlite.
One solution is parsing the data file (as a .txt file) prior to streaming it in.
yelp <- readLines("yelp_academic_dataset_review.json")[1:500]
yelp <- stream_in(textConnection(gsub("\\n", "", yelp)))
I'm assuming your file is local?
I have had success with actual piping/streaming json in the past. Ie, from the command line,
cat x.json | parse_json.py
Then you write your python script:
import json,sys
for line in sys.stdin:
js_line = json.loads(line.rstrip())
try:
# do something with js_line['x']['y']
except ValueError:
pass
I'm not sure why you want to use stream_in, but this somewhat manual approach can be effective
I use this code for extracting 1400001 to 1450000 lines of yelp:
setwd("d:/yelp_dataset")
rm(list=ls())
library(jsonlite)
rev<- 'd:/yelp_dataset/review.JSON'
revu<-jsonlite::stream_in(textConnection(readLines(rev)[1400001:1450000],verbose=F)
Cannot find the answer how to load 7z file in R. I can't use this:
s <- system("7z e -o <path> <archive>")
because of error 127. Maybe that's because I'm on Windows? However, 7z opens when I click in TotalCommander.
I'm trying something like this:
con <- gzfile(path, 'r')
ff <- readLines(con, encoding = "UTF-8")
h <- fromJSON(ff)
I have Error:
Error: parse error: trailing garbage
7z¼¯' ãSp‹ Ë:ô–¦ÐÐY#4U¶å¿ç’
(right here) ------^
The encoding is totally not there, when I load this file uncompressed it's ok without specifying the encoding. Moreover it's 2x longer. I have thousands of 7z files need to read them one by one in a loop, read, analyze and get out. Could anyone give me some hints how to do it effectively?
When uncompressed it easily works using:
library(jsonlite)
f <- read_json(path, simplifyVector = T)
EDIT
There are many json files in one 7z file. The above error is probably caused by parser which reads raw data of whole file. I don't know how to link these files or specify the connection attributes.
For my PhD project I need to read a JSONL file into R (the extension isn't json, is jsonl) and transform it in a csv.
I tried to use this code based on jsonlite but it gives me an error:
library(jsonlite)
data <- "/Users/chiarap/tweets.jsonl"
dat <- fromJSON(sprintf("[%s]", paste(readLines(data), collapse=",")))
Error: parse error: unallowed token at this point in JSON text
EFEF","notifications":null}},,,{"in_reply_to_status_id_str":
(right here) ------^
Thanks
If you have a large file, pasting all of the rows together may result in errors. You can process each line separately and then combine them into a data frame.
library(jsonlite)
library(dplyr)
lines <- readLines("myfile.jsonl")
lines <- lapply(lines, fromJSON)
lines <- lapply(lines, unlist)
x <- bind_rows(lines)
Then x is a data frame that you can continue to work with or write to file.
I try to learn how to handle out huge json file in R and convert them to csv in order to make more statistic process with them.
I try to experement with this dataset.
What I have now which calls some packages, take as input the json file (if you extract it is huge) and converts it to json and save it to csv. However the csv have only 48 rows and I think it must have more.
require(RJSONIO)
require(rjson)
library("rjson")
filename2 <- "path for json file"
json_data <- fromJSON(file = filename2)
json <- do.call("rbind", json_data)
write.csv(json,file='C:/json.csv', row.names=FALSE)
I've been trying to import a JSON file into R for a while using both the rjson and RJSONIO package, but I can't get it to work. One of the variations of code I used:
json_file <- "http://toolserver.org/~emw/index.php?c=rawdata&m=get_traffic_data&p1=USA&project1=en&from=12/10/2007&to=4/1/2011"
json_data <- fromJSON(paste(readLines(json_file), collapse=""))
This results in an error message:
Error in fromJSON(paste(readLines(json_file), collapse = "")) :
unexpected character '<'
I think the problem is in the first line of code, because json_file contains the source of the website, not the actual content. I've tried getURL() and getURLContent(), but without any success. Any help would be much appreciated!
Edit: as pointed out by Martin Morgan, the problem seems to be with the URL, not the code!
library(rjson)
fromJSON(readLines('http://toolserver.org/~emw/index.php?c=rawdata&m=get_traffic_data&p1=USA&project1=en&from=12/10/2007&to=4/1/2011')[1])
works for me, with a warning