Is any better way to handle JSON in R | performance improvement - json

Question

Data Preparation
comp <-
c('[{"id": 28, "name": "Google"}, {"id": 12, "name": "Microsoft"}]',
'[{"id": 32, "name": "Microsoft"}, {"id": 878, "name": "Facebook"}]')
id = c(1,2)
jsonData = as.data.frame(id,comp)
jsonData
id
[{"id": 28, "name": "Google"}, {"id": 12, "name": "Microsoft"}] 1
[{"id": 32, "name": "Microsoft"}, {"id": 878, "name": "Facebook"}] 2
I am not sure why 'comp' not came as column name and why 'id' came later if it's defined before, Also its giving error if I define 'as.data.frame(comp,id)'
Now I am dealing with JSON data
library(jsonlite)
library(tidyverse)
library(dplyr)
data <- jsonData %>% mutate(x = lapply(comp,fromJSON)) %>% unnest(x)
data
id id1 name
1 1 28 Google
2 1 12 Microsoft
3 2 32 Microsoft
4 2 878 Facebook
Is there any better way to deal with JSON in R, like any library which directly convert JSON to normal column, currently I am taking small data so its look easy but I have multiple columns having JSON input and Its too much performance hit for my report

Answer 1

JSON is text. Text parsing is slow. Also not sure why library(dplyr) is there since it comes with the tidyverse. And, you should consider reading up on how to make data frames.
Regardless. We'll make an representative example: 500,000 rows:
library(tidyverse)
data_frame(
id = rep(c(1L, 2L), 250000),
comp = rep(c(
'[{"id": 28, "name": "Google"}, {"id": 12, "name": "Microsoft"}]',
'[{"id": 32, "name": "Microsoft"}, {"id": 878, "name": "Facebook"}]'
), 250000)
) -> xdf
There are many JSON processing packages in R. Test out a few. This uses ndjson which has a function flatten() which takes a character vector of JSON strings and makes a "completely flat" structure from it.
I'm only using different data frame variables for explanatory clarity and benchmarking later.
pull(xdf, comp) %>%
ndjson::flatten() %>%
bind_cols(select(xdf, id)) -> ydf
That makes:
ydf
## Source: local data table [500,000 x 5]
##
## # A tibble: 500,000 x 5
## `0.id` `0.name` `1.id` `1.name` id
## <dbl> <chr> <dbl> <chr> <int>
## 1 28. Google 12. Microsoft 1
## 2 32. Microsoft 878. Facebook 2
## 3 28. Google 12. Microsoft 1
## 4 32. Microsoft 878. Facebook 2
## 5 28. Google 12. Microsoft 1
## 6 32. Microsoft 878. Facebook 2
## 7 28. Google 12. Microsoft 1
## 8 32. Microsoft 878. Facebook 2
## 9 28. Google 12. Microsoft 1
## 10 32. Microsoft 878. Facebook 2
## # ... with 499,990 more rows
We can turn that back into a more tidy data frame:
bind_rows(
select(ydf, id = id, id1=`0.id`, name=`0.name`),
select(ydf, id = id, id1=`1.id`, name=`1.name`)
) %>%
mutate(id1 = as.integer(id1))
## Source: local data table [1,000,000 x 3]
##
## # A tibble: 1,000,000 x 3
## id id1 name
## <int> <int> <chr>
## 1 1 28 Google
## 2 2 32 Microsoft
## 3 1 28 Google
## 4 2 32 Microsoft
## 5 1 28 Google
## 6 2 32 Microsoft
## 7 1 28 Google
## 8 2 32 Microsoft
## 9 1 28 Google
## 10 2 32 Microsoft
## # ... with 999,990 more rows
Now, we'll benchmark with 1,000 rows since I'm not waiting for the full 500,000 run to microbenchmark:
data_frame(
id = rep(c(1L, 2L), 500),
comp = rep(c(
'[{"id": 28, "name": "Google"}, {"id": 12, "name": "Microsoft"}]',
'[{"id": 32, "name": "Microsoft"}, {"id": 878, "name": "Facebook"}]'
), 500)
) -> xdf
microbenchmark::microbenchmark(
faster = {
pull(xdf, comp) %>%
ndjson::flatten() %>%
bind_cols(select(xdf, id)) -> ydf
bind_rows(
select(ydf, id = id, id1=`0.id`, name=`0.name`),
select(ydf, id = id, id1=`1.id`, name=`1.name`)
) %>%
mutate(id1 = as.integer(id1))
}
)
## Unit: milliseconds
## expr min lq mean median uq max neval
## faster 12.46409 13.71483 14.73997 14.40582 15.47529 21.09543 100
So:
15ms for 1,000 rows
15ms * 500 = 7.5s for 500,000
If you're not pedantic about the id1 column needing to be an integer, you can likely shave off a few ms.
There are other approaches. And, if you regularly work with columns of JSON data, I highly recommend checking out Apache Drill and the sergeant package.