Dealing with Duplicates in a Combined Dataframe - duplicates

first of all a big shout-out and big thank you to all in helping to answer my questions. You guys are amazing.
I would need your help once again in Coding with R.
The situation arises with two Dataframes, where Dataframe1 one describes a Portuguese class and Dataframe2 describes a Math class. I do want to find the duplicate (as there are some, as one student takes both classes) and not delete him, but expand the column "Class" by indicating, he is on both classes, something like "Math+Portuguese".
I tried to simplify my Dataframes (in reality they are much bigger, but the final approach should be the sam) by creating two new ones. There is one duplicate (the student where both parents are doctors). I just want to have him one time in the Dataframe, with the wording "Math+Portuguese" in the column "Class".
For the identification of the duplicates, the column "Grades" has to be ignored.
Thank you very much for you help.
All the best,
Alexander
# Creation of Dataset 1 (Portuguese students)
school <- c(rep("S1",7),rep("S2",3))
Age <- c(18,18,19,19,20,20,21,21,22,22)
professionf <- c(rep("teacher",9),rep("doctor",1))
professionm <- c(rep("police",9),rep("doctor",1))
Class <- rep("Portuguese",10)
Grade <- round(runif(10,1,5),0)
DataframeP <- cbind(school, Age, professionf,professionm,Grade,Class)
View(DataframeP)
#Creation of Dataset 2 (Math students)
school <- c(rep("S1",7),rep("S2",3))
Age <- c(18,18,19,19,20,20,21,21,22,22)
professionf <- c(rep("lawyer",9),rep("doctor",1))
professionm <- c(rep("police",9),rep("doctor",1))
Class <- rep("Math",10)
Grade <- round(runif(10,1,5),0)
DataframeM <- cbind(school, Age, professionf,professionm,Grade,Class)
View(DataframeM)
#Combination of the two Dataframes, where the identification of the dupicates should take place
DF_All <- rbind(DataframeM,DataframeP)
View(DF_All)

That should do it, dear Alexander!
library(data.table)
require(dplyr)
df_merged <- merge(x = DataframeP, y = DataframeM, by = c("school", "Age", "professionf", "penter code hererofessionm"), all = TRUE)
df_merged <- within(df_merged, Class.x[Class.x == 'Portuguese' & Class.y == 'Math'] <- 'Portoguese + Math')
df_merged$Class.x = coalesce(df_merged$Class.x, df_merged$Class.y)
df_merged$Grade.x = coalesce(df_merged$Grade.x, df_merged$Grade.y)
df_merged <- df_merged[1:(length(df_merged)-2)]
setnames(df_merged, old = c('Grade.x','Class.x'), new = c('Grade','Class'))
df_merged

Related

Create a loop within a function so that URLs return a dataframe

I was provided with a list of identifiers (in this case the identifier is called an NPI). These identifiers can be copied and pasted to this website (https://npiregistry.cms.hhs.gov/registry/?). I want to return the name of the NPI number, name of the physician, address, phone number, and specialty.
I have over 3,000 identifiers so a copy and paste is not efficient and not easily repeatable for future use.
If possible, I would like to create a list of URLs, pass them into a function, and received a dataframe that provides me with the variables mentioned above (NPI, NAME, ADDRESS, PHONE, SPECIALTY).
I was able to write a function that produces the URLs needed:
Here are some NPI numbers for reference: 1417024746, 1386790517, 1518101096, 1255500625.
This is my code for reading in the file that contains my NPIs
npiList <- c("1417024746", "1386790517", "1518101096", "1255500625")
npiList <- as.list(npiList)
npiList <- unlist(npiList, use.names = FALSE)
This is the function to return the list of URLs:
npiaddress <- function(x){
url <- paste("https://npiregistry.cms.hhs.gov/registry/search-results-
table?number=",x,"&addressType=ANY", sep = "")
return(url)
}
I saved the list to a variable and perhaps this is my downfall:
npi_urls <- npiaddress(npiList)
From here I wrote a function that can accept a single URL, retrieves the data I want and turns it into a dataframe. My issue is that I cannot pass multiple URLs:
npiLookup <- function (x){
url <- x
webpage <- read_html(url)
npi_html <- html_nodes(webpage, "td")
npi <- html_text(npi_html)
npi[4] <- gsub("\r?\n|\r", " ", npi[4])
npi[4] <- gsub("\r?\t|\r", " ", npi[4])
npiFinal <- npi[c(1:2,4:6)]
npiFinal <- as.data.frame(npiFinal)
npiFinal <- t(npiFinal)
npiFinal <- as.data.frame(npiFinal)
names(npiFinal) <- c("NPI", "NAME", "ADDRESS", "PHONE", "SPECIALTY")
return(npiFinal)
}
For example:
If I wanted to get a dataframe for the following identifier (1417024746), I can run this and it works:
x <- npiLookup("https://npiregistry.cms.hhs.gov/registry/search-results-table?number=1417024746&addressType=ANY")
View(x)
My output for the example returns the NPI, NAME, ADDRESS, PHONE, SPECIALTY as desired, but again, I need to do this for several thousand NPI identifiers. I feel like I need a loop within npiLookup. I've also tried to put npi_urls into the npiLookup function but it does not work.
Thank you for any help and for taking the time to read.
You're most of the way there. The final step uses this useful R idiom:
do.call(rbind,lapply(npiList,function(npi) {url=npiaddress(npi); npiLookup(url)}))
do.call is a base R function that applies a function (in this case rbind) to the list produced by lapply. That list is the result of running your npiLookup function on the url produced by your npiaddress for each element of npiList.
A few further comments for future reference should anyone else come upon this question: (1) I don't know why you're doing the as.list, unlist sequence at the beginning; it's redundant and probably unnecessary. (2) The NPI registry provides a programming interface (API) that avoids the need to scrape data from the HTML pages; this might be more robust in the long run. (3) The NPI registry provides the entire dataset as a downloadable file; this might have been an easier way to go.

Dynamic, General Indexing in R

Is there a way to index across many lists, data frames, etc., in R? That is to say, generally? For example, you could retrieve a list of the second element of the second element of lists a & b via c(a[[2]][[2]],b[[2]][[2]]), but how can you do this without writing the names of each list and the respective indexing brackets?
Input:
l1 <- as.list(c(1,2,3,4,5))
l2 <- as.list(c(6,7,8,9,10))
a <- list(l1,l2)
l4 <- as.list(c(1,2,3,4,5))
l5 <- as.list(c(6,7,8,9,10))
b <- list(l4,l5)
Desired output:
[1] 7 7
I know that you could create a list of only upper nested lists - assuming the same naming convention - with this:
nol <- objects()
nol <- grep("^[a-z]$", nol, value=TRUE)
I just don't know how to apply across this list.
You can do this via vapply as follows:
vapply(mget(nol), function(x) x[[2]][[2]], FUN.VALUE = double(1), USE.NAMES = FALSE)
The idea here is that mget gives you a list of your objects. you could also create it by list(a,b). And the anonymous function function(x) x[[2]][[2]] returns your values.

Is it possible, in R, to access the values of a list with a for loop on the names of the fields?

I have a big json file, containing 18 fields, some of which contain some other subfields. I read the file in R in the following way:
json_file <- "daily_profiles_Bnzai_20150914_20150915_20150914.json"
data <- fromJSON(sprintf("[%s]", paste(readLines(json_file), collapse=",")))
This gives me a giant list with all the fields contained in the json file. I want to make it into a data.frame and do some operations in the meantime. For example if I do:
doc_length <- data.frame(t(apply(as.data.frame(data$doc_lenght_map), 1, unlist)))
os <- data.frame(t(apply(as.data.frame(data$operating_system), 1, unlist)))
navigation <- as.data.frame(data$navigation)
monday <- data.frame(t(apply(navigation[,grep("Monday",names(data$navigation))],1,unlist)))
Monday <- data.frame(apply(monday, 1, sum))
works fine, I get what I want, with all the right subfields and then I want to join them in a final data.frame that I will use to do other operations.
Now, I'd like to do something like that on the subset of fields where I don't need to do operations. So, for example, the days of the week contained in navigation are not included. I'd like to have something like (suppose I have a data.frame df):
for(name in names(data))
{
df <- cbind(df, data.frame(t(apply(as.data.frame(data$name), 1, unlist)))
}
The above loop gives me errors. So, what I want to do is finding a way to access all the fields of the list in an automatic way, as in the loop, where the iterator "name" takes on all the fields of the list, without having to call them singularly and then doing some operations with those fields. I tried even with
for(name in names(data))
{
df <- cbind(df, data.frame(t(apply(as.data.frame(data[name]), 1, unlist)))
}
but it doesn't take all of the subfields. I also tried with
data[, name]
but it doesn't work either. So I think I need to use the "$" operator.
Is it possible to do something like that?
Thank you a lot!
Davide
Like the other commenters, I am confused, but I will throw this out to see if it might point you in the right direction.
# make mtcars a list as an example
data <- lapply(mtcars,identity)
do.call(
cbind,
lapply(
names(data),
function(name){
data.frame(data[name])
}
)
)

JSON parsing in a for loop in R

I'm trying to write a for loop that will take zip codes, make an API call to a database of Congressional information and then parse out only the parties of congressmen representing at zip code.
The issue is that some of the zip codes have more than one congressman and others have none at all, (an error on the part of the database, I think). That means I need to loop through the count returned by the original pull until there are no more representatives.
The issue is that the number of congressmen representing each zip code is different. Thus, I'd like to be able to write new variable names into my dataframe for each new congressman. That is, if there are 2 congressmen, I'd like to write new columns named "party.1" and "party.2", etc.
I have this code so far and I feel that I'm close, but I'm really stuck on what to do next. Thank you all for your help!
EDIT: I found this way to be easier, but I'm still not getting the results I'm looking for
library(rjson)
library(RCurl)
zips <- (c("10001","92037","90801", "94011")
test <- matrix(nrow=4,ncol=7)
temp <- NULL
tst <- NULL
for (i in 1:length(zips)) {
for (n in length(temp$count)) {
temp <- (fromJSON(getURL(paste('https://congress.api.sunlightfoundation.com/legislators/locate?zip=',
zips[i],'&apikey= 'INSERT YOUR API KEY', sep=""), .opts = list(ssl.verifypeer = FALSE))))
tst <- try(temp$results[[n]]$party, silent=T)
if(is(tst,"try-error"))
test[i,n] <- NA
else
test[i,n] <- (temp$results[[n]]$party)
}
}
install.packages("rsunlight")
library("rsunlight")
zips <- c("10001","92037","90801", "94011")
out <- lapply(zips, function(z) cg_legislators(zip = z))
# results for some only
sapply(out, "[[", "count")
# peek at results for one zip code
head(out[[1]]$results[,1:4])
bioguide_id birthday chamber contact_form
1 S000148 1950-11-23 senate http://www.schumer.senate.gov/Contact/contact_chuck.cfm
2 N000002 1947-06-13 house https://jerroldnadler.house.gov/forms/writeyourrep/default.aspx
3 M000087 1946-02-19 house https://maloney.house.gov/contact-me/email-me
4 G000555 1966-12-09 senate http://www.gillibrand.senate.gov/contact/
You can change as needed within a lapply or for loop to add columns, etc.
To pull out party could be as simple as lapply(zips, function(z) cg_legislators(zip = z)$results$party).

Parallel programming in R

I have a file that consists of multiple JSON objects. I need to read through these files and extract certain fields from the JSON objects. To complicate things, some of the objects do not contain all the fields. I am dealing with a large file of over 200,000 JSON objects. I would like to split job across multiple cores. I have tried to experiment with doSNOW, foreach, and parallel and really do not understand how to do this. The following is my code that I would like to make more efficient.
foreach (i in 2:length(linn)) %dopar% {
json_data <- fromJSON(linn[i])
if(names(json_data)[1]=="info")
next
mLocation <- ifelse('location' %!in% names(json_data$actor),'NULL',json_data$actor$location$displayName)
mRetweetCount <- ifelse('retweetCount' %!in% names(json_data),0,json_data$retweetCount)
mGeo <- ifelse('geo' %!in% names(json_data),c(-0,-0),json_data$geo$coordinates)
tweet <- rbind(tweet,
data.frame(
record.no = i,
id = json_data$id,
objecttype = json_data$actor$objectType,
postedtime = json_data$actor$postedTime,
location = mLocation,
displayname = json_data$generator$displayName,
link = json_data$generator$link,
body = json_data$body,
retweetcount = mRetweetCount,
geo = mGeo)
)
}
Rather than trying to parallelize an iteration, I think you're better off trying to vectorize (hmm, actually most of the below is still iterating...). For instance here we get all our records (no speed gain yet, though see below...)
json_data <- lapply(linn, fromJSON)
For location we pre-allocate a vector of NAs to represent records for which there is no location, then find records that do have a location (maybe there's a better way of doing this...) and update them
mLocation <- rep(NA, length(json_data))
idx <- sapply(json_data, function(x) "location" %in% names(x$actor))
mLocation[idx] <- sapply(json_data[idx], function(x) x$location$displayName)
Finally, create a 200,000 row data frame in a single call (rather than your 'copy and append' pattern, which makes a copy of the first row, then the first and second row, then the first, second, third row, then ... so N-squared rows, in addition to recreating factors and other data.frame specific expenses; this is likely where you spend most of your time)
data.frame(i=seq_along(json_data), location=mLocation)
The idea would be to accumulate all the columns, and then do just one call to data.frame(). I think you could cheat on parsing line-at-a-time, by pasting everything into a single string repersenting a JSON array, and parsing in one call
json_data <- fromJSON(sprintf("[%s]", paste(linn, collapse=",")))