Related
I am trying to create a dataframe out of nested JSON files and am running into issues. So far, I've tried using tidyjson to stack the different columns like author, source, etc. However, when I get to "readers" (see the second JSON record), instead of column headers proceeding the ":" there is a unique id. I can't figure out how to get around that and create a dataframe. Here is the code I have so far:
art_df <- artJSON %>%
as.tbl_json %>%
gather_array %>%
enter_object("articles") %>% gather_array %>% # stack the articles
enter_object("author") %>% gather_array %>% # stack the authors
spread_values(articles.author = jstring("author")) %>% # extract item name and price
spread_values(articles.date = jstring("date")) %>%
) %>%
And the JSON file is structured like:
"articles" : {
"-2145189873" : {
"author" : "",
"date" : "",
"partialRecord" : true,
"source" : "reuters",
"text" : "",
"title" : "U.S. government loses to Russia's disinformation campaign: advisers | Reuters",
"url" : "reuters.com/article/us-usa-russia-disinformation-analysis-idUSKBN1492PA"
},
"-2143576379" : {
"author" : [ "", "", "CNN" ],
"date" : "Updated 8:21 AM ET, Mon June 19, 2017",
"readers" : {
"112851530401572877131" : true,
"115171523766335420644" : true
},
"source" : "cnn",
"text" : "Story highlightsThe higher the pressure, the more defiant the President becomesTrump continues to call the Russia probe a 'witch hunt' Washington (CNN)America is edging ever closer to a new long, national nightmare, given President Donald Trump's explosive response to signs that special counsel Robert Mueller is approaching the epicenter of White House power.The investigation has taken a serious turn in recent days, raising the clear possibility that whether Mueller finds wrongdoing or not, a period of political stress and upheaval is inevitable. Trump appears willing to test the bounds of convention and his own powers against a legal and political establishment that he believes is conspiring against him. The higher the pressure, the more defiant the President becomes -- even if it puts him at odds with his legal team.Trump has decided that \"this is a political fight and he's going to fight it,\" a White House official told CNN's Jeff Zeleny.But the sense of chaos around the White House is deepening. The President again called the Russia probe a \"witch hunt\" over the weekend while a legal adviser tried to walk back Trump's admission Friday that he is under investigation.Read More\"The MAKE AMERICA GREAT AGAIN agenda is doing very well despite the distraction of the Witch Hunt,\" Trump tweeted on Sunday.var id = '//platform.twitter.com/widgets.js'.replace(/\\s+/g, '');!!document.getElementById(id) || (function makeEmbedScript(d, id) (document, id));The MAKE AMERICA GREAT AGAIN agenda is doing very well despite the distraction of the Witch Hunt. Many new jobs, high business enthusiasm,..— Donald J. Trump (#realDonaldTrump) June 18, 2017The political stakes rose significantly in recent days amid signs the probe on Russian election hacking could eventually expand to consider Trump's actions.The President and his allies have declared open season on Mueller and Deputy Attorney General Rod Rosenstein, who is overseeing the probe. \"You are witnessing the single greatest WITCH HUNT in American political history - led by some very bad and conflicted people! #MAGA.\" Trump said in a tweet last week that signified an attempt to undermine both men./* global CNN, Modernizr */'use strict';CNN.INJECTOR.executeFeature('video').then(function () {CNN.VideoPlayer.handleUnmutePlayer = function handleUnmutePlayer(containerId, dataObj) {'use strict';var playerInstance,playerPropertyObj,rememberTime,unmuteCTA,unmuteIdSelector = 'unmute_' + containerId,isPlayerMute;dataObj = dataObj || {};if (CNN.VideoPlayer.getLibraryName(containerId) === 'fave') else isPlayerMute = (typeof dataObj.muted === 'boolean') ? dataObj.muted : false;if (CNN.VideoPlayer.playerProperties && CNN.VideoPlayer.playerProperties[containerId]) if (playerPropertyObj.mute && playerPropertyObj.contentPlayed) };CNN.VideoPlayer.showFlashSlate = function showFlashSlate(container) ;CNN.autoPlayVideoExist = (CNN.autoPlayVideoExist === true) ? true : false;var configObj = ,autoStartVideo = false,callbackObj,containerEl,currentVideoCollection = [],currentVideoCollectionId = '',isLivePlayer = false,moveToNextTimeout,mutePlayerEnabled = false,nextVideoId = '',nextVideoUrl = '',turnOnFlashMessaging = false,videoPinner,videoEndSlateImpl;if (CNN.autoPlayVideoExist === false) configObj.autostart = autoStartVideo;CNN.VideoPlayer.setPlayerProperties(configObj.markupId, autoStartVideo, isLivePlayer, mutePlayerEnabled);CNN.VideoPlayer.setFirstVideoInCollection(currentVideoCollection, configObj.markupId);videoEndSlateImpl = new CNN.VideoEndSlate('body-text_15');/*** Finds the next video ID and URL in the current collection, if available.* #param currentVideoId The video that is currently playing* #param containerId The parent container Id of the video element*/function findNextVideo(currentVideoId) findNextVideo('tv/2017/06/18/reliable-sources-journalists-trump-twitter-habit.cnn');/*** Navigates to the video leaf page of the next video in the current collection, if available.* #param currentVideoId The video that is currently playing* #param containerId The parent container Id of the video element*/function navigateToNextVideo(currentVideoId, containerId) callbackObj = ;if (typeof configObj.context !== 'string' || configObj.context.length 0) CNN.autoPlayVideoExist = (CNN.autoPlayVideoExist === true) ? true : false;CNN.VideoPlayer.getLibrary(configObj, callbackObj, isLivePlayer);});/* videodemanddust is a default feature of the injector */CNN.INJECTOR.scriptComplete('videodemanddust');JUST WATCHEDWhy journalists like Trump's Twitter habitReplayMore Videos ...MUST WATCHWhy journalists like Trump's Twitter habit 05:59The tweet sparked speculation that Trump may pull a nuclear option by firing Mueller and Rosenstein, a move senior aides have fervently advised against -- given that it could provoke a constitutional crisis.\"It looks like we have got a simmering tea kettle here, and the President seems to be testing the waters about whether he ought to go ahead and fire Robert Mueller,\" said Richard Ben-Veniste, a former Watergate prosecutor who is now a CNN legal analyst.\"You don't mess with the criminal justice system in this way ... the President is on very tenuous ground here,\" Ben-Veniste told CNN's Don Lemon Friday.Cuba to Trump: US in no 'condition to lecture us' on human rightsThe scorched-earth approach may have grave political consequences.Trump and his aides may well end up in the clear, but months of recriminations ahead will further polarize the capital and cast a shadow over his administration, which is already struggling for traction after a tumultuous first five months.If Mueller decides there is a case to answer for either the President or his aides in the issue of alleged collusion with Russia or over a potential cover-up, the political uproar will be incalculable.At best, it could further complicate Trump's efforts to pass a meaningful legislative agenda and ultimately his hopes of a successful presidency.At worst, it could force the nation to confront a third debate on whether to remove an elected President in just 45 years. The lessons of history suggest such traumas raise questions of political legitimacy and constitutional process that have the potential to sour life in Washington for years in the future./* global CNN, Modernizr */'use strict';CNN.INJECTOR.executeFeature('video').then(function () {CNN.VideoPlayer.handleUnmutePlayer = function handleUnmutePlayer(containerId, dataObj) {'use strict';var playerInstance,playerPropertyObj,rememberTime,unmuteCTA,unmuteIdSelector = 'unmute_' + containerId,isPlayerMute;dataObj = dataObj || {};if (CNN.VideoPlayer.getLibraryName(containerId) === 'fave') else isPlayerMute = (typeof dataObj.muted === 'boolean') ? dataObj.muted : false;if (CNN.VideoPlayer.playerProperties && CNN.VideoPlayer.playerProperties[containerId]) if (playerPropertyObj.mute && playerPropertyObj.contentPlayed) };CNN.VideoPlayer.showFlashSlate = function showFlashSlate(container) ;CNN.autoPlayVideoExist = (CNN.autoPlayVideoExist === true) ? true : false;var configObj = ,autoStartVideo = false,callbackObj,containerEl,currentVideoCollection = [],currentVideoCollectionId = '',isLivePlayer = false,moveToNextTimeout,mutePlayerEnabled = false,nextVideoId = '',nextVideoUrl = '',turnOnFlashMessaging = false,videoPinner,videoEndSlateImpl;if (CNN.autoPlayVideoExist === false) configObj.autostart = autoStartVideo;CNN.VideoPlayer.setPlayerProperties(configObj.markupId, autoStartVideo, isLivePlayer, mutePlayerEnabled);CNN.VideoPlayer.setFirstVideoInCollection(currentVideoCollection, configObj.markupId);videoEndSlateImpl = new CNN.VideoEndSlate('body-text_25');/*** Finds the next video ID and URL in the current collection, if available.* #param currentVideoId The video that is currently playing* #param containerId The parent container Id of the video element*/function findNextVideo(currentVideoId) findNextVideo('politics/2017/06/18/jay-sekulow-trump-obstruction-of-justice-sotu.cnn');/*** Navigates to the video leaf page of the next video in the current collection, if available.* #param currentVideoId The video that is currently playing* #param containerId The parent container Id of the video element*/function navigateToNextVideo(currentVideoId, containerId) callbackObj = ;if (typeof configObj.context !== 'string' || configObj.context.length 0) CNN.autoPlayVideoExist = (CNN.autoPlayVideoExist === true) ? true : false;CNN.VideoPlayer.getLibrary(configObj, callbackObj, isLivePlayer);});/* videodemanddust is a default feature of the injector */CNN.INJECTOR.scriptComplete('videodemanddust');JUST WATCHEDSekulow: POTUS has power to fire FBI directorReplayMore Videos ...MUST WATCHSekulow: POTUS has power to fire FBI director 00:10Trump's in 'a political fight and he's going to fight it'Back in the present, the President's mood is fueling the sense of crisis.\"One thing we've learned I think from the testimony of multiple people now is the President's pretty fired up about this, OK?\" Republican Sen. Marco Rubio said on NBC's \"Meet the Press\" Sunday.\"He, from every pronouncement we have seen, feels very strongly that he did nothing wrong, and he wants people to say that, because he feels very strongly about it,\" Rubio said. \"I don't think that's a mystery. And he's expressing himself in that way.\"But the more frustrated that Trump gets, the more he often hurts himself politically. A case in point is the firing of Comey and Trump's subsequent admission that he had Russia on his mind when he dismissed the FBI director, an admission that could provide a rationale for an obstruction of justice probe.\"Trump has a compulsion to counter-attack and is very pugnacious. I don't think it serves him well,\" former House Speaker Newt Gingrich, a Trump supporter, on ABC's \"This Week,\" adding that the President was right to be infuriated by \"this whole Russia baloney.\"/* global CNN, Modernizr */'use strict';CNN.INJECTOR.executeFeature('video').then(function () {CNN.VideoPlayer.handleUnmutePlayer = function handleUnmutePlayer(containerId, dataObj) {'use strict';var playerInstance,playerPropertyObj,rememberTime,unmuteCTA,unmuteIdSelector = 'unmute_' + containerId,isPlayerMute;dataObj = dataObj || {};if (CNN.VideoPlayer.getLibraryName(containerId) === 'fave') else isPlayerMute = (typeof dataObj.muted === 'boolean') ? dataObj.muted : false;if (CNN.VideoPlayer.playerProperties && CNN.VideoPlayer.playerProperties[containerId]) if (playerPropertyObj.mute && playerPropertyObj.contentPlayed) };CNN.VideoPlayer.showFlashSlate = function showFlashSlate(container) ;CNN.autoPlayVideoExist = (CNN.autoPlayVideoExist === true) ? true : false;var configObj = ,autoStartVideo = false,callbackObj,containerEl,currentVideoCollection = [],currentVideoCollectionId = '',isLivePlayer = false,moveToNextTimeout,mutePlayerEnabled = false,nextVideoId = '',nextVideoUrl = '',turnOnFlashMessaging = false,videoPinner,videoEndSlateImpl;if (CNN.autoPlayVideoExist === false) configObj.autostart = autoStartVideo;CNN.VideoPlayer.setPlayerProperties(configObj.markupId, autoStartVideo, isLivePlayer, mutePlayerEnabled);CNN.VideoPlayer.setFirstVideoInCollection(currentVideoCollection, configObj.markupId);videoEndSlateImpl = new CNN.VideoEndSlate('body-text_34');/*** Finds the next video ID and URL in the current collection, if available.* #param currentVideoId The video that is currently playing* #param containerId The parent container Id of the video element*/function findNextVideo(currentVideoId) findNextVideo('politics/2017/06/18/sotu-rubio-trump-talked-russia-probe-in-private.cnn');/*** Navigates to the video leaf page of the next video in the current collection, if available.* #param currentVideoId The video that is currently playing* #param containerId The parent container Id of the video element*/function navigateToNextVideo(currentVideoId, containerId) callbackObj = ;if (typeof configObj.context !== 'string' || configObj.context.length 0) CNN.autoPlayVideoExist = (CNN.autoPlayVideoExist === true) ? true : false;CNN.VideoPlayer.getLibrary(configObj, callbackObj, isLivePlayer);});/* videodemanddust is a default feature of the injector */CNN.INJECTOR.scriptComplete('videodemanddust');JUST WATCHEDRubio: Trump talked Russia probe in privateReplayMore Videos ...MUST WATCHRubio: Trump talked Russia probe in private 01:05Legal team contradicts the clientIn another sign of the gravity of the situation, Trump's legal team blitzed Sunday talk shows to contradict the President's tweet Friday where he said he is already under investigation.\"The President is not a subject or target of an investigation. That tweet was in response to a Washington Post story that ran with five unnamed sources, without identifying the agencies they represented, saying that the special counsel had broadened out his investigation to include the president,\" said Jay Sekulow, a Trump legal adviser on CNN's \"State of the Union.\"Sekulow also opened a conflict between the President's political team and his legal one, hinting that the President's tweets should be discounted even though the White House said they are an authentic expression of his views.\"It was 141 characters. There's a limitation on Twitter, as we all know,\" Sekulow said.Then, in a subsequent interview on \"Fox News Sunday,\" Sekulow sowed further confusion by saying Trump was under investigation -- then denied he said anything of the sort.Special counsel moving forwardLaw enforcement sources have told CNN that the special counsel is gathering information and considering whether there is evidence to launch a full scale obstruction investigation. Mueller's investigators have asked for information and will talk to Director of National Intelligence Dan Coats and National Security Agency Director Adm. Mike Rogers, according to a source familiar with the matter. Investigators have also sought information from recently retired NSA Deputy Richard Ledgett, according to the source. Ledgett wrote a memo, according to the source, documenting a conversation in which the president allegedly urged Rogers to help get the FBI to lift the cloud of the Russia investigation.Any confirmation that Mueller has expanded the investigation to consider whether the President obstructed justice with the Comey firing would carry no guarantee of a guilty finding.But it would deal a political blow to the White House in the battle of perception, since the idea that Trump is not under investigation has been at the centerpiece of his administration's defense strategy./* global CNN, Modernizr */'use strict';CNN.INJECTOR.executeFeature('video').then(function () {CNN.VideoPlayer.handleUnmutePlayer = function handleUnmutePlayer(containerId, dataObj) {'use strict';var playerInstance,playerPropertyObj,rememberTime,unmuteCTA,unmuteIdSelector = 'unmute_' + containerId,isPlayerMute;dataObj = dataObj || {};if (CNN.VideoPlayer.getLibraryName(containerId) === 'fave') else isPlayerMute = (typeof dataObj.muted === 'boolean') ? dataObj.muted : false;if (CNN.VideoPlayer.playerProperties && CNN.VideoPlayer.playerProperties[containerId]) if (playerPropertyObj.mute && playerPropertyObj.contentPlayed) };CNN.VideoPlayer.showFlashSlate = function showFlashSlate(container) ;CNN.autoPlayVideoExist = (CNN.autoPlayVideoExist === true) ? true : false;var configObj = ,autoStartVideo = false,callbackObj,containerEl,currentVideoCollection = [],currentVideoCollectionId = '',isLivePlayer = false,moveToNextTimeout,mutePlayerEnabled = false,nextVideoId = '',nextVideoUrl = '',turnOnFlashMessaging = false,videoPinner,videoEndSlateImpl;if (CNN.autoPlayVideoExist === false) configObj.autostart = autoStartVideo;CNN.VideoPlayer.setPlayerProperties(configObj.markupId, autoStartVideo, isLivePlayer, mutePlayerEnabled);CNN.VideoPlayer.setFirstVideoInCollection(currentVideoCollection, configObj.markupId);videoEndSlateImpl = new CNN.VideoEndSlate('body-text_48');/*** Finds the next video ID and URL in the current collection, if available.* #param currentVideoId The video that is currently playing* #param containerId The parent container Id of the video element*/function findNextVideo(currentVideoId) findNextVideo('tv/2017/06/18/inside-politics-6-18-notebooks.cnn');/*** Navigates to the video leaf page of the next video in the current collection, if available.* #param currentVideoId The video that is currently playing* #param containerId The parent container Id of the video element*/function navigateToNextVideo(currentVideoId, containerId) callbackObj = ;if (typeof configObj.context !== 'string' || configObj.context.length 0) CNN.autoPlayVideoExist = (CNN.autoPlayVideoExist === true) ? true : false;CNN.VideoPlayer.getLibrary(configObj, callbackObj, isLivePlayer);});/* videodemanddust is a default feature of the injector */CNN.INJECTOR.scriptComplete('videodemanddust');JUST WATCHEDTrump heads west to POTUS-friendly IowaReplayMore Videos ...MUST WATCHTrump heads west to POTUS-friendly Iowa 04:17",
"title" : "Trump defiant as pressure grows",
"url" : "cnn.com/2017/06/19/politics/trump-investigation/index.html"
},
Any help would be greatly appreciated. I've been stuck trying to read this file in for some time and cant figure it out. Thanks!!!
see https://rdrr.io/cran/jsonlite/man/flatten.html
data2 <- fromJSON("so.json", flatten = TRUE)
I am trying to parse the plain text from multiple scientific articles for subsequent text analysis. So far I use a R script by Tony Breyal based on the packages RCurl and XML. This works fine for all targeted journals, except for those published by http://www.sciencedirect.com. When I try to parse the articles from SD (and this is consistent for all tested journals I need to access from SD), the text object in R just stores the first part of the whole document in it. Unfortunately, I am not too familiar with html, but I think the problem should be in the SD html code, since it works in all other cases.
I am aware that some journals are not open accessible, but I have access authorisations and the problems also occur in open access articles (check the example).
This is the code from Github:
htmlToText <- function(input, ...) {
###---PACKAGES ---###
require(RCurl)
require(XML)
###--- LOCAL FUNCTIONS ---###
# Determine how to grab html for a single input element
evaluate_input <- function(input) {
# if input is a .html file
if(file.exists(input)) {
char.vec <- readLines(input, warn = FALSE)
return(paste(char.vec, collapse = ""))
}
# if input is html text
if(grepl("</html>", input, fixed = TRUE)) return(input)
# if input is a URL, probably should use a regex here instead?
if(!grepl(" ", input)) {
# downolad SSL certificate in case of https problem
if(!file.exists("cacert.perm")) download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.perm")
return(getURL(input, followlocation = TRUE, cainfo = "cacert.perm"))
}
# return NULL if none of the conditions above apply
return(NULL)
}
# convert HTML to plain text
convert_html_to_text <- function(html) {
doc <- htmlParse(html, asText = TRUE)
text <- xpathSApply(doc, "//text()[not(ancestor::script)][not(ancestor::style)][not(ancestor::noscript)][not(ancestor::form)]", xmlValue)
return(text)
}
# format text vector into one character string
collapse_text <- function(txt) {
return(paste(txt, collapse = " "))
}
###--- MAIN ---###
# STEP 1: Evaluate input
html.list <- lapply(input, evaluate_input)
# STEP 2: Extract text from HTML
text.list <- lapply(html.list, convert_html_to_text)
# STEP 3: Return text
text.vector <- sapply(text.list, collapse_text)
return(text.vector)
}
This is now my code and an example article:
target <- "http://www.sciencedirect.com/science/article/pii/S1754504816300319"
temp.text <- htmlToText(target)
The unformatted text stops somewhere in the Method section:
DNA was extracted using the MasterPure™ Yeast DNA Purification Kit
(Epicentre, Madison, Wisconsin, USA) following the manufacturer's
instructions.
Any suggestions/ideas?
P.S. I also tried html_text based on rvest with the same outcome.
You can prbly use your existing code and just add ?np=y to the end of the URL, but this is a bit more compact:
library(rvest)
library(stringi)
target <- "http://www.sciencedirect.com/science/article/pii/S1754504816300319?np=y"
pg <- read_html(target)
pg %>%
html_nodes(xpath=".//div[#id='centerContent']//child::node()/text()[not(ancestor::script)][not(ancestor::style)][not(ancestor::noscript)][not(ancestor::form)]") %>%
stri_trim() %>%
paste0(collapse=" ") %>%
write(file="output.txt")
A bit of the output (total for that article was >80K):
Fungal Ecology Volume 22 , August 2016, Pages 61–72 175394|| Species richness
influences wine ecosystem function through a dominant species Primrose J. Boynton a , , ,
Duncan Greig a , b a Max Planck Institute for Evolutionary Biology, Plön, 24306, Germany
b The Galton Laboratory, Department of Genetics, Evolution, and Environment, University
College London, London, WC1E 6BT, UK Received 9 November 2015, Revised 27 March 2016,
Accepted 15 April 2016, Available online 1 June 2016 Corresponding editor: Marie Louise
Davey Abstract Increased species richness does not always cause increased ecosystem function.
Instead, richness can influence individual species with positive or negative ecosystem effects.
We investigated richness and function in fermenting wine, and found that richness indirectly
affects ecosystem function by altering the ecological dominance of Saccharomyces cerevisiae .
While S. cerevisiae generally dominates fermentations, it cannot dominate extremely species-rich
communities, probably because antagonistic species prevent it from growing. It is also diluted
from species-poor communities,
I am working on a JSON code. EG:
{"resultSet":{"arrival":[{"feet":null,"departed":false,"scheduled":1460553672000,"
shortSign":"34 To Oregon City TC","detoured":false,"tripID":"6317022","dir":0,
"blockID":3402,"route":34,"piece":"1","fullSign":"34 River Rd to Oregon City TC",
"id":"6317022_22872_13","vehicleID":null,"locid":10,"newTrip":false,
"status":"scheduled"}],"queryTime":1460522014873,
"location":[{"lng":-122.603190086107,"passengerCode":"E","id":10,
"dir":"Southbound","lat":45.3847203071942,"desc":"Abernethy & Barclay"}]}}
and I need to convert it into a HTML Table view.
But the JSON data may not always appear the same structure. It can be different like this.
{"resultSet":{"arrival":[{"feet":50907,"inCongestion":false,"departed":false,
"scheduled":1460524334000,"loadPercentage":0,"shortSign":"78 To Beaverton TC",
"estimated":1460524334000,"detoured":false,"tripID":"6322386","dir":1,
"blockID":7672,"route":78,"piece":"1","fullSign":"78 Beaverton TC",
"id":"6322386_79934_12","vehicleID":"2918","locid":1233,"newTrip":false,
"status":"estimated"},{"feet":null,"departed":false,"scheduled":1460528004000,
"shortSign":"78 To Beaverton TC","detoured":false,"tripID":"6322387","dir":1,
"blockID":7867,"route":78,"piece":"1","fullSign":"78 Beaverton TC",
"id":"6322387_83604_12","vehicleID":null,"locid":1233,"newTrip":false,
"status":"scheduled"},{"feet":null,"departed":false,"scheduled":1460557107000,
"shortSign":"37 To Tualatin P&R","detoured":false,"tripID":"6317319","dir":0,
"blockID":3767,"route":37,"piece":"1",
"fullSign":"37 Lake Grove to Tualatin Park & Ride","id":"6317319_26307_13",
"vehicleID":null,"locid":1233,"newTrip":false,"status":"scheduled"}],
"queryTime":1460522323801,"location":[{"lng":-122.697651868185,
"passengerCode":"E","id":1233,"dir":"Westbound","lat":45.423844983607,
"desc":"Country Club & Wembley Park"}]}}
How would I be able to achieve the result in HTML Table view?
I am afraid this is anything BUT elegant but I did get the solution in the end. ;-)
Maybe you can use some ideas ... (like collecting the attributes with function getkeys() ...)
// first test case:
var dat={"resultSet":{"arrival":[{"feet":50907,"inCongestion":false,"departed":false,
"scheduled":1460524334000,"loadPercentage":0,"shortSign":"78 To Beaverton TC",
"estimated":1460524334000,"detoured":false,"tripID":"6322386","dir":1,
"blockID":7672,"route":78,"piece":"1","fullSign":"78 Beaverton TC",
"id":"6322386_79934_12","vehicleID":"2918","locid":1233,"newTrip":false,
"status":"estimated"},{"feet":null,"departed":false,"scheduled":1460528004000,
"shortSign":"78 To Beaverton TC","detoured":false,"tripID":"6322387","dir":1,
"blockID":7867,"route":78,"piece":"1","fullSign":"78 Beaverton TC",
"id":"6322387_83604_12","vehicleID":null,"locid":1233,"newTrip":false,
"status":"scheduled"},{"feet":null,"departed":false,"scheduled":1460557107000,
"shortSign":"37 To Tualatin P&R","detoured":false,"tripID":"6317319","dir":0,
"blockID":3767,"route":37,"piece":"1",
"fullSign":"37 Lake Grove to Tualatin Park & Ride","id":"6317319_26307_13",
"vehicleID":null,"locid":1233,"newTrip":false,"status":"scheduled"}],
"queryTime":1460522323801,"location":[{"lng":-122.697651868185,
"passengerCode":"E","id":1233,"dir":"Westbound","lat":45.423844983607,
"desc":"Country Club & Wembley Park"}]}};
// second test case with missing "arrival" sub-array:
var dat ={"resultSet":{"queryTime":1460959167994,"location":[{"lng":-122.79683594921,
"passengerCode":"E","id":997,"dir":"Southbound",
"lat":45.506252040801,"desc":"SW Cedar Hills & Berkshire"}]}};
var maxr=0;
function getkeys(arr) { var keys={};
// the following "if ... else" was added to deal with missing sub-arrays:
if (arr) {arr.forEach(function(o,i){for (a in o) keys[a]=1;});
var ret=Object.keys(keys); if ((ret.len=arr.length)>maxr) maxr=ret.len;
} else ret=[];
// ret.len is the number of rows found in the array arr
// and should not be confused with ret.length (=the length of ret)
return ret;
}
var rs=dat.resultSet,
okeys={rs:getkeys([rs]),arrival:getkeys(rs.arrival),location:getkeys(rs.location)};
// first title row
var r=[okeys.rs.map(function(rk){
return '<th'+(okeys[rk]?' colspan="'+okeys[rk].length+'"':'')+'>'+rk+'</th>';}).join('')],
tmp=[];
// second title row
okeys.rs.forEach(function(rk){
if (okeys[rk]) [].push.apply(tmp,okeys[rk]); // push a group of columns
else tmp.push(rk); // push a single column into array tmp
});
r.push('<td>'+tmp.join('</td><td>')+'</td>');
// data rows
for (i=0;i<maxr;i++){ tmp=[];
okeys.rs.forEach(function(rk){var ok=okeys[rk],l,td;
if (ok) { l=ok.len; td='<td'+(maxr>l && l==i+1?' rowspan="'+(maxr-i)+'">':'>');
if (i<l) [].push.apply(tmp,ok.map(function(k){return td+(rs[rk][i][k]||'')+'</td>';}));
}
else if (!i) tmp.push('<td rowspan="'+maxr+'">'+rs[rk]+'</td>'); // push a single column into array tmp
});
r.push(tmp.join(''));
}
// output onto page
document.getElementById("out").innerHTML=
'<table><tr>'+r.join('</tr>\n<tr>')+'</tr></table>';
th,td {border: solid 1px}
<div id="out"></div>
Edit:
I just added an if ... else block in the getkeys() function to deal with cases when not all of the sub-arrays, as defined in the okeys-object, are present in dat. In the latest test case the sub-array "arrival" was missing.
I am adapting an existing D3JS map-drawing file to work with a different JSON file. In the file 'ny.json' there are things called "objects" and inside of those there are things called "tracts" so I get how the references work for the current (working) code, which comes from here and looks like:
d3.json("./shapefiles/ny.json", function(error, ny) {
if (error) throw error;
var tracts = ny.objects.tracts;
tracts.geometries = tracts.geometries
.filter(function(d) { return (d.id / 10000 | 0) !== 99; });
svg.append("g")
.selectAll("path")
.data(topojson.feature(ny, tracts).features)
.enter().append("path")
.attr("class", "tract")
.attr("d", path)
However, the new JSON file doesn't have those 'objects' or 'tracts' and I'm trying to figure out how to filter it so that it can be used to draw a map not of New York State (as in the first one) but just of the Bronx county. It's possible that the two files are too different in structure that an easy filter change isn't possible, but I thought I would ask in case there's another solution. I haven't seen the 'meta' features in json before.
I am afraid it is not possible to display only Bronx in the map because census tract ids in ny.json are unique only at borough level. For instance, census tract 014300 appears in Bronx, Brooklyn, Manhattan, and Queens. ny.json stores only census tract ids in ny.objects.tracts.id, and not their FIPS County Codes.
This makes filtering on Borough name impossible: https://bl.ocks.org/memoryfull/ba6121d19f80f85f0beb
Even if the filtering was feasible, it would be suboptimal. Instead, I would create own GeoJSON of Bronx in any GIS software with census tract ids and then call it in your d3 code.
There seems to be some difficulty about getting individual address components out of a JSON Google geocoder result. For example, I want to get county names based on addresses. Could someone please explain why, for example, this code doesn't work?
function county(){
var geo = Maps.newGeocoder().geocode('address');
for (var i=0; i<geo.results.address_components.length; i++)
for (var j=0; j<geo.results.address_components[i].types.length; j++)
if (geo.results.address_components[i].types[j]=="administrative_area_level_2")
var county = geo.results.address_components[i].long_name;
}
I know this is overly simplistic, but it can't even recognize the results.address_components
Any suggestions?
You can use the Logger object to look at intermediary data like this:
Logger.log(geo)
You'll see something like this in the view menu, under logs:
{results=[{address_components=[{long_name=Address, short_name=Address, types=[point_of_interest, establishment]}, {long_name=Madrid, short_name=Madrid, types=[locality, political]}, {long_name=Madrid, short_name=M, types=[administrative_area_level_2, political]}, {long_name=Community of Madrid, short_name=Community of Madrid, types=[administrative_area_level_1, political]}, {long_name=Spain, short_name=ES, types=[country, political]}, {long_name=28039, short_name=28039, types=[postal_code]}], formatted_address=Address, 28039 Madrid, Spain, types=[hospital, point_of_interest, establishment], geometry={viewport={southwest={lng=-3.708366880291502, lat=40.44598801970851}, northeast={lng=-3.705668919708498, lat=40.44868598029151}}, location_type=APPROXIMATE, location={lng=-3.7070179, lat=40.447337}}}, {address_components=[{long_name=Address:, short_name=Address:, types=[point_of_interest, establishment]}, {long_name=Pineville, short_name=Pineville, types=[locality, political]}, {long_name=1, Charlotte, short_name=1, Charlotte, types=[administrative_area_level_3, political]}, {long_name=Mecklenburg County, .........
If you run
function myFunction() {
var geo = Maps.newGeocoder().geocode('1600 pennsylvania ave, washington, dc');
Logger.log(geo.results[0].address_components);
}
myFunction();
You can see the components in the log, it will look like this:
{long_name=1600, short_name=1600, types=[street_number]}, {long_name=President's Park, short_name=President's Park, types=[establishment]}, {long_name=Pennsylvania Avenue Northwest, short_name=Pennsylvania Ave NW, types=[route]}, {long_name=Washington, short_name=Washington, types=[locality, political]}, {long_name=District of Columbia, short_name=DC, types=[administrative_area_level_1, political]}, {long_name=United States, short_name=US, types=[country, political]}, {long_name=20500, short_name=20500, types=[postal_code]}
In this case, none of components are "administrative_area_level_2", but in your case they might be.
You should also consider using Array.forEach.
Hopefully this will help you nail down what the more specific problem is.
To add on to Aaron and Micah's answers, you're really close, just mostly need to select the first result. geocode produces an array of results, but if you're geocoding a single address, grabbing the first will work. Their answer uses the White House address, which appears not to have a county, but using Google HQ (as Google tends to do in examples) produces a valid result.
Here's a generic function for selecting any component of the geocoding responses, also leveraging indexOf to check if the desired component is in the types array (saving a for loop):
function getAddressComponent(address, component) {
var geo = Maps.newGeocoder().geocode(address);
var resultComponents = geo.results[0].address_components;
for (var i = 0; i < resultComponents.length; i++) {
if (resultComponents[i].types.indexOf(component) > -1) {
return resultComponents[i].long_name;
}
}
}
Testing with Logger, we get the correct value of 'Santa Clara County':
function getAddressComponent_test() {
var address = '1600 Amphitheatre Parkway, Mountain View, CA';
Logger.log(getAddressComponent(address, 'administrative_area_level_2'));
}