Get list of all intersections in a city - google-maps

What is the best source and way to get a list of all intersections in a major city?

If one doesn't mind a few false positives the following Overpass API script gets road intersections out of OpenStreetMap data pretty easily:
http://overpass-turbo.eu/s/QD
(the script can't detect false intersections – where only two lines meet, though, e.g. when a road is represented by multiple way objects in the OSM data)
In case that the script goes offline a more readable version directly here:
Dependent on which kind of ways you are interested in, add the types of ways which should not count as intersection to the regv attribute (at two script sections). The type of ways can be found here: highway tags.
The BoundingBox is the part of the map you are viewing on the Overpass-tourbo Website.
Sample Script:
<!-- Only select the type of ways you are interested in -->
<query type="way" into="relevant_ways">
<has-kv k="highway"/>
<has-kv k="highway" modv="not" regv="footway|cycleway|path|service|track"/>
<bbox-query {{bbox}}/>
</query>
<!-- Now find all intersection nodes for each way independently -->
<foreach from="relevant_ways" into="this_way">
<!-- Get all ways which are linked to this way -->
<recurse from="this_way" type="way-node" into="this_ways_nodes"/>
<recurse from="this_ways_nodes" type="node-way" into="linked_ways"/>
<!-- Again, only select the ways you are interested in, see beginning -->
<query type="way" into="linked_ways">
<item set="linked_ways"/>
<has-kv k="highway"/>
<has-kv k="highway" modv="not" regv="footway|cycleway|path|service|track"/>
</query>
<!-- Get all linked ways without the current way -->
<difference into="linked_ways_only">
<item set="linked_ways"/>
<item set="this_way"/>
</difference>
<recurse from="linked_ways_only" type="way-node" into="linked_ways_only_nodes"/>
<!-- Return all intersection nodes -->
<query type="node">
<item set="linked_ways_only_nodes"/>
<item set="this_ways_nodes"/>
</query>
<print/>
</foreach>

You can do it using OpenStreetMap data.
Download the data for the city (use the export link: http://www.openstreetmap.org/export or get the data from here: http://metro.teczno.com/; there are other sources but this isn't the place to list them).
Find all the elements with appropriate values for the 'highway' tag (http://wiki.openstreetmap.org/wiki/Key:highway).
For each such way, get the node IDs that make it up.
Create an array containing entries consisting of the highway information (name, etc.) and a node, one for every node.
Sort the array on node IDs. That groups entries by the node, so that a set of entries with duplicate nodes represents an intersection.
Traverse the array, extracting each group of entries with more than one entry in it and adding a new entry to your list of intersections. At this point you can extract the highway information so that an intersection can be characterised by the highways which meet there.
This is a short summary, I know. But I know it works, because it is the system I use in my map rendering library to identify intersections when creating routing data.

Related

How to identify period end contexts in xbrl filings?

I am trying to find only the current period concepts and facts for the three main financial statements. The goal is to be able to iterate through filings of different companies in different periods.
Using Ebay 2017 10-k as an example.
For concepts that capture YoY change, like those in income statement and statement of cash flows, I can use context found in any of the dei tags, for example:
<dei:DocumentFiscalYearFocus contextRef="FD2017Q4YTD" id="Fact-2E3E1FD4D81352F693510AE035FDC862-wk-Fact-2E3E1FD4D81352F693510AE035FDC862">2017</dei:DocumentFiscalYearFocus>
dei:DocumentFiscalYearFocus tag is required, and its context "FD2017Q4YTD" is also found in all IS and SCF period end concepts, so that's easy.
However, balance sheet concepts use a different context:
<us-gaap:CashAndCashEquivalentsAtCarryingValue contextRef="FI2017Q4" decimals="-6" id="d15135667e874-wk-Fact-3E4A0A2B272B59DE9DAF004097ECF968" unitRef="usd">2120000000</us-gaap:CashAndCashEquivalentsAtCarryingValue>
Any idea how to identify the "FI2017Q4" context (or otherwise find current period balance sheet concepts)?
The XBRL document instance you are viewing contains one or more schemaRef elements, each of which loads an XBRL taxonomy, or data dictionary, for the XBRL instance. Somewhere, within that reference graph of files (and there could be several files), is the definition of each context. The definition will look something like this:
<context id="CONTEXT_ID_NAME">
<!-- ... child elements appear here ... -->
</context>
If you can find the <context> element with an attribute of id that matches the contextRef you're interested in, then you've found what you're looking for. In your case, you're looking in the related XBRL taxonomy files for something that says <context ID="FD2017Q4YTD"> and <context ID="FI2017Q4">.
The child elements of the <context> element describes the dates for the context. There are two types of XBRL contexts:
instant, which specifies a context with a single date
period, which specifies a context with a start date and an end date
The child elements of the <context> element will describe which type of context being described.
This work is all manually doable, but it might be best to use XBRL processing software, which will perform all of this work for you.
The value of the contextRef attribute is purely an identifier that references a context definition elsewhere in the document. Using the eBay example, you'll find this context definition:
<context id="FI2017Q4">
<entity>
<identifier scheme="http://www.sec.gov/CIK">0001065088</identifier>
</entity>
<period>
<instant>2017-12-31</instant>
</period>
</context>
The value in the "instant" element is what tells you what date facts associated with this context relate to.
In order to properly understand the XBRL facts, you need to fully understand the associated contexts. There may be other information, such as additional dimensions, defined here.
I'd strongly recommend working with an existing XBRL processor that will resolve the contextual information for you, such as the open source Arelle processor, or the API provided by XBRL US.
One possible approach to working with XBRL data is to use a processor that converts data to the newer xBRL-JSON format, which provides fact objects with all contextual information fully resolved.

Extract Q&A pairs from the XML stackexchange dumps

I am wanting to extract the question/answer pairs from https://archive.org/download/stackexchange, specifically from the Posts.xml file from any of the dumps (I randomly chose the Anime dump as it was fairly small and close to the top). My understanding of how this file is layed out is that there are two PostTypeId types, 1 being the question (includes the body of the question, title, and other meta data) and 2 being the answer (includes score, the body of the answer, and other meta data).
The data relates easily enough where if we have an entry such as
<row Id="1" PostTypeId="1" AcceptedAnswerId="8" CreationDate="2012-12-11T20:37:08.823" Score="69" ViewCount="22384" Body="<p>Assuming the world in the One Piece universe is round, then there is not really a beginning or an end of the Grand Line.</p>
<p>The Straw Hats started out from the first half and are now sailing across the second half.</p>
<p>Wouldn't it have been quicker to set sail in the opposite direction from where they started? </p>
" OwnerUserId="21" LastEditorUserId="1398" LastEditDate="2015-04-17T19:06:38.957" LastActivityDate="2015-05-26T12:50:40.920" Title="The treasure in One Piece is at the end of the Grand Line. But isn't that the same as the beginning?" Tags="<one-piece>" AnswerCount="5" CommentCount="0" FavoriteCount="2" />
The corresponding answer would be:
<row Id="8" PostTypeId="2" ParentId="1" CreationDate="2012-12-11T20:47:52.167" Score="60" Body="<p>No, there is a reason why they can't. </p>
<p>Basically the <a href="http://onepiece.wikia.com/wiki/New_World">New World</a> is beyond the <a href="http://onepiece.wikia.com/wiki/Red_Line">Red Line</a>, but you cannot "walk" on it, or cross it. It's a huge continent, very tall that you cannot go through. You can't cross the <a href="http://onepiece.wikia.com/wiki/Calm_Belt">Calm Belt</a> either, unless you have some form of locomotion such as the Navy or <a href="http://onepiece.wikia.com/wiki/Boa_Hancock">Boa Hancock</a>.</p>
<p>So the only way is to start from one of the Four Seas, then to go the <a href="http://onepiece.wikia.com/wiki/Reverse_Mountain">Reverse Mountain</a> and follow the Grand Line until you reach <em><a href="http://onepiece.wikia.com/wiki/Raftel">Raftel</a></em>, which supposedly is where One Piece is located.</p>
<p><img src="http://i.stack.imgur.com/69IZ0.png" alt="enter image description here"></p>
" OwnerUserId="15" LastEditorUserId="1528" LastEditDate="2013-05-06T19:21:04.703" LastActivityDate="2013-05-06T19:21:04.703" CommentCount="1" />
Where inside the first xml snippet PostTypeId="1" indicates that this row is a question and AcceptedAnswerId="8" indicates the Id of the answer. And in the second xml snippet we have the Id="8" being the AcceptedAnswerId from the question, PostTypeId="2" indicating that this is an answer, and ParentId being the questions Id.
Now with this being said how could I easily poll this data for the question/answer pairs. Ideally it would be useful if I could convert this to a SQLite3 or Mysql database where I am familiar with these kinds of data structures. If that is not possible (either through the database functions itself or through a scripted wrapper to populate the database) how would I parse this data in Ruby so that I can go through the entire XML document extracting the title and body of the question, then pair it with the appropriate answer body.
Thanks for your time.
The Stack Exchange Creative Commons Data Dump is just a (sanitized) dump from the Stack Exchange production Microsoft SQL Server database. So, considering that the data came from a SQL database and truly is relational data, you can import it back into one.
The database schemata are described in the Data Dump's README, and you can find some old scripts for importing it into a database on Meta Stack Exchange. Of course, if all you want is SQL-like relational query interface, you could just use the Stack Exchange Data Explorer.

Wikipedia Api get amounts of words

I'm a bit stuck in all the options the Wikipedia api has.
My goals is to get the amount of words of an wikipedia page.
I have the url of the wiki.
The search option does return this value:
http://en.wikipedia.org/w/api.php?format=xml&action=query&list=search&srsearch=camera&srlimit=1
Wil return
<api>
<query-continue>
<search sroffset="1"/>
</query-continue>
<query>
<searchinfo totalhits="68658"/>
<search>
<p ns="0" title="Camera" snippet="A <span class='searchmatch'>camera</span> is an optical instrument that records image s that can be stored directly, transmitted to another location, or both. <b>...</b> " size="43246" wordcount="6348" timestamp="2014-04-29T15:48:07Z"/>
</search>
</query>
</api>
(scroll a bit to the right and you find wordcount
But this query is making a search and shows 1 top result. However, when I search on the wikipedia name in the URL, it doesnt always find that record as the first result.
So is there a way to get this wordcount a Wikipedia page?
No other APIs provide this information, so the kludge with list=search is the only way. If you know the exact title you can get better results by appending &srwhat=nearmatch to the query (it will always return 1 result though). See the docs and try the sandbox to learn more.
Note that word counts are not stored in database so the API has to go to Lucene/Elasticsearch for this information which is not exactly fast, so if you need this information en masse you should download a dump instead.

how to find intersections from OpenStreetMap?

How to extract intersections in the OpenStreetMap? I need the longitude and latitude of the intersections, thanks!
There has been a similar question here. There is no direct API call to retrieve intersections. But you can query all ways in a given bounding box (for example directly via the API or via the Overpass API) and look for nodes shared by two or more ways as explained in the other answer.
Try the GeoNames findNearestIntersectionOSM API:
http://api.geonames.org/findNearestIntersectionOSMJSON?lat=37.451&lng=-122.18&username=demo
The input is lng and lat of location and the response contains lng and lat of nearest intersection:
{"intersection":{...,"lng":"-122.1808293","lat":"37.4506505"}}
It is provided by GeoNames, but seems to be based on OpenStreetMap
As #scai perfectly explained, you have to process the raw OSM data yourself to find intersection nodes of ways.
Instead of writing your own program, you can try different algorithms first using the OverpassAPI.
Dependent on which kind of ways you are interested in, add the types of ways which should not count as intersection to the regv attribute (at two script sections). The type of ways can be found here: highway tags.
The BoundingBox is the part of the map you are viewing on the Overpass-tourbo Website.
The script is based on a scripted linked in another question's reply but I rewrote it to make it more readable and added comments (both here and in the original reply).
Sample Script:
<!-- Only select the type of ways you are interested in -->
<query type="way" into="relevant_ways">
<has-kv k="highway"/>
<has-kv k="highway" modv="not" regv="footway|cycleway|path|service|track"/>
<bbox-query {{bbox}}/>
</query>
<!-- Now find all intersection nodes for each way independently -->
<foreach from="relevant_ways" into="this_way">
<!-- Get all ways which are linked to this way -->
<recurse from="this_way" type="way-node" into="this_ways_nodes"/>
<recurse from="this_ways_nodes" type="node-way" into="linked_ways"/>
<!-- Again, only select the ways you are interested in, see beginning -->
<query type="way" into="linked_ways">
<item set="linked_ways"/>
<has-kv k="highway"/>
<has-kv k="highway" modv="not" regv="footway|cycleway|path|service|track"/>
</query>
<!-- Get all linked ways without the current way -->
<difference into="linked_ways_only">
<item set="linked_ways"/>
<item set="this_way"/>
</difference>
<recurse from="linked_ways_only" type="way-node" into="linked_ways_only_nodes"/>
<!-- Return all intersection nodes -->
<query type="node">
<item set="linked_ways_only_nodes"/>
<item set="this_ways_nodes"/>
</query>
<print/>
</foreach>
In case you have imported OSM data to postgis postgresql with osm2pgsql, you can query intersections like this:
SELECT
ST_AsText(st_intersection(a.way, b.way)) AS way
FROM
(select * from planet_osm_line where highway='service') AS a,
(select * from planet_osm_line where highway='footway') AS b
WHERE
ST_Intersects(a.way, b.way)
AND a.osm_id != b.osm_id;

How to display a tree of objects in a JTree?

Imagine a collection of objects such as World, Country, Region and City. World contains a list of Country objects, Country contains a list of Region objects etc.
I would like to represent this structure in a JTree and be able to add, remove and move objects around the tree.
Can I easily create a TableModel from this structure? World would be the root object and I would need to perform some object-specific rendering.
Any one know of an appropriate tutorial that goes beyond building a tree from simple text nodes?
You might start with the examples mentioned in How to Use Trees. You may need profile your code if the intended number of leaf nodes is large. You may want to consider a map as an alternate interface for the uppermost levels. See also the three-part series of articles starting with Creating TreeTables in Swing.
Addendum: Part three in the series cited includes an example TreeTableModelAdapter.
$ ls -1 src/bookmarks/
AbstractTreeTableModel.java
Bookmarks.java
BookmarksModel.java
DynamicTreeTableModel.java
JTreeTable.java
TreeTableExample3.java
TreeTableModel.java
TreeTableModelAdapter.java