As the question title states, I need the crime data from all the united states. I can't find a dataset for that, only numerous small ones for the different cities and countrysides.
Is there such a united API or should I maintain these small ones as well ?
There is currently no single open dataset or API (Socrata maintained or otherwise) that covers the US completely. Many cities publish crime reports to their open data portals, but the coverage is still pretty sparse.
There's also the FBI Uniform Crime Reporting datasets, but those are aggregated at the city level (which again is somewhat sparse) and the most recent data is a partial update from the first half of 2015.
Background: I am building a website which crawls data from real-estate websites (a sample project) in hong kong. I crawl different websites, for various property listings. Hence I get a variety of locations formats from the raw crawled locations
Example:
few Samples of raw crawled locations:
North Point
8, Java Road, North Point, Hong Kong
Hong Kong Island , North Point
As can be seen from the above addresses, the same district in hong kong, that is North point can have many forms, few have been listed above. Issue is that this data doesnt make sense unless it is mapped/structured
Requirement:
I want some elegant solution, to be able to map these raw crawled locations to a pre-defined set of values. One way I can think of is, create a database of locations (I am using mysql), and find out the matching locationId for each crawled raw location.
If the above scheme sounds good, then my database would have fields like the below:
LOCATION_REF, country, state,city, locality, subLocality1, subLocality2, lat, long
Additional information: I am also using apache solr (lucene) to built this website, so any solr related matching suggestions are also welcome
Thoughts: I think there must be some predefined APIs to do things like this, because almost every app I see have location based searching and filtering. Is there a google api to do this? If yes, I would appreciate if anyone could help with good examples
PS: I use Jsoup to parse data from the pages (to extract raw location)
I have found out a way to do it, and Google geocoding anwers all the questions that I asked.
Google geocoding API maps the raw address format to its own format, gives structured data of the location based on the location hirarchy (example street number, neighbourhood, locality etc etc . This is what I needed
Here is the link for reference: https://developers.google.com/maps/documentation/geocoding/intro
I'm trying to implement maps service in my project. The project is focused to russian-speaking countries. But in future I hope to expand audience. The problem is that different services show different names.
OSM shows local names- for example Chinese towns are not clear for me. Google maps shows - user sees all names in the world in his native language: if a'm russian I can see all names in russian lunguage, including names of streets in USA for example. But Google maps costs a lot for me now.
How to solve the problem and what free/cheap services did you use? Сartography isn't the main feature in the project but important.
This isn't a full answer, but may be of some help. There are two ways of approaching the problem using OpenStreetMap data. 1. Russian names are sometimes provided: the tag is name:ru. However, there are very few of them. They're mostly for countries and major cities. For example, London has name:ru=Лондон. 2. But if you want everything which doesn't have a name:ru to appear in the Cyrillic alphabet, you'll have to transliterate from the most convenient source: the local name if it's in some easily transliterable script like Roman or Greek; or the English name (say) if not.
So, taking the second approach, the way I would do it would be to download the OSM data and pre-process it, inserting a transliterated name:ru where one did not exist, then render the map (using, for example, Mapnik, which is free), using a style sheet modified to use name:ru, where available, for all labels.
This, however, is an off-line solution. I realise that you probably want something on-line. Perhaps you could use Mapbox with on-line vector tiles, again pre-processed in some way before they are rendered.
I use the downloadable Geonames database for all countries, states, counties and cities in the world, but I wonder if there is a database for streets?
So you could pick: country -> state/department/district -> (county/region) -> city -> street in whatever country you like.
Because when I do a search for a street in Google Maps I can see all the upper levels (country/region/city). Is there a database you could download for MySQL for this?
There has to be a database for this, cause there are always national companies that provide this kind of information. Where did they get it from?
Have you checked out OpenStreetMap? It is free and has a export screen that lets you download the data in XML and other formats.
Those concerned with the accruacy of the OpenStreetMap data may want to read what ITO has to say about OpenStreetMap's data for Haiti.
Enter OpenStreetMap, the wiki map of the world, CrisisMappers and an impromptu community of volunteers who collaborated to produce the most authoritative map of Haiti in existence.
If you have 5 minutes you may want to watch this TED video by Tim Berners-Lee talking about what is being done with OpenStreetMap data.
You're looking for Navteq or Tele Atlas.
They will charge lots of money.
The US Census Bureau provides what they call TIGER files, which might help you out for the US anyway... As far as freely available worldwide data I seriously doubt you're gonna find much, but many governments probably provide something similar to the TIGER files. Good luck.
I think one good solution would be Listreet
Samples seems to contain country, region, city, streets and even elevation with longitude and latitude
Prices are fair and the team behind is responsive
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
There are examples online with web select boxes that have a huge list of countries and that probably will be good enough for me to use. However, by Murphy's law, there's bound to be some random country that someone is from and isn't on my list (and probably someone else also ran into this and has updated their local list). Also, when new countries are added, I won't know about it.
Basically, I feel it's better practice and a better smell if there is some centralized list of country names that I can use / trust. (also it could set/follow standards for exact namings "United St..." vs "USA" etc.)
I would prefer a solution that isn't IIS specific if possible
There are many list of countries, check this wikipedia article; there you can find some lists like:
ISO 3166-1 countries codes
IOC country codes
Alternative country names
And more...
We maintain a list of 'PUBLIC DOMAIN' Worldwide Country names in all official formats. The information comes from the ISO 3166-1 Maintenance Agency for offical English and French short names, the US Board on Geographic Names (BGN) for english short, long and local short, long names, and United Nations Group of Experts on Geographic Names (UNGEGN) format (long) English, French names, short and formal local names and Spanish names.
There still is a problem representing some Arabic characters as romanized characters (will see ?). But these are limited to the local names of a few Arab countries.
Note, the English, Spanish and French cover the 3 Western languages of the UN's official 6 languages. Metadata, information on the sources, and download can be found at:
http://www.opengeocode.org/download.php#countrynames
The Open Geocode Team
OpenGeoCode.org
Jan. 26, 2014.
We updated the list to include country names in Italian and German. We used the UN Food & Agriculture list of countries in Italian and the German Government's Federal Foreign Office's list of countries in German.
I recommend pulling data out of the Unicode CLDR (Common Locale Data Repository), which include a professionally-maintained list of countries and country name data.
Grab the data from there once, and do updates once in a while; the CLDR data will come in a consistent format, so you won't need to fuss over it once it's part of your workflow.
An answer to this question contains a useful link to a Github project that has lists in various formats and the script that produced them, making it easy to obtain updated versions.
No list is comprehensive.
"there's bound to be some random country that someone is from and isn't on my list"
If that was all there was to it, it would be simple.
There's no "world law" or "world constitution", so there's no single list of countries, republics, territories, protectorates, autonomous regions, independent governments, and disputed territories.
Indeed, it's not possible to come to an agreed-upon definition of "country" which would lead to a final list of country codes. The definition of "country" is politically charged. What, for example, is Tibet? Country or region of China? Northern Ireland? The Holy See?
Pick a list, and know that it's subject to some dispute.
You could, for example, use the IANA country code database: http://www.iana.org/domains/root/db/
It's as good as any, and since it's part of the IANA, it has some standing as a standard. Further, it's pretty accessible as easy-to-parse web content.
I have a recent list ready to go on my website at http://www.john.geek.nz/index.php/2009/01/sql-tips-list-of-countries/
It's both at sql and tab delimited - The original list was sourced from wikipedia
I don't know what IIS is, but ISO 3166 specifies 2-letter codes for each country; AFAIK, their list is comprehensive. ISO 3166 site
This is probably way too late, but there's a web service that you can call that would theoretically allow you to automatically databind your controls:
http://www.webservicex.net/country.asmx
May give you another approach and would be better than hard-coding a list yourself.
I've posted a few files to github:gist
Including:
The HTML Select enumerations for the Alpha-2, Alpha-3, and Numeric-3 values, as well as an XSD snippet of those values as enumerations for a simple type restriction.
Check on this link
The CIA world fact book has this information, however, just as you point out in your question there are some disputed countries that are not on their list sometimes, ie. Palestine.
Another source for country names is Natural Earth Data and their cultural map download links which come as shape files ready to be plotted as maps. Here is a direct link to the medium quality map data download page.
Check out angrymonkeycloud.com/geography to get the full list of countries.
It's a free .Net client and so easy to start with, and it uses an API to retrieve values so you should always get the latest updates.
Much, much easier is to use a web service for this task rather than holding your own data store. This way its updated and you can do things like have country-state ajax dropdown sets. http://geodata.solutions is the best one to use, and it has lots of cools stuff like being able to pre-select the user's country based on their IP, and ordering lists by their population.