Recently I found myself in need to perform large batch reverse-geocoding operations.
Large means something about 20k points per request.
I was looking at Nominatim as standalone server. But there is no clue in docs about batch requests (or I just couldn't find it).
Thus the questions is:
1: Could I perform something like this with Nominatim
2: If not - is there other standalone solutions [Not service. May be proprietary]. Main zone of interest is Europe, if it's relevant.
3: What will be approximate time consumption of such request.
Or I am facing building own geocoder above pgsql + postGIS?
thx in advance.
Not a current question, but in the absence of any answers.. There seem to be 2 main solutions. You can either:
use the Google geocoding API, but this will limit you to a check every few seconds, which will make large bulk geocoding very slow, or
download a comprehensive database of place names (such as http://www.geonames.org/), and use a GIS to calculate the nearest points to your data at whatever hierarchical level is most appropriate.
Related
I am trying to make public transport time schedule app using google maps directions api.
Whats the best way to get all of the possible departure_time's for a specific route from one place to another from a specific time?
The problem is, the server is always responding with only one route for one specific time. How can I get all of the following departure_times?
The worst way to do this is asking server every minute if there is some new travel link. But hey, its gonna take a lot of time!
So I thought google might be providing some kind of transport schedules but I can't find any info on google developers webs. I saw only the way to give google schedule information with the help of General Transit Feed Specification (GTFS) here or here.
But I can't find the way to get it from them.
I don't believe google maps directions api will return the information you are looking for as a collection.
The problem with transit data is that calculating a future schedule can require a lot of processing (especially if there are multiple routes involved in the rider reaching their destination) because, basically, the system needs to do a trip plan for each scheduled trip at starting point for the time range.
Google hints at this in their API regarding the alternatives parameter
alternatives — If set to true, specifies that the Directions service may provide more than one route alternative in the response. Note that providing route alternatives may increase the response time from the server.
Also, the different future departure times may actually be different routes or combination of routes (e.g. where multiple routes may come together on the same street for a while - for instance, near a college campus or other transit hub)
In order to get the underlying route data that would have the actual stop times you are looking for you would need to download the transit agencies' GTFS data directly and process it yourself (check our GTFS Data Exchange). This is what your competitors are already doing (e.g. Transit App, Moovit, etc.). There are packages that will do some of this processing for you (e.g. One Bus Away). However, even with the use of existing libraries, there is some heavy lifting involved here (from a development point of view).
As a final note, if you want to pursue using google maps directions api you wouldn't need query it for each minute within a time-range in order to get a series of departure times. You should be able to make a series of calls with the departure time set just past the departure time you got back in the previous call. For example, if the first trip time was 1:00pm set departure_time to 1:05pm and request again, then if the second trip time was 1:20pm set the next departure_time to 1:25pm and request again, and so on to build your list of future trips.
Okay. Firstly your question is not in the right spirit as stackoverflow demands. Check at google's developer console , API section and check if they offer any such API to give you all transport schedules in 24hrs or not ? If there is any such API then good, you can hit that but if not then I am afraid you wont be able to get it unless you hit API after some intervals.
Another suggestion is that , you can try yahoo or bing maps and check if they have any such API for your query.
I'd like to geocode around 2M strings representing cities or states or countries of the whole world every day.
Theses strings are not very clean, (you have to make distinction between "Paris,France" and "Paris,TX" for instance, but you have you also have to geocode "Paris" with the french city !) that's why I'm looking to use GMaps API or OSM Nominatim API.
The brute force solution would be to dump OSM data on my computer and process it locally, but I hope I can find an easier way to do it.
Obviously, I can reduce the amount of strings to be geocoded, store the results not to have to query twice a string, but I would still have hundred thousands strings to geocode...
Thanks !
To comply with the usage policy when using OSM's Nominatim instance you have to wait 1 second between each request. Geocoding 100.000 requests would take around 28 hours which could be feasible for you. Alternatively you can use one of the other Nominatim instances (after checking their usage policy!) or try to install it locally.
I'm new to geocoding so I'm not certain this is even the question I should be asking, but all of the other discussions I've seen on this topic (here and on the Google API forum) are so application specific that I feel like I might be missing a very elementary step - I don't need to know how to implement a store finder - I need to know if I should.
Here is my specific situation - I have been contracted to design an application wherein we will build a database of shops (say, independently owned bars and pubs). This list will continually grow and change as shops close and new ones open. The user can enter his/her point of origin (zip code or address) and be shown a list or map containing all the various shops within a given radius in order of proximity.
I know how to deliver these results from a static database:
One would store the longitude and latitude as columns for each row and then just use that information to check distances.
But I have inherited an (already fairly large) database of shops which have addresses but not coordinates - so I'm not sure what the best way to get those addresses is. I could write a script to query them one at a time against google geocoding, I could have a data entry person manually look up the coordinates for each one and populate the data that way, or maybe there is a third option I'm not aware of.
Is this the right place to be asking this question? Google Maps Geocoding doesn't host a forum of their own, but refers people to Stack Overflow. Other forums on the net dealing with this topic are all relating to a specific technical question but no one seems to be talking about it from a top-down perspective (ie the big picture).
Google imposes a 2,500 queries per day limit on free users and a 100,000 queries a day limit on paid ones - neither of these seem to be up to the task of a site with even moderate traffic if, every time a user makes a request, the entire database (perhaps thousands of shops) are being checked against Google's data. It seems certain we must store the coords locally but even storing them locally, there will have to be checks against Google in order to plot them on a map. If I had a finite number of locations (if, for example, I had six hardware shops) and I wanted to make a store locator, there would be a wealth of discussions, tutorials, and stack overflow questions available to point the way for me, but I'm dealing with a potentially vast number of records and not sure how to proceed or where to begin.
Any advice would be welcome - Additionally, if this is not the best place to be asking this question, a helpful response would be to indicate a better place to post it. I've searched for three days but haven't found what looks like a good resource for asking such subjective questions.
The best way of course would be when you use a geocoding-service to get coordinates and store the coordinates in your DB. But it's not possible with google's geocoding-service, because it's not permitted to store geocoded data permanent.
There are free services without this restriction, some keywords to search for: mapquest, nominatim, geonames(but these services are less accurate than google)
Another option would be to use a FusionTable. The geocoding would run automatically(but the daily limits are the same as for the geocoding-service). The benefit: the geocoding is permanent(you can't access the locations directly by e.g. downloading the DB-dump), but you may use the coordinates for plotting markers(via a FusionTablesLayer) or filtering(e.g. by distance)
The number of entries shouldn't be an issue, 100k is no problem for a database
This is using either Bing Maps API or Google Maps API.
Problem: I am trying to create a route for a salesperson that may have 1 or more meetings that day plus visit selections from a pool. Meaning the salesperson has 1 meeting at a client and they want to stop by some leads but around that meeting time.
I doubt either service can handle this problem directly.Is their as way to get a "table of time between destinations" from these services, without running up my requests.
The only way I know how is to submit each variation on the route, store in a table or array, then based on the services "time to get there", calculate the best order. Then resubmit again. My client could easily burn off 20 - 30 requests per try.
Is there any other option? Maybe an free open source one with sh*ty algorithm then submit the final list to one of the above?
You can use the distance matrix service, see the documentation here:
https://developers.google.com/maps/documentation/distancematrix/
In this way, you will be able to compute multiple route between multiple points and certainly reduce the number of requests overall.
If your request is just to get an optimization for a sales man, you can also check the option on Routing service and its optimize option, see: https://developers.google.com/maps/documentation/directions/#Waypoints
On Bing, route optimization (waypoint reordering) is not available yet but you can achieve this with the Route Matrix API.
My client wants some of the functionality of Google maps namely:
- geocoding
- generating maps with points based on postal code or long.lat
- optimal trip mapping
Their issues with Google maps
- cannot control outages
- postal codes are sometimes inaccurate or not updated frequently for Canada/UK
- they have no way to correct inaccurate information
They would prefer to host the mapping application themselves, but will require postal code updates.
Can anyone suggest such a product?
thanks
"cannot control outages - postal codes are sometimes inaccurate or not updated frequently for Canada/UK - they have no way to correct inaccurate information"
Outages
hosting your own mapping is the only way to control this, but you would be very very hard pushed to beat Google Maps / Bing Maps uptime over the last 5 years. Take a look at the following:
OpenStreetMap for the road imagery data, this is open source data very good in the UK (Im not sure about canada) and you can make your own changes and submit them (or just change the data you have downloaded)
Geoserver, Mapnik or MapServer will read openstreetmapdata and create the image tiles needed to create your own maps in whatever style you wish. Depending on if you dont want all countries and all zoom levels these products can create all the tiles you will need in advance, but usually they have to be created in real time and cached. You need a BIG fast server to manage tile crunching
Openlayers or Leaflet are open source javascript mapping platforms that will display your tiles for you
Obviously this is just for road maps, aerial imagery would cost you an absolute fortune.
Post Code Data
Many people do not realize that UK postcode data for latitude and longitude is now completely free and available to download every quarter from the official source (ordinance survey) http://www.ordnancesurvey.co.uk/oswebsite/products/code-point-open/index.html.
This is the same data source Google will use and there is none better but it will always contain inaccuracies and always be a few months out of date.
Finally
Hopefully that answer the question you asked and gives you information to inform your client. Now for the question you didn't ask "Is this approach good value to my client?".
I won't presume to know your business or client, however what I described above is possible but with one to many months of work involved to get it all working together and even then it wont have any where near the performance or uptime of something like google /bing maps and only offers a small subset of their features.
I think you're looking for something like Caliper-It's a very custom, and I would expect expensive, solution. Not suggested.
http://www.caliper.com/GISMappingSoftwareDevelopment.htm
One solution could be to use two different mapping services and compare their results, this way there's a much better chance the data is accurate. You can also fix inaccurate data by creating a system which acts as a barrier between the API and your user, where data you know is inaccurate is corrected before it's displayed. Not sure exactly what you're doing though, so this might not work for you.
Is trip mapping/routing the basic functionality you want to do?
Before rushing into rolling your own, I'd suggest a good think about the consequences of doing so. The first that springs to mind is whilst the pros are that you can now control your data, the cons are that you now control your data.
So you are going to have to consider where and when you get updates and the processes you are going to have to employ to keep your maps in sync with the rest of the world. There are a lot of headaches involved in these things which is why so many people use externally hosted solutions such as Googles.