I have a json file thats roughly 480mb of geolocation points. I was wondering if someone knows of a good 'pattern' to use when trying to visualise the data. The problem I'm encountering is that the data has to be loaded into Google maps from the get go. This is causing all kinds of obvious issues.
I don't have to do this through google. It just seemed like the obvious choice.
With that much data, it may make more sense to handle it on the server side instead of client side. You could set up Geoserver with your appropriate data points. Using OpenLayers, you could overlay your points from Geoserver on top of Google Maps or potentially even on top of your own map if you want to cut out Google Maps all together. The heavy duty processing then happens on the server and only images are displayed in the browser. This cuts down on network traffic and the amount of processing the browser has to do. If you set up Geoserver to do caching, the server won't even have to work very hard.
It really depends on what kind of data this is.
If these are points for polylines or polygons you might try to encode the points (http://code.google.com/apis/maps/documentation/utilities/polylinealgorithm.html and http://code.google.com/apis/maps/documentation/utilities/polylineutility.html). There are also functions which you can use to encode the points. This will significantly reduce the size of your data.
You might also want to consider loading data depending on zoom level on the map. (I am not sure what you mean by "data has to be loaded from the get go" - you can load the data into the map depending on events, etc...) .
Fusion tables mentioned above will only accept 100MB of data.
I can be more specific if you explain the nature of your data and what you trying to do in more details. Hope this helps.
Try Google Fusion Tables
Related
Is there any good library that will enable to see the coarse location distribution.
I have some million data points to plot on a map. Doing it on google maps is going to be heavy. I want this to be like a dot plot.
The closest I can think to the functionality you are looking for would be to implement a heatmap layer : https://developers.google.com/maps/documentation/javascript/examples/layer-heatmap.
This would allow you to see the distribution of points without plotting them individually and thus saving you some client-side work.
How well this will work with a million+ points however I am not sure. You might be best off looking into a fusion tables combination.
We are creating a speed limit map application using different colors to highlight street with different speed limits (similar to ITO speed limit map: http://www.itoworld.com/map/124?lon=-79.37151&lat=43.74796&zoom=12).
The problem we have is that we are conducting our own research and have our own speed limit data instead of pulling the data from OpenStreetMap or Google Map like ITO map. We also need to create a data storage in order to dynamically update the map as we add more speed limit information in the future.
Is there anyway to create our own instance of OpenStreetMap and replace only the speed limit information with our own data? We don't have any vector data and we have no experience working with them.
Is there any suggestion of tools to use for creating highlighting layers based on the speed limit we have? Is OpenLayers a good option?
Any help is appreciated, thank you very much.
Update 2013/11/20
Thank you very much for your answers, now we have a much better understanding of your problem. This is a university design project so we basically have no budget. We are looking for:
1) A basic "base map" that include the basic tile information (openstreetmap seems a good choice since google map api doesn't provide free road information as long as we can find)
2) A geo data server that can host our own street speed limit data (looks like geoserver and mapserver are good choices), or a design simple database that can fulfill our need(doesn't know is it possible yet)
3) A plotting tool that can render our speed limit data as "group of lines" on the map since these data will be changed frequently (openlayers and leaflet are good candidates).
Is there anything else needed?
What you want to do is a trivial programming task once you have decided a few things:
These are probably the three biggest questions you need to answer. I added some commentary, but look at each of these questions beyond this post to find what works for you.
Who do you want to use for your map? Since you only have one type of data you will want to display that data on someone else's nice looking map. The big choices are Bing, Google, OpenLayers/OSM, and ESRI. Your choice will most likely be driven by the licensing of the above services and if you are willing to pay or not. A need to support mobile devices may also factor into your decision. Since the map is what your users will see, choose the best looking map you can afford.
How will you serve up your data? You have several options to serve your speed limit data. GeoServer and MapServer and ESRI are some popular mapping software packages. If you only displaying a few layers of data all mapping software will be overkill. The actual software to render your map data will most likely affect only your pocket book, so free is good here usually.
Tiles vs Lines
You will server your data as either a group of lines sent to the browser, or as pre-rendered tiles to be loaded on top of the map. If you data changes frequently you will want to serve it dynamically as line data (an array of points.) If your data does not change frequently, you should consider tiling your data. Tiling involves pre-rending of the entire map at all zoom levels. This allows the map to be loaded very fast and this how almost all base maps are rendered. The downside is that the tile generation can take a long amount of time and tiles can take a large amount of space.
This is a very broad question. There are many components to drawing your own speed limit map.
On the front-end, there is a web browser map interface. OpenLayers is good at that. There are plenty of other tools that can do this as well, such as Leaflet or even Google Maps API.
Next is something to provide the actual speed limit route data. This can be served as a vector layer or a raster layer. There are plenty of tools here, too. UMN Mapserver is free and reasonably good. ESRI makes a whole fleet of products in this area as well.
The speed limit route data also needs to be saved somehow. This can be done in files or in a database such as PostGIS. Again, lots of great options.
It is the role of the system architect to determine which technologies to employ to solve the problem.
My client wants some of the functionality of Google maps namely:
- geocoding
- generating maps with points based on postal code or long.lat
- optimal trip mapping
Their issues with Google maps
- cannot control outages
- postal codes are sometimes inaccurate or not updated frequently for Canada/UK
- they have no way to correct inaccurate information
They would prefer to host the mapping application themselves, but will require postal code updates.
Can anyone suggest such a product?
thanks
"cannot control outages - postal codes are sometimes inaccurate or not updated frequently for Canada/UK - they have no way to correct inaccurate information"
Outages
hosting your own mapping is the only way to control this, but you would be very very hard pushed to beat Google Maps / Bing Maps uptime over the last 5 years. Take a look at the following:
OpenStreetMap for the road imagery data, this is open source data very good in the UK (Im not sure about canada) and you can make your own changes and submit them (or just change the data you have downloaded)
Geoserver, Mapnik or MapServer will read openstreetmapdata and create the image tiles needed to create your own maps in whatever style you wish. Depending on if you dont want all countries and all zoom levels these products can create all the tiles you will need in advance, but usually they have to be created in real time and cached. You need a BIG fast server to manage tile crunching
Openlayers or Leaflet are open source javascript mapping platforms that will display your tiles for you
Obviously this is just for road maps, aerial imagery would cost you an absolute fortune.
Post Code Data
Many people do not realize that UK postcode data for latitude and longitude is now completely free and available to download every quarter from the official source (ordinance survey) http://www.ordnancesurvey.co.uk/oswebsite/products/code-point-open/index.html.
This is the same data source Google will use and there is none better but it will always contain inaccuracies and always be a few months out of date.
Finally
Hopefully that answer the question you asked and gives you information to inform your client. Now for the question you didn't ask "Is this approach good value to my client?".
I won't presume to know your business or client, however what I described above is possible but with one to many months of work involved to get it all working together and even then it wont have any where near the performance or uptime of something like google /bing maps and only offers a small subset of their features.
I think you're looking for something like Caliper-It's a very custom, and I would expect expensive, solution. Not suggested.
http://www.caliper.com/GISMappingSoftwareDevelopment.htm
One solution could be to use two different mapping services and compare their results, this way there's a much better chance the data is accurate. You can also fix inaccurate data by creating a system which acts as a barrier between the API and your user, where data you know is inaccurate is corrected before it's displayed. Not sure exactly what you're doing though, so this might not work for you.
Is trip mapping/routing the basic functionality you want to do?
Before rushing into rolling your own, I'd suggest a good think about the consequences of doing so. The first that springs to mind is whilst the pros are that you can now control your data, the cons are that you now control your data.
So you are going to have to consider where and when you get updates and the processes you are going to have to employ to keep your maps in sync with the rest of the world. There are a lot of headaches involved in these things which is why so many people use externally hosted solutions such as Googles.
I am working on creating a tool to select a place via Google Maps API. This part is simple and I have no problems with it, but it is too specific. I want to do something similar to this, but need to accommodate the entire world, not just zip codes.
I would think the same effect should be able to be achieved by the highlighting an administrative level. Does this require KML file, and if so can anyone through a provider that might have this level of detail?
You can get administrative boundaries for the world here http://www.gadm.org/ or here http://www.unsalb.org/. You will not be able to achieve this with a single KML file though, as this would be one big file (I am talking about at least four hundred MB and there is a 4MB limit for KML files on Maps).
You will need to store the data in the db and pull out only relevant boundaries and draw them as polygons on the map. In other words it sounds like you might need a GIS server(open or ArcGis). Although you could build it from scratch using any geo-enabled db such as mySQL or MSSQL.
How do I take shapefiles and extract lat/lng coords so I can plot polygons on Google Maps?
http://www2.census.gov/cgi-bin/shapefiles/national-files
I asked this question here:
http://groups.google.com/group/Google-Maps-API/browse_thread/thread/18763b4b0cb996c7
and they told me WHAT to do, but not HOW to do it =P
Thx!
It depends on how you need to accomplish this. If you just need a few shapes, you can look up the coordinates in those files yourself. You can use those coordinates to create a GPolygon in Google Maps.
If you need lots of shapes - you'll need to do it programmatically. I would suggest using your favorite language to parse the XML file and retrieve the coordinates for each shape.
I had a similar problem last year when I was developing a screensaver to render presidential polling data. I didn't really want to invest the time to parse the Shapefiles data on the census site (The spec is here if you missed it).
Not sure if I actually saved any time here, but I ended writing a python app to render the 50 states onscreen, trace the edges and then store the data in a simple text format. Not sure if my data is high res enough for your application, but you can grab the data I generated here:
http://www.cannonade.net/pnt.zip
N.B. The data I generate are not latitude/longitudes, but with some scaling you should be able to translate them.
Good luck.
I had better luck using the ARC files at http://www.census.gov/geo/www/cob/index.html
I can't find the webpage right now, but I did find one that had actual code. Google something like "arc to kml" and go from there.