I have a big shape file (about 1GB) and I need to display and update data from this file with
an online map (mapbox).
As I understand I need to convert this shapefile into some DB (Spatialite? PostGIS)
and then query it by map BBOX to display by parts.
What is the easiest way to do this? Thanks
Thats a huge topic and depends very much on the usecase. If you want to dive deeper into the topic I would recommend this articel. It describes the underlaying theory and gives a working example for a SQL-Query. For me, it was sufficient to get going with a viable prototype. I think Postgres (with it's Postgis extension) would be a good choice since the data can be directly served as mvt.
I found this site very useful for displaying GeoJson on the fly, it usually copes well with large data, failing that you can import it too. https://geojson.io
Related
TL;DR: I'm looking for some resources on generating GRIB2 data sets on the fly, ideally using in-house-generated wind data in a CSV format.
We have a bunch of data for a series of localized weather stations monitoring wind information around our city. They report in at ~2-3 minute intervals (far more frequent than standard weather data), and from their reports we have lat, lon, wind speed, and wind direction. Someone went and told the boss about these really slick visualizations, like this that can display wind speed and direction, and it's my job to make it happen.
The above plug-in for Leaflet, GitHub here, as well as several others, all use GRIB2 data, which from my research involves a left/right set of data and an up/down set of data for a series of points plotted out across a region.
The problem I'm having is that I've only found a handful of tools that interact with GRIB2 data, and most seem to decode data from the GRIB2 dataset, and only one tool running on Fortran seems to exist that compiles GRIB2 data together.
So, is there any way to generate GRIB2 data on the fly using proprietary data at 2-3 minute intervals?
I've gone through this resource on NOAA's website, which is where I found a few tools.
I know how frustrating it can be to work with GRIB and some of the other science/weather related formats. This may not be the best answer, but it might be your only answer as I find these types of questions to only gather dust because of the general lack of knowledge with the formats and tools.
From what I remember, CDO tools (link here) can do some magical things - but I am not that experienced with it. I do use it for converting satellite data to plain text and it's been an absolute lifesaver! So I will explain :
My suggestion was to first convert the CSV to netCDF. I had a link saved for this a long time ago, but never came to really needing it. (discussion here). Essentially, some python code should be able to do the conversion for you. There may be several ways to do this, but I have never looked into it beyond initial research.
Next, you should be able to convert .nc to .grib using CDO. I know it can do quite alot. Here is a discussion regarding this, so it must be able to be done.
I also see at this link where someone converts grib to netcdf, but you should be able to do it in reverse as well. I just don't know the exact commands. From the link :
As an example of use of CDO, converting from GRIB to netCDF can be as simple as
cdo -f nc copy file.grb file.nc
I would suspect its just the reverse but probably something like :
cdo -f grb file.nc file.grb
Hopefully you can put things together for it to work without being too hack-y.
You can do this in a simple python script using pandas , xarray and cfgrib
import pandas as pd
import cfgrib
data = pd.read_csv('your_csv_data.csv')
xarray_data = data.to_xarray()
cfgrib.to_grib(xarray_data, 'out2.grib')
Please note that you have to define grib specifications first before you store as grib data.
I know this is a somewhat subjective question but I want to throw it out there anyway because if there is some insight that I've missed, asking this question will save me (and hopefully others) a great deal of searching :-)
Basically, what is the best type of database for when you have a number of items interlinked. For example:
A->B->C->D->E->F->G->H->I
And you want to be able to quickly find the shortest path (or the number of paths ideally) between A and I?
I would normally just use a relational database for this, but I'm not sure a MapReduce style database is any better and then I wondered if there was a type of database I hadn't considered altogether...
As always, all help gratefully received :)
This is exactly the sort of thing neo4j is designed for. It's a graph database that provides operations for doing graph traversal and other common operations.
I would suggest looking into MongoDB - It's NOT a relational database, and all your data gets saved and passed as JSON, it's easy scale-able and build for performance, and comes with a couple of tricks up its sleeve as well, like MapReduce build into mongoDB and way way more.
Since the data is saved as JSON you can save objects within objects and arrays well:
{
foo:"bar",
animals: ['cat','dog','fish'],
blog:{
post:"Hello World!",
comments:["this is cool","Hello back"]
}
}
Checkout - http://www.mongodb.org/
or http://www.mongodb.org/display/DOCS/MapReduce/
for more info
I want to edit the corresponding text/numeric data type columns of a vector through openlayres, when a user click/mouse over a vector. like,
http://dev4.mapgears.com/bdga/bdgaWFS-T.html#
any tutorial page for how to do it?
I gone through it ,
it was difficult for me to understand from its js codes.
-posted in gis.stackexchange
In the example, OpenLayers is getting data from a WFS server with a Postgres/PostGIS backend.
You could try FeatureServer, but the GIS StackExchange is probably the best place to ask for help on this.
I have a json file thats roughly 480mb of geolocation points. I was wondering if someone knows of a good 'pattern' to use when trying to visualise the data. The problem I'm encountering is that the data has to be loaded into Google maps from the get go. This is causing all kinds of obvious issues.
I don't have to do this through google. It just seemed like the obvious choice.
With that much data, it may make more sense to handle it on the server side instead of client side. You could set up Geoserver with your appropriate data points. Using OpenLayers, you could overlay your points from Geoserver on top of Google Maps or potentially even on top of your own map if you want to cut out Google Maps all together. The heavy duty processing then happens on the server and only images are displayed in the browser. This cuts down on network traffic and the amount of processing the browser has to do. If you set up Geoserver to do caching, the server won't even have to work very hard.
It really depends on what kind of data this is.
If these are points for polylines or polygons you might try to encode the points (http://code.google.com/apis/maps/documentation/utilities/polylinealgorithm.html and http://code.google.com/apis/maps/documentation/utilities/polylineutility.html). There are also functions which you can use to encode the points. This will significantly reduce the size of your data.
You might also want to consider loading data depending on zoom level on the map. (I am not sure what you mean by "data has to be loaded from the get go" - you can load the data into the map depending on events, etc...) .
Fusion tables mentioned above will only accept 100MB of data.
I can be more specific if you explain the nature of your data and what you trying to do in more details. Hope this helps.
Try Google Fusion Tables
How do I take shapefiles and extract lat/lng coords so I can plot polygons on Google Maps?
http://www2.census.gov/cgi-bin/shapefiles/national-files
I asked this question here:
http://groups.google.com/group/Google-Maps-API/browse_thread/thread/18763b4b0cb996c7
and they told me WHAT to do, but not HOW to do it =P
Thx!
It depends on how you need to accomplish this. If you just need a few shapes, you can look up the coordinates in those files yourself. You can use those coordinates to create a GPolygon in Google Maps.
If you need lots of shapes - you'll need to do it programmatically. I would suggest using your favorite language to parse the XML file and retrieve the coordinates for each shape.
I had a similar problem last year when I was developing a screensaver to render presidential polling data. I didn't really want to invest the time to parse the Shapefiles data on the census site (The spec is here if you missed it).
Not sure if I actually saved any time here, but I ended writing a python app to render the 50 states onscreen, trace the edges and then store the data in a simple text format. Not sure if my data is high res enough for your application, but you can grab the data I generated here:
http://www.cannonade.net/pnt.zip
N.B. The data I generate are not latitude/longitudes, but with some scaling you should be able to translate them.
Good luck.
I had better luck using the ARC files at http://www.census.gov/geo/www/cob/index.html
I can't find the webpage right now, but I did find one that had actual code. Google something like "arc to kml" and go from there.