Geomesa: saving KML in Accumulo - geomesa

How can I save KML data in Geomesa? My main interest is programmatically and not with a Geomesa cmd line tool. Of'course a spatial index should be used.


The export could be handled by getting a GeoMesa FeatureCollection representing the desired features and using the XSD-KML GeoTools module [1] to encode the collection as KML.
Disclaimer: I haven't used this plugin myself, but looks like it'd do exactly what you want!
http://docs.geotools.org/latest/userguide/extension/xsd/kml.html#

Related

Indexing JSON in GeoMesa

Assume I want to perist JSON files in GeoMesa (on Accumulo). These JSON files have geometries and time. Can I use a XZ3 index? If yes then how?
NB: By JSON I am not refering to GeoJSON.

You can write a GeoMesa converter (a configuration file) to extract the values you want out of your JSON and into a GeoTools SimpleFeature, and ingest those into GeoMesa. Download the Accumulo distribution from github and look at the example under examples/ingest/json/.
Full documentation for converters is available here.
You also have the option of storing JSON strings as attributes, and querying them using JSON-Path. There is more information on that here.
The indices created for your data will depend on the attributes present. If you have a non-point geometry and a date defined, then you will automatically build an XZ3 index. More information on indices is available here and here

AWS Glue Crawler Classifies json file as UNKNOWN

I'm working on an ETL job that will ingest JSON files into a RDS staging table. The crawler I've configured classifies JSON files without issue as long as they are under 1MB in size. If I minify a file (instead of pretty print) it will classify the file without issue if the result is under 1MB.
I'm having trouble coming up with a workaround. I tried converting the JSON to BSON or GZIPing the JSON file but it is still classified as UNKNOWN.
Has anyone else run into this issue? Is there a better way to do this?

I have two json files which are 42mb and 16mb, partitioned on S3 as path:
s3://bucket/stg/year/month/_0.json
s3://bucket/stg/year/month/_1.json
I had the same problem as you, crawler classification as UNKNOWN.
I were able to solved it:
You must create custom classifier with jsonPath as "$[*]" then create new crawler with the classifier.
Run your new crawler with the data on S3 and proper schema will be created.
DO NOT update your current crawler with the classifier as it won't apply the change, I don't know why, maybe because of classifier versioning AWS mentioned in their documents. Create new crawler make them work

As mentioned in
https://docs.aws.amazon.com/glue/latest/dg/custom-classifier.html#custom-classifier-json
When you run a crawler using the built-in JSON classifier, the entire file is used to define the schema. Because you don’t specify a JSON path, the crawler treats the data as one object, that is, just an array.
That is something which Dung also pointed out in his answer.

Please also note that file encoding can lead to JSON being classified as UNKNOWN. Please try and re-encode the file as UTF-8.

Convert Json file into GraphJSON to be imported into Titan

I have been looking at ways to convert a JSON file into a GraphJSON graph and I have come across the GraphJSON Reader and Writer Library.
However, what I do not really understand is whether I can read out directly from a path where a JSON file resides and parse it into a graph/GraphJSON.
Can you help?

This is how I would solve this issue:
Read your JSON files using GSON or Jackson, then
Feed this data into a subclass of Vertex/Edge of your implementation of these Tinkerpop 3 interfaces.
Use the GraphSON writer methods to "graphitise" your data, save your data into an OutputStream.
I'm assuming you're using Tinkerpop3 and Titan 1.0.0, this is the right documentation.
Good luck!
P.S: If you're doing this for the sack of importing data into Titan, you might be overcomplicating the issue of data import. Just import it straight away.

Use JSON file in Gephi

I have a json file containing network data that I'd like to visualize using a network visualization software like Gephi. But Gephi does not accept this type of files. Is there a way to convert the json file or are there any other network visualization softwares out there that can? I am a Mac user.

You need to use networkx in Python, construct your graph and Export your graph in a format that Gephi can load such as gexfor graphml. See the link for examples of the supported formats. Apparently R is an option, too, provided that igraph has a json reader

Visjs can import Gephi JSON and display it as a graph/network:
http://visjs.org/examples/network/data/importingFromGephi.html
You probably could adapt your JSON to fit this format.

Best way to overlay an ESRI shapefile on google maps?

What is the best way to overlay a shapefile in Google Maps?
After some reading suggests that, converting shapefile into KML and overlaying KML in Google map is an option. If that is the case, how do I convert a shapefile into KMLs?
I believe there would be a few open source tools available for this conversion. Can anyone point to a tool to covert a shapefile to KML without any loss of data?

I like using (open source and gui friendly) Quantum GIS to convert the shapefile to kml.
Google Maps API supports only a subset of the KML standard. One limitation is file size.
To reduce your file size, you can Quantum GIS's "simplify geometries" function. This "smooths" polygons.
Then you can select your layer and do a "save as kml" on it.
If you need to process a bunch of files, the process can be batched with Quantum GIS's ogr2ogr command from osgeo4w shell.
Finally, I recommend zipping your kml (with your favorite compression program) for reduced file size and saving it as kmz.

as of 12.03.2019 FusionTables is no more...
Import the Shapefile into Google FusionTables ( http://www.google.com/fusiontables ) using http://www.shpescape.com/ and from there you can use the data in a number of ways, eg. display it using GoogleMaps.

I would not use KML. Instead, use GeoJSON which you can natively consume in Google Maps API now. It is a newer feature that didn't exist from the original responses.
In any case, simply open the SHP file in Quantum GIS, and then you can output it in any format you like (KML, GeoJSON).
If you are using Google Maps for Work, I found a premium extension that handles loading shapefiles directly where you can just connect direct to the shapefile that you generate from ESRI. I did a search on the CMaps site and found this snippet which loaded US by state shapefile: https://gmapsplugin.net/cmapsanalytics/assets/shapes/usstates.shp
var cMap = new centigon.locationIntelligence.MapView();
cMap.key([your_api_key]);
cMap.layerNames(["Basic Shapes"]);
cMap.dbfKeys([['Alabama','Alaska','Arizona','Arkansas','California','Colorado','Connecticut','Delaware','District of Columbia','Florida','Georgia','Hawaii','Idaho','Illinois','Indiana','Iowa','Kansas','Kentucky','Louisiana','Maine','Maryland','Massachusetts','Michigan','Minnesota','Mississippi','Missouri','Montana','Nebraska','Nevada','New Hampshire','New Jersey','New Mexico','New York','North Carolina','North Dakota','Ohio','Oklahoma','Oregon','Pennsylvania','Rhode Island','South Carolina','South Dakota','Tennessee','Texas','Utah','Vermont','Virginia','Washington','West Virginia','Wisconsin','Wyoming']]);
cMap.userShapeKeys([['Massachusetts','Minnesota','Montana','North Dakota','Hawaii','Idaho','Washington','Arizona','California','Colorado','Nevada','New Mexico','Oregon','Utah','Wyoming','Arkansas','Iowa','Kansas','Missouri','Nebraska','Oklahoma','South Dakota','Louisiana','Texas','Connecticut','New Hampshire','Rhode Island','Vermont','Alabama','Florida','Georgia','Mississippi','South Carolina','Illinois','Indiana','Kentucky','North Carolina','Ohio','Tennessee','Virginia','Wisconsin','West Virginia','Delaware','District of Columbia','Maryland','New Jersey','New York','Pennsylvania','Maine','Michigan','Alaska']]);
cMap.labels([['Massachusetts','Minnesota','Montana','North Dakota','Hawaii','Idaho','Washington','Arizona','California','Colorado','Nevada','New Mexico','Oregon','Utah','Wyoming','Arkansas','Iowa','Kansas','Missouri','Nebraska','Oklahoma','South Dakota','Louisiana','Texas','Connecticut','New Hampshire','Rhode Island','Vermont','Alabama','Florida','Georgia','Mississippi','South Carolina','Illinois','Indiana','Kentucky','North Carolina','Ohio','Tennessee','Virginia','Wisconsin','West Virginia','Delaware','District of Columbia','Maryland','New Jersey','New York','Pennsylvania','Maine','Michigan','Alaska']]);
cMap.polyDataSources([centigon.locationIntelligence.CMapAnalytics.DATA_PROVIDERS.SHAPE_DATAPROVIDER]);
cMap.layerTypes([centigon.mapping.Layer.TYPE.POLY]);
cMap.locations([["https://gmapsplugin.net/cmapsanalytics/assets/shapes/usstates.shp"]]);
cMap.panTo("USA");
cMap.zoomLevel(3);

Do you mean shapefile as in an Esri shapefile? Either way, you should be able to perform the conversion using ogr2ogr, which is available in the GDAL packages. You need the .shp file and ideally the corresponding .dbf file (which will provide contextual information).
Also, consider using a tool like MapShaper to reduce the complexity of your shapefiles before transforming them into KML; you'll reduce filesize substantially depending on how much detail you need.

Free "Export to KML" script for ArcGIS 9
Here is a list of available methods that someone found.
Also, it seems to me that the most efficient representation of a polygon layer is by using Google Maps API's polyline encoding, which significantly compresses lat-lng data. But getting into that format takes work: use ArcMap to export Shape as lat/lng coordinates, then convert into polylines using Google Maps API.

Just to update these answers, ESRI has included this tool, known as Layer to KML in ArcMap 10.X. Also, a Map to KML tool exists.
Simply import the desired layer (vector or raster) and choose the output location, resolution, etc. Very simple tool.

2018 already... I've found this fantastic online tool http://mapshaper.org/ to convert from ESRI shapefiles to SVG, TopoJSON, GeoJSON.
Here is the explanation of how to use it https://www.statsilk.com/maps/convert-esri-shapefile-map-geojson-format
Fast and straightforward! :)