I have a csv file that I want to convert to GeoJson.
lon,lat,val,id
-97.1589432,25.9642008,0,2690
-97.1682294,25.9761856,0,2691
-97.1775156,25.9881704,0,2692
I run the following ogr2ogr command
ogr2ogr -f GEOJson geo_result.json source.csv
and I get this result
{
"type": "FeatureCollection",
"features": [
{ "type": "Feature", "properties": { "lon": "-97.1589432", "lat": "25.9642008", "val": "0", "id": "2690" }, "geometry": null },
{ "type": "Feature", "properties": { "lon": "-97.1682294", "lat": "25.9761856", "val": "0", "id": "2691" }, "geometry": null },
{ "type": "Feature", "properties": { "lon": "-97.1775156", "lat": "25.9881704", "val": "0", "id": "2692" }, "geometry": null }
]
}
Why is geometry null?
How can I tell ogr2ogr which values are lat & lon?
You need to wrap the CSV file in a VRT file which specifies the meaning of the columns.
<OGRVRTDataSource>
<OGRVRTLayer name="source">
<SrcDataSource>source.csv</SrcDataSource>
<GeometryType>wkbPoint</GeometryType>
<LayerSRS>WGS84</LayerSRS>
<GeometryField encoding="PointFromColumns" x="lon" y="lat"/>
</OGRVRTLayer>
</OGRVRTDataSource>
And then use the VRT file as the input:
ogr2ogr -f GEOJson geo_result.json source.vrt
QGIS (which uses OGR) has an interface which cleverly guesses the columns, and that works very well. So if you have to use your conversion only once it might be worth trying it with a GUI like QGIS.
See the CSV driver page from OGR for more details:
http://www.gdal.org/ogr/drv_csv.html
Related
I have a JSON file that contains the attribute of several parks along with location (geometry) as a point. I was wondering how to convert the JSON to postgreSQL format. Indeed, I have tried several ways such as SQLizer and MapForce, but I was not able to convert them. Is there any way to convert this JSON which has geometry, to postgreSQL format?
I appreciate any help.
Below you can find the script.
var lenneparks = {
"type": "FeatureCollection",
"crs": {
"type": "name",
"properties": {
"name": "urn:ogc:def:crs:OGC:1.3:CRS84"
}
},
"features": [
{
"type": "Feature",
"properties": {
"place": "Aachen Kurpark",
"year": "1853 (Hi)",
"text": "Elisengarten, kleine Parkanlage in der Innenstadt, rückwärtig vom Elisenbrunnen"
},
"geometry": {
"type": "Point",
"coordinates": [
6.086027,
50.774247
]
}
},
{
"type": "Feature",
"properties": {
"place": "Aachen",
"year": "ca. 1862 (Hi)",
"text": "Staatsprokurator Dubusc"
},
"geometry": {
"type": "Point",
"coordinates": [
6.0838868,
50.7653455
]
}
}
]
};
EDIT 1: Corrected SQL comments
Have you tried the PostGIS extension? It comes with really handy functions to import such data, such as:
-- To create a geometry object from your GeoJSON
SELECT ST_GeomFromGeoJSON('{"type":"Point","coordinates":[-48.23456,20.12345]}') As geometry;
-- To see the WKT of your GeoJSON
SELECT ST_AsText(ST_GeomFromGeoJSON('{"type":"Point","coordinates":[-48.23456,20.12345]}')) As geometry;
EDIT 2: Creating records for each geometry
This function will create a table containing one record for each json element in the array features, from there you can start parsing the data you need for creating your tables... I hope it helps:
CREATE TEMPORARY TABLE features AS
SELECT json_array_elements('{
"type": "FeatureCollection",
"crs": {
"type": "name",
"properties": {
"name": "urn:ogc:def:crs:OGC:1.3:CRS84"
}
},
"features": [
{
"type": "Feature",
"properties": {
"place": "Aachen Kurpark",
"year": "1853 (Hi)",
"text": "Elisengarten, kleine Parkanlage in der Innenstadt, rückwärtig vom Elisenbrunnen"
},
"geometry": {
"type": "Point",
"coordinates": [
6.086027,
50.774247
]
}
},
{
"type": "Feature",
"properties": {
"place": "Aachen",
"year": "ca. 1862 (Hi)",
"text": "Staatsprokurator Dubusc"
},
"geometry": {
"type": "Point",
"coordinates": [
6.0838868,
50.7653455
]
}
}
]
}'::JSON -> 'features') as features;
SELECT * FROM features;
EDIT 3: Query to extract info from json table
SELECT
features -> 'geometry' -> 'coordinates' -> 0 AS lat,
features -> 'geometry' -> 'coordinates' -> 1 AS lon,
features -> 'properties' -> 'place'::TEXT,
features -> 'properties' -> 'year'::TEXT,
features -> 'properties' -> 'text'::TEXT
FROM features;
EDIT 4: Extracting geometries from the json table and converting them into WKT and Geometry
SELECT ST_GeomFromGeoJSON((features -> 'geometry')::text)
FROM features;
SELECT ST_AsText(ST_GeomFromGeoJSON((features -> 'geometry')::TEXT))
FROM features;
I am using python spark sql to read streaming files sent from Kinesis. The nested Json streams are saved and zipped in S3 bucket. These gzip files are not standard json format, it just has a bunch of json objects without comma to separate. And it does not have [ ] at the beginning nor the end of the file. Here is the sample, there are 2 json objects in the example, I expect each object could be a row in the output:
{
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" },
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
}
}
{
"id": "0002",
"type": "donut",
"name": "Cake",
"ppu": 0.65,
"batters":
{
"batter":
[
{ "id": "1221", "type": "Regular" },
{ "id": "1223", "type": "Chocolate" },
{ "id": "1225", "type": "Blueberry" },
{ "id": "1228", "type": "Devil's Food" }
]
}
}
I can use pyspark sqlContext.read.json('streaming.json') to read this file, but the function only returns first object. When I add [ ] and comma seperate to those json objects, it reads those 2 objects succeesfully. But in my case, I have 1 TB of files for each day, its hard to translate the streaming into standard json files. I am new to spark, is there a way to read those streaming json objects using spark or sparkSQL? The output I expect is json flatten csv or dataframe saved back to S3. Thanks a lot.
I have a problem in retrieving entities using georeferenced queries.
Use the v2 syntax.
This is my query:
GET /v2/entities?georel=near;maxDistance:1000&geometry=point&coords=13.52,43.61
and this is my entity:
{
"id": "p1",
"type": "pm",
"address": {
"type": "Text",
"value": "Via Roma "
},
"allowedVehicleType": {
"type": "Text",
"value": "car"
},
"category": {
"type": "Text",
"value": "onstreet"
},
"location": {
"type": "geo:json",
"value": {
"type": "Point",
"coordinates": [ 13.5094, 43.6246 ]
}
},
"name": {
"type": "Text",
"value": "p1"
},
"totalSpotNumber": {
"type": "Number",
"value": 32
}
}
What is wrong?
I followed the official documentation but I can not get any results as well.
I also tried to reverse the coordinates, but the result does not change.
Any suggestion is welcome.
Note that longitude comes before latitude in GeoJSON coordinates, while the coords parameters does in the opposite way.
Thus, assuming that your entity is located in Ancona city, I think that using "coordinates": [ 43.6246, 13.5094 ] will solve the problem.
I have a roughly 10G JSON file. Each line contains exactly one JSON document. I was wondering what is the best way to convert this to Avro. Ideally I would like to keep several documents (like 10M) per file. I think Avro supports having multiple documents in the same file.
You should be able to use Avro tools' fromjson command (see here for more information and examples). You'll probably want to split your file into 10M chunks beforehand (for example using split(1)).
The easiest way to convert a large JSON file to Avro is using avro-tools from the Avro website.
After creating a simple schema the file can be directly converted.
java -jar avro-tools-1.7.7.jar fromjson --schema-file cpc.avsc --codec deflate test.1g.json > test.1g.deflate.avro
The example schema:
{
"type": "record",
"name": "cpc_schema",
"namespace": "com.streambright.avro",
"fields": [{
"name": "section",
"type": "string",
"doc": "Section of the CPC"
}, {
"name": "class",
"type": "string",
"doc": "Class of the CPC"
}, {
"name": "subclass",
"type": "string",
"doc": "Subclass of the CPC"
}, {
"name": "main_group",
"type": "string",
"doc": "Main-group of the CPC"
}, {
"name": "subgroup",
"type": "string",
"doc": "Subgroup of the CPC"
}, {
"name": "classification_value",
"type": "string",
"doc": "Classification value of the CPC"
}, {
"name": "doc_number",
"type": "string",
"doc": "Patent doc_number"
}, {
"name": "updated_at",
"type": "string",
"doc": "Document update time"
}],
"doc:": "A basic schema for CPC codes"
}
I'm stuck with issue for getting web-service response from JSON in Sencha Touch.
I'm getting response from server.
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"id": "business_poi_en.1",
"geometry": {
"type": "Point",
"coordinates": [
28.21962354993591,
36.452844361147314
]
},
"geometry_name": "geom",
"properties": {
"status": "M",
"score": 100,
"side": "R",
"ref_id": 0,
"id": null,
"business_p": "AQUARIOUM",
}
},
{
"type": "Feature",
"id": "business_poi_en.2",
"geometry": {
"type": "Point",
"coordinates": [
28.225417523605692,
36.436470953176716
]
},
"geometry_name": "geom",
"properties": {
"status": "M",
"score": 68.44,
"match_type": "A",
"side": "L",
"ref_id": 0,
"id": null,
"business_p": "ZIGOS",
}
},
.... So On ....
I want to fetch data for coordinates from geometry tag, so I can display it on map via these coordinates.I also want to fetch data for business_p from properties tag as displaying title on map.
But, I can't get both of values for coordinates & business_p at same time from same response.
Any idea for getting values at same time ?
Any suggestion will be appreciated.
Please help with this issue.
Thanks in advance.
Your Store's proxy's reader should have correct rootProperty which is common parent of both coordinates and business_p. For that your response should be wrapped into a top level tag like this
{
"featureResponse": [
{
"type": "FeatureCollection",
"features": []
},
{
"type": "Feature",
"features": []
},
]
}
Then you can define reader like this:
reader: {
type: 'json',
rootProperty: 'featureResponse'
}
once you get record from this store, you can go to data or raw children object to fetch required data.