I build a software that generate trails for my own use
I would like to test the software so I create A CSV file that contain the longitude and latitude of the trail points
What is the format of a CSV file that can imported to Google maps
The documentation isn't very specific about CSV files, so I just tried a bunch of formats.
Option 1 is to have separate latitude and longitude columns. You will be able to specify columns in the upload wizard.
lon,lat,title
-20.0390625,53.27835301753182,something
-17.841796875,53.27835301753182,something
Option 2 is to have a single coordinate column with the coordinates separated by space. You will be able to chose the order of the coordinate pair in the upload wizard.
lonlat,title
-20.0390625 53.27835301753182,something
-17.841796875 53.27835301753182,something
You'll also need one column that acts as the description for your points, it is, again, selectable in the wizard.
There seems to be no way to import CSVs as line geometries and no way to convert points to lines later on. Well-known-text (WKT) in the coordinate column fails to import.
The separator needs to be comma ,. Semicolons ;, spaces and tabs don't work.
Related
I am loading a CSV into Apache Beam, but the CSV I am loading has commas in the fields. It looks like this:
ID, Name
1, Barack Obama
2, Barry, Bonds
How can I go about fixing this issue?
This is not specific to Beam, but a general problem with CSV. It's unclear if the second line should have ID="2, Barry" Name="Bonds" or the other way around.
If you can use some context (e.g. ID is always an integer, only one field could possibly contain commas) you could solve this by reading it as a text file line by line and parsing it into separate fields with a custom DoFn (assuming rows also contain newlines).
Generally, non-separating commas should be inside quotes in well-formed CSV, which makes this much more tractable (e.g. it would just work with the Beam Dataframes read_csv.)
I've a large TSV file that I'm viewing through MS Excel. I have a column showing arrays of coordinates. Unfortunately, the database from which the coords were taken did not include a comma separator between lat and long. The coords are formatted thus:
[[4.47917 51.9225],[-3.179559057 55.951719876],[-3.17055 55.9721],[-3.297777777 55.625],[-3.355388888 55.752611111]]
Whereas I need them formatted like:
[[4.47917, 51.9225],[-3.179559057, 55.951719876],[-3.17055, 55.9721],[-3.297777777, 55.625],[-3.355388888, 55.752611111]]
Is there a quick way that I can add these commas (the TSV has too many values for me to manually correct them).
I'm v much finding my way with this stuff so any help would be much appreciated.
Cheers
Yes, lucky you 😊 Ctrl+H to replace and replace with , . Replace space with coma followed by space.
The QGIS documentation for CSVT Files (https://docs.qgis.org/2.18/en/docs/user_manual/managing_data_source/supported_data.html#csvt-files)
explicitly states:
You can even specify width and precision of each column, e.g.: "Integer(6)","Real(5.5)","String(22)"
You can find more information at GDAL CSV Driver.
The GDAL link states that precision support was added in GDAL 2.0 and I have verified that my Windows QGIS install has : Running against GDAL/OGR 2.4.3
However, whatever I try, QGIS seems to ignore all precision and width values and creates fields with the default precision and width. I know that QGIS is using the CSVT file since my test file has integer values that would result in an integer field if the "Real" override in my CSVT was not used.
My test data file ';' delimiter and field headers is:
intval;intwidthval;rval;sval;WKT
9;1;5.5;Hello;LINESTRING(-13615773 6048754,-13615788 6048751,-13615804 6048756,-13615824 6048775)
My test csvt file is:
"Real","Integer(5)","Real","String","String"
Has anyone had any success using the precision and width from a csvt?
according to this answer
There are two options to load .csv files into QGIS:
1_ "Add delimited text layer" 2_ "Add vector layer"
If you are loading a .csv with a .csvt you should do it by using “Add vector layer”
I tried adding your data with "Add vector layer" and it gives me intwidthval with length 5
What is the best way to load the following geojson file in Google Big Query?
http://storage.googleapis.com/velibs/stations/test.json
I have a lot of json files like this (much bigger) on Google Storage, and I cannot download/modify/upload them all (it would take forever). Note that the file is not newline-delimited, so I guess it needs to be modified online.
Thanks all.
Step by step 2019:
If you get the error "Error while reading data, error message: JSON parsing error in row starting at position 0: Nested arrays not allowed.", you might have a GeoJSON file.
Transform GeoJSON into new-line delimited JSON with jq, load as CSV into BigQuery:
jq -c .features[] \
san_francisco_censustracts.json > sf_censustracts_201905.json
bq load --source_format=CSV \
--quote='' --field_delimiter='|' \
fh-bigquery:deleting.sf_censustracts_201905 \
sf_censustracts_201905.json row
Parse the loaded file in BigQuery:
CREATE OR REPLACE TABLE `fh-bigquery.uber_201905.sf_censustracts`
AS
SELECT FORMAT('%f,%f', ST_Y(centroid), ST_X(centroid)) lat_lon, *
FROM (
SELECT *, ST_CENTROID(geometry) centroid
FROM (
SELECT
CAST(JSON_EXTRACT_SCALAR(row, '$.properties.MOVEMENT_ID') AS INT64) movement_id
, JSON_EXTRACT_SCALAR(row, '$.properties.DISPLAY_NAME') display_name
, ST_GeogFromGeoJson(JSON_EXTRACT(row, '$.geometry')) geometry
FROM `fh-bigquery.deleting.sf_censustracts_201905`
)
)
Alternative approaches:
With ogr2ogr:
https://medium.com/google-cloud/how-to-load-geographic-data-like-zipcode-boundaries-into-bigquery-25e4be4391c8
https://medium.com/#mentin/loading-large-spatial-features-to-bigquery-geography-2f6ceb6796df
With Node.js:
https://github.com/mentin/geoscripts/blob/master/geojson2bq/geojson2bqjson.js
The bucket in the question no longer exists.... However five years later there is a new answer.
In July 2018, Google announced an alpha (now beta) of BigQuery GIS.
The docs highlight a limitation that
BigQuery GIS supports only individual geometry objects in GeoJSON.
BigQuery GIS does not currently support GeoJSON feature objects,
feature collections, or the GeoJSON file format.
This means that any Feature of Feature Collection properties would need to be added to separate columns, with a geography column to hold the geojson geography.
In this tutorial by a Google trainer, polygons in a shape file are converted into geojson strings inside rows of a CSV file using gdal.
ogr2ogr -f csv -dialect sqlite -sql "select AsGeoJSON(geometry) AS geom, * from LAYER_NAME" output.csv inputfilename.shp
You want to end up with one column with the geometry content like this
{"type":"Polygon","coordinates":[[....]]}
Other columns may contain feature properties.
The CSV can then be imported to BQ. Then a query on the table can be viewed in BigQuery Geo Viz. You need to tell it which column contains the geometry.
I wonder if there is any way to generate culture neutral CSV file or at least specify data format of certian columns present in file.
For example I generated CSV file that contains numbers with decimal separator (.), and after
pass it to the client which is in the country where decimal separator is (,), client opens it with Excel and sees all values changed.
Is there any way to resolve this isure, or just in this case do not use CSV file ?
Thank you in advance.
What you want is a "quoted CSV file".
That is as well as separating your values with commas you also enclose them in (usually) double quotes.
Like so:-
"first","second","3,00","Some other text, etc."
This format is quite common and supported by EXCEL.
Two ways I came up with to avoid the decimal separator altogether:
1) Use scientific notation, so 1.25 would be: 123E-2
2) Make it a formula, so 1.25 would be: =125/100
Both pretty crappy, depending on your target audience, but at least Excel sees them as numbers and can calculate with them.
A CSV file will be separated by commas (the 'C' in CSV) but you can output a text with any delimiter and qualifier and you'll be able to open it in Excel - you specify them in the step 2 of the import text wizard.
A common choice for situations like this is to use tabs (TSV).
You can use Tab Separated Values, which does not vary between cultures and are supported by Microsoft Excel. Common file extensions are .tsv and .tab.
http://en.wikipedia.org/wiki/Tab-separated_values