I'm trying to generate MBTiles file using tippecanoe and to do that first I am converting a shape file downloaded from Geofabric downloads to a GeoJSON file using ogr2ogr and then using that GeoJSON file to MBTiles file using tippecanoe.
The step I'm stuck on is this
ogr2ogr -f GeoJSON turk.geojson turkmenistan-latest-free.shp
The file which I downloaded from Geofabric is a .shp.zip file - https://download.geofabrik.de/asia/turkmenistan-latest-free.shp.zip. On extracting this zip file I get a folder which contains various files like gis_osm_buildings_a_free_1.cpg etc. However, there was no turkmenistan-latest-free.shp file - or one .shp file which contained all of Turkmenistan.
When I run ogr2ogr command directly on the folder which was extracted out of the .zip file, I get this error
ERROR 1: Layer 'gis_osm_landuse_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Terminating translation prematurely after failed
translation of layer gis_osm_landuse_a_free_1 (use -skipfailures to skip errors)
If I run the same command with -skipfailures, I get more errors
ERROR 1: Layer 'gis_osm_landuse_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_natural_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_natural_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_places_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_places_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_pofw_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_pofw_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_pois_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_pois_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_railways_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_roads_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_traffic_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_traffic_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_transport_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_transport_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_water_a_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
ERROR 1: Layer 'gis_osm_waterways_free_1' does not already exist in the output dataset, and cannot be created by the output driver.
However, on inspecting the folder all these files already exist in it, but ogr2ogr is somehow unable to find it. These are the files in the folder
README gis_osm_natural_free_1.shp gis_osm_pofw_free_1.prj gis_osm_roads_free_1.dbf gis_osm_transport_free_1.cpg
gis_osm_buildings_a_free_1.cpg gis_osm_natural_free_1.shx gis_osm_pofw_free_1.shp gis_osm_roads_free_1.prj gis_osm_transport_free_1.dbf
gis_osm_buildings_a_free_1.dbf gis_osm_places_a_free_1.cpg gis_osm_pofw_free_1.shx gis_osm_roads_free_1.shp gis_osm_transport_free_1.prj
gis_osm_buildings_a_free_1.prj gis_osm_places_a_free_1.dbf gis_osm_pois_a_free_1.cpg gis_osm_roads_free_1.shx gis_osm_transport_free_1.shp
gis_osm_buildings_a_free_1.shp gis_osm_places_a_free_1.prj gis_osm_pois_a_free_1.dbf gis_osm_traffic_a_free_1.cpg gis_osm_transport_free_1.shx
gis_osm_buildings_a_free_1.shx gis_osm_places_a_free_1.shp gis_osm_pois_a_free_1.prj gis_osm_traffic_a_free_1.dbf gis_osm_water_a_free_1.cpg
gis_osm_landuse_a_free_1.cpg gis_osm_places_a_free_1.shx gis_osm_pois_a_free_1.shp gis_osm_traffic_a_free_1.prj gis_osm_water_a_free_1.dbf
gis_osm_landuse_a_free_1.dbf gis_osm_places_free_1.cpg gis_osm_pois_a_free_1.shx gis_osm_traffic_a_free_1.shp gis_osm_water_a_free_1.prj
gis_osm_landuse_a_free_1.prj gis_osm_places_free_1.dbf gis_osm_pois_free_1.cpg gis_osm_traffic_a_free_1.shx gis_osm_water_a_free_1.shp
gis_osm_landuse_a_free_1.shp gis_osm_places_free_1.prj gis_osm_pois_free_1.dbf gis_osm_traffic_free_1.cpg gis_osm_water_a_free_1.shx
gis_osm_landuse_a_free_1.shx gis_osm_places_free_1.shp gis_osm_pois_free_1.prj gis_osm_traffic_free_1.dbf gis_osm_waterways_free_1.cpg
gis_osm_natural_a_free_1.cpg gis_osm_places_free_1.shx gis_osm_pois_free_1.shp gis_osm_traffic_free_1.prj gis_osm_waterways_free_1.dbf
gis_osm_natural_a_free_1.dbf gis_osm_pofw_a_free_1.cpg gis_osm_pois_free_1.shx gis_osm_traffic_free_1.shp gis_osm_waterways_free_1.prj
gis_osm_natural_a_free_1.prj gis_osm_pofw_a_free_1.dbf gis_osm_railways_free_1.cpg gis_osm_traffic_free_1.shx gis_osm_waterways_free_1.shp
gis_osm_natural_a_free_1.shp gis_osm_pofw_a_free_1.prj gis_osm_railways_free_1.dbf gis_osm_transport_a_free_1.cpg gis_osm_waterways_free_1.shx
gis_osm_natural_a_free_1.shx gis_osm_pofw_a_free_1.shp gis_osm_railways_free_1.prj gis_osm_transport_a_free_1.dbf
gis_osm_natural_free_1.cpg gis_osm_pofw_a_free_1.shx gis_osm_railways_free_1.shp gis_osm_transport_a_free_1.prj
gis_osm_natural_free_1.dbf gis_osm_pofw_free_1.cpg gis_osm_railways_free_1.shx gis_osm_transport_a_free_1.shp
gis_osm_natural_free_1.prj gis_osm_pofw_free_1.dbf gis_osm_roads_free_1.cpg gis_osm_transport_a_free_1.shx
What am I doing incorrectly here and how can I properly generate the GeoJSON file?
First, there is no turkmenistan-latest-free.shp shape file, so you can't convert it to GeoJSON.
You can convert either of the shapefiles in your directory:
gis_osm_buildings_a_free_1.shp
gis_osm_landuse_a_free_1.shp
gis_osm_natural_a_free_1.shp
gis_osm_natural_free_1.shp
gis_osm_places_a_free_1.shp
gis_osm_places_free_1.shp
gis_osm_pofw_a_free_1.shp
gis_osm_pofw_free_1.shp
gis_osm_pois_a_free_1.shp
gis_osm_pois_free_1.shp
gis_osm_railways_free_1.shp
gis_osm_roads_free_1.shp
gis_osm_traffic_a_free_1.shp
gis_osm_traffic_free_1.shp
gis_osm_transport_a_free_1.shp
gis_osm_transport_free_1.shp
gis_osm_water_a_free_1.shp
gis_osm_waterways_free_1.shp
by just running
ogr2ogr output.json input.shp
like for example:
ogr2ogr gis_osm_buildings_a_free_1.json gis_osm_buildings_a_free_1.shp
You can check if the files (input or output) are valid with ogrinfo:
ogrinfo gis_osm_buildings_a_free_1.json
# INFO: Open of `gis_osm_buildings_a_free_1.json'
# using driver `GeoJSON' successful.
# 1: gis_osm_buildings_a_free_1
Related
I use Stata 12.
I want to add some country code identifiers from file df_all_cities.csv onto my working data.
However, this line of code:
merge 1:1 city country using "df_all_cities.csv", nogen keep(1 3)
Gives me the error:
. run "/var/folders/jg/k6r503pd64bf15kcf394w5mr0000gn/T//SD44694.000000"
file df_all_cities.csv not Stata format
r(610);
This is an attempted solution to my previous problem of the file being a dta file not working on this version of Stata, so I used R to convert it to .csv, but that also doesn't work. I assume it's because the command itself "using" doesn't work with csv files, but how would I write it instead?
Your intuition is right. The command merge cannot read a .csv file directly. (using is technically not a command here, it is a common syntax tag indicating a file path follows.)
You need to read the .csv file with the command insheet. You can use it like this.
* Preserve saves a snapshot of your data which is brought back at "restore"
preserve
* Read the csv file. clear can safely be used as data is preserved
insheet using "df_all_cities.csv", clear
* Create a tempfile where the data can be saved in .dta format
tempfile country_codes
save `country_codes'
* Bring back into working memory the snapshot saved at "preserve"
restore
* Merge your country codes from the tempfile to the data now back in working memory
merge 1:1 city country using `country_codes', nogen keep(1 3)
See how insheet is also using using and this command accepts .csv files.
Here is the situation - jmeter results were recorded in .csv quite a long time ago (around 6 month). The version of jmeter was not changed (3.0) but the config files did. Now I've been trying to generate a report from the old csv as usual - using
jmeter.bat -g my.csv -o reportFolder
Also, to defeat the incompatibility of configurations, I created a file named local-saveservice.properties, and passed it through -q command line option. Playing with settings in this file, I managed to defeat several errors like "column number mismatch" or "No column xxx found in sample metadata", but I still didn't generate the report succesfully, and here is the trouble:
File 'D:\ .....\load_NSI_stepping3_2017-03-24-new.csv' does not contain the field names h
eader, ensure the jmeter.save.saveservice.* properties are the same as when the CSV file was created or the file may be
read incorrectly
An error occurred: Error while processing samples:Consumer failed with message :Consumer failed with message :Consumer f
ailed with message :Consumer failed with message :Error in sample at line:1 converting field:Latency at column:11 to:lon
g, fieldValue:'UTF-8'
However,in my .csv column number 11 has the header "Latency" and contains numeric values, though 'UTF-8' is the content of next column - "Encoding"
Here are first lines of my .csv
timeStamp,elapsed,label,responseCode,responseMessage,success,bytes,grpThreads,allThreads,URL,Latency,Encoding,SampleCount,ErrorCount,Connect,threadName
1490364040950,665,searchItemsInCatalogRequest,200,OK,true,25457,1,1,http://*.*.*.*:9080/em/.....Service,654,UTF-8,1,0,9,NSI - search item in catalog
1490364041620,507,searchItemsInCatalogRequest,200,OK,true,25318,1,1,http://*.*.*.*:9080/em/.....Service,499,UTF-8,1,0,0,NSI - search item in catalog
1490364042134,495,searchItemsInCatalogRequest,200,OK,true,24266,2,2,http://*.*.*.*:9080/em/.....Service,487,UTF-8,1,0,0,NSI - search item in catalog
1490364043595,563,searchItemsInCatalogRequest,200,OK,true,24266,2,2,http://*.*.*.*:9080/em/.....Service,556,UTF-8,1,0,6,NSI - search item in catalog
PS I had to add threadName manually, 'cos it was not saved during initial data recording (my knowledge of Jmeter was even less then now :) )
First you should update to JMeter 3.3 as there are bugs fixed in report generation in the 3 versions after 3.0.
Second add to your command line:
jmeter.bat -p <path to jmeter.properties> -q <path to your custom.properties used when you generated file> -g my.csv -o reportFolder
Ensure that in "your custom.properties" you set to false all fields that are prefixed by jmeter.save.saveservice that didn't yet exist at the time you generated the file.
I have a CSV data file with a header row that I am using to populate a BigQuery table:
$ cat dummy.csv
Field1,Field2,Field3,Field4
10.5,20.5,30.5,40.5
10.6,20.6,30.6,40.6
10.7,20.7,30.7,40.7
When using the Web UI, there is a text box where I am able to specify how many header rows to skip. However, if I upload the data into BigQuery using the bq command line tool, I do not have an option to do this, and always get the following error:
$ bq load my-project:my-dataset.dummydata dummy.csv Field1:float,Field2:float,Field3:float,Field4:float
Upload complete.
Waiting on bqjob_r7eccfe35f_0000015e3e8c_1 ... (0s) Current status: DONE
BigQuery error in load operation: Error processing job 'my-project:bqjob_r7eccfe35f_0000015e3e8c_1': CSV table encountered too many errors, giving up. Rows: 1;
errors: 1.
Failure details:
- file-00000000: Could not parse 'Field1' as double for field Field1
(position 0) starting at location 0
The bq command line tool quickstart documentation also does not mention any options for skipping headers.
One simple/obvious solution is to edit dummy.csv to remove the header row, but this is not an option if pointing to a CSV file on Google Cloud Storage instead of the local file dummy.csv.
This is possible to do through the web interface, and through the Python API, so it should also be possible to do with the bq tool.
Checking bq help load revealed a --skip_leading_rows option:
--skip_leading_rows : The number of rows at the beginning of the source file to skip.
(an integer)
Also found this option in the bq command line tool documentation (which is not the same as the quickstart documentation, linked to above).
Adding a --skip_leading_rows=1 to the bq load command worked like a charm.
Here is the successful command:
$ bq load --skip_leading_rows=1 my-project:my-dataset.dummydata dummy.csv Field1:float,Field2:float,Field3:float,Field4:float
Upload complete.
Waiting on bqjob_r43eb07bad58_0000015ecea_1 ... (0s) Current status: DONE
I am trying to load schema and data from GCS as JSON files. I am using command line for this purpose.
bq load --source_format=NEWLINE_DELIMITED_JSON --schema=gs://1samtest/JSONSample/personsDataSchema.json SSData.persons_data gs://1samtest/JSONSample/personsData.json
But I get this error:
//1SAMTEST/JSONSAMPLE/PERSONSDATASCHEMA.JSON is not a valid value
But when I change all paths to my local machine it works completely file. But don't know why its getting error for json.
If I run like below after creating table in BigQuery it works fine.
bq load --source_format=NEWLINE_DELIMITED_JSON SSData.persons_data "gs://1samtest/JSONSample/personsData.json"
The schema flag/param doesn't support URIs for GCS i.e. using gs://...
bq load --help
The [destination_table] is the fully-qualified table name of table to
create, or append to if the table already exists.
The [source] argument can be a path to a single local file, or a
comma-separated list of URIs.
The [schema] argument should be either the name of a JSON file or a text schema. This schema should be omitted if the table already has one.
Only the source flag/param (i.e. the data) can be used with GCS URIs.
I wanted to try out the encrypted big query client for google big query and I've been having some trouble.
I'm following the instructions outlined in this PDF:
https://docs.google.com/file/d/0B-WB8hYCrhZ6cmxfWFpBci1lOVE/edit
I get to the point where I'm running this command:
ebq load --master_key_filename="key_file" testdataset.cars cars.csv cars.schema
And I'm getting an error string which ends with:
raise ValueError("No JSON object could be decoded")
I've tried a few different formats for my .csv and .schema files but none have worked. Here are my latest versions.
cars.schema:
[{"name": "Year", "type": "integer", "mode": "required", "encrypt": "none"}
{"name": "Make", "type": "string", "mode": "required", "encrypt": "pseudonym"}
{"name": "Model", "type": "string", "mode": "required", "encrypt": "probabilistic_searchwords"}
{"name": "Description", "type": "string", "mode": "nullable", "encrypt": "searchwords"}
{"name": "Website", "type": "string", "mode": "nullable", "encrypt": "searchwords","searchwords_separator": "/"}
{"name": "Price", "type": "float", "mode": "required", "encrypt": "probabilistic"}
{"name": "Invoice_Price", "type": "integer", "mode": "required", "encrypt": "homomorphic"}
{"name": "Holdback_Percentage", "type": "float", "mode": "required", "encrypt":"homomorphic"}]
cars.csv:
1997,Ford,E350, "ac\xc4a\x87, abs, moon","www.ford.com",3000.00,2000,1.2
1999,Chevy,"Venture ""Extended Edition""","","www.cheverolet.com",4900.00,3800,2.3
1999,Chevy,"Venture ""Extended Edition, Very Large""","","www.chevrolet.com",5000.00,4300,1.9
1996,Jeep,Grand Cherokee,"MUST SELL! air, moon roof,loaded","www.chrysler.com/jeep/grandcherokee",4799.00,3950,2.4
I believe the issue may be that you need to move the --master_key_filename argument before the load argument. If that doesn't work, can you send the output of adding --apilog=- as the first argument?
Also, there is an example script file of running ebq here:
https://code.google.com/p/bigquery-e2e/source/browse/#git%2Fsamples%2Fch13