I am loading a xml file into my postgresql / postgis database the following field contains cords for lat long of the area im wanting to convert to a polygon. Is there a way to use ST_MPolyFromText to make this work the samples have the long and lat reversed from the order than the way its loading it into the db?
39.43,-80.29 39.46,-80.49 39.43,-80.52 39.46,-80.66 39.98,-80.76 40.07,-80.43 39.46,-79.91 39.39,-80.10 39.40,-80.11 39.39,-80.18 39.43,-80.29
You can format this as WKT quite easily. First, translate your commas to spaces, and spaces to commas, then flip your coordinates from lat,long to long,lat (x,y). Here is a PostGIS function:
CREATE FUNCTION polygon_from_funny_format(latlon text) RETURNS geometry AS
$BODY$SELECT
ST_FlipCoordinates(('SRID=4326;POLYGON((' ||
translate($1, ', ', ' ,') || '))')::geometry);
$BODY$ LANGUAGE sql IMMUTABLE STRICT;
Now you can do:
SELECT polygon_from_funny_format('39.43,-80.29 39.46,-80.49 39.43,-80.52 39.46,-80.66 39.98,-80.76 40.07,-80.43 39.46,-79.91 39.39,-80.10 39.40,-80.11 39.39,-80.18 39.43,-80.29');
Related
I'm trying to convert a query from Redshift to Snowflake SQL.
The Redshift query looks like this:
SELECT
cr.creatives as creatives
, JSON_ARRAY_LENGTH(cr.creatives) as creatives_length
, JSON_EXTRACT_PATH_TEXT(JSON_EXTRACT_ARRAY_ELEMENT_TEXT (cr.creatives,0),'previewUrl') as preview_url
FROM campaign_revisions cr
The Snowflake query looks like this:
SELECT
cr.creatives as creatives
, ARRAY_SIZE(TO_ARRAY(ARRAY_CONSTRUCT(cr.creatives))) as creatives_length
, PARSE_JSON(PARSE_JSON(cr.creatives)[0]):previewUrl as preview_url
FROM campaign_revisions cr
It seems like JSON_EXTRACT_PATH_TEXT isn't converted correctly, as the Snowflake query results in error:
Error parsing JSON: more than one document in the input
cr.creatives is formatted like this:
"[{""previewUrl"":""https://someurl.com/preview1.png"",""device"":""desktop"",""splitId"":null,""splitType"":null},{""previewUrl"":""https://someurl.com/preview2.png"",""device"":""mobile"",""splitId"":null,""splitType"":null}]"
It seems to me that you are not working with valid JSON data inside Snowflake.
Please review your file format used for the copy into command.
If you open the "JSON" text provided in a text editor , note that the information is not parsed or formatted as JSON because of the quoting you have. Once your issue with double quotes / escaped quotes is handled, you should be able to make good progress
Proper JSON on Left || Original Data on Right
If you are not inclined to reload your data, see if you can create a Javascript User Defined Function to remove the quotes from your string, then you can use Snowflake to process the variant column.
The following code is working POJO that can be used to remove the doublequotes for you.
var textOriginal = '[{""previewUrl"":""https://someurl.com/preview1.png"",""device"":""desktop"",""splitId"":null,""splitType"":null},{""previewUrl"":""https://someurl.com/preview2.png"",""device"":""mobile"",""splitId"":null,""splitType"":null}]';
function parseText(input){
var a = input.replaceAll('""','\"');
a = JSON.parse(a);
return a;
}
x = parseText(textOriginal);
console.log(x);
For anyone else seeing this double double quote issue in JSON fields coming from CSV files in a Snowflake external stage (slightly different issue than the original question posted):
The issue is likely that you need to use the FIELD_OPTIONALLY_ENCLOSED_BY setting. Specifically, FIELD_OPTIONALLY_ENCLOSED_BY = '"' when setting up your fileformat.
(docs)
Example of creating such a file format:
create or replace file format mydb.myschema.my_tsv_file_format
type = CSV
field_delimiter = '\t'
FIELD_OPTIONALLY_ENCLOSED_BY = '"';
And example of querying from a stage using this file format:
select
$1 field_one
$2 field_two
-- ...and so on
from '#my_s3_stage/path/to/file/my_tab_separated_file.csv' (file_format => 'my_tsv_file_format')
I am trying to read some data into julia into a data frame to work with it. A minimal example of the .csv file could look like this:
A; B; C; D
ab; 1,23; 4; 9,2
ab; 3,4; 7; 1,1
ba; 6; 2,3; 8,6
I load the following to packages and read the data:
using DataFrames
using CSV
d = CSV.read( "test.csv", delim=";")
Julia recognizes the following types:
eltypes(d)
CategoricalArrays.CategoricalString{UInt32}
String
String
String
How could I now turn whole columns to floats with the comma replaced by a dot? My first idea was to use:
float(d[1,2])
But I did not find an option to tell julia to replace the comma with a dot.
My next idea was to first replace the comma and then convert it:
float(replace(d[1,2], ",", "."))
That works fine on a single cell but not on a whole column:
float(replace(d[:,2], ",", "."))
MethodError: no method matching
replace(::WeakRefStrings.WeakRefStringArray{WeakRefString{UInt8},1,Union{}},
::String, ::String)
I also tried:
d = CSV.read( "test.csv", delim=";", decimal=",")
which also just gives an error ...
Any ideas how to handle this problem and how to efficiently read the data into julia?
Thanks a lot!
Best regards.
One straightforward way is to read the file to string, replace the comma decimal separators by dots and then create the DataFrame from it:
s = replace(readstring("test.csv"), ",", ".")
CSV.read(IOBuffer(s); delim=';', types=[String, Float64, Float64, Float64])
Note that you can use the types keyword to specifiy the column types (it will then implicitly parse the string entries).
EDIT: According to this github issue the CSV.jl's read method supports a decimal keyword (from version v0.2.0 on) which allows you to do
CSV.read("test.csv"; delim=';', decimal=',', types=[String, Float64, Float64, Float64])
EDIT: Removed hint to alternatively use readtable from DataFrames.jl because it seems to be deprecated in favor of CSV.read.
Try to read this the json OV-fiets (http://fiets.openov.nl/locaties.json) in a postgres database with json_array_elements. Some names of train station contains the character ' .
Example ..... "description": "Helmond 't Hout"
I believe that my script fails because of the ' between Helmond and the t.
The script i use:
WITH data AS (SELECT 'paste the json from http://fiets.openov.nl/locaties.json'::json AS fc)
SELECT
row_number() OVER () AS gid,
feat->'locaties' AS locaties,
FROM (
SELECT json_array_elements(fc->'locaties') AS feat
FROM data
) AS f;*
++++++++++++++++++++++++++++++
The error i get:
*syntax error at or near "Hout"
LINE 3: ...Images": [], "name": "HMH - OV-fiets - Helmond 't Hout", "ex.*
How can i change the script to avoid the syntax error due to the character '
the easiest workaround here would probably be dollar quotes:
SELECT $dq$paste the json from http://fiets.openov.nl/locaties.json$dq$::json
In SQL, single quotes need to be escaped by doubling them, e.g.:
select 'Arthur''s house';
As an alternative (in Postgres) you can use dollar quoting to avoid changing the string:
SELECT $data$Arthur's house$data$
On a new project I work on I have data in CSV format to import into a mysql table. One of the columns is a price field which stores currency in the european format ie. 345,83.
The isssue I have is storing this decimal seperator. In most European currencies the decimal seperator is "," but when I try to insert a decimal number into a field (ex. 345,83), I get the following error: "Data truncated for column 'column_name' at row 'row #'". If I use '.' instead of ',' it works fine. Could you please help me with, how to store this format in mysql?
you can store it as a regular decimal field in the database, and format the number european style when you display it
edit: just added an example of how it might be achieved
$european_numbers = array('123.345,78', '123 456,78', ',78');
foreach($european_numbers as $number) {
echo "$number was converted to ".convert_european_to_decimal($number)."\n";
// save in database now
}
function convert_european_to_decimal($number) {
// i am sure there are better was of doing this, but this is nice and simple example
$number = str_replace('.', '', $number); // remove fullstop
$number = str_replace(' ', '', $number); // remove spaces
$number = str_replace(',', '.', $number); // change comma to fullstop
return $number;
}
Use number_format or money_format, it's pretty much what you preffer.
It's worse than you think. The number 1234.56 may be written in Europe as:
1234,56
1 234,56 (space as a group separator)
1.234,56 (dot as a group separator)
In .net the number parser can works according to a given culture, so if you know the format it does the hard work for you. I'm sure you can find a PHP equivalent, it'd save you a lot of trouble.
You could import the currency field into a VARCHAR column and then copy this column into a DECIMAL column while replacing the , by a . in all rows using MySQL string-manipulation-functions.
UPDATE <<table>>
SET <<decimal-currency-col>> = REPLACE(<<varchar-currency-col>>, ',', '.');
Some data types do not have a direct
correlation between SQL Server or
Access and MySQL. One example would be
the CURRENCY data type: MySQL does not
(yet) have a CURRENCY data type, but
creating a column with the definition
DECIMAL(19,4) serves the same purpose.
While MSSQL defaults to Unicode
character types such as nCHAR and
nVARCHAR, MySQL does not so tightly
bind character sets to field types,
instead allowing for one set of
character types which can be bound to
any number of character sets,
including Unicode.
from http://dev.mysql.com/tech-resources/articles/migrating-from-microsoft.html
You could also consider multiplying it by 100 and storing it as INT.
Before inserting the price to the DB:
$price = (int)$price*100;
After receiving price from the DB:
$price = number_format($price, 2, ',', ' ');
Try replacing the "," with "."?
$price = str_replace(",", ".", $price);
I would like to convert from EPSG:4326 to UTM (30N/EPSG:32630 or 29N/EPSG:32629) in PostGIS. I do the following query but I get wrong results:
SELECT ST_AsText(ST_Transform(ST_GeomFromText('POINT(36.5277099609375 -5.86424016952515)',4326),32630)) As check;
I get "POINT(5262418.33128724 -839958.963432011)" when it should be something approximate to 243625.00,4046330.00 in UTM 30N. If I do the conversion from 4326 to UTM I get the right result but not from UTM to 4326.
What's wrong with the query?
And are there anyway to get the UTM timezone from the coordinates in
EPSG:4326 because I don't know if they belong to 30N or 29N?
1) Your query is correct but you coordinates are inverted. The correct coordinates order in the WKT format is POINT(x y), also POINT(longitude latitude)
This query give you the expected result:
SELECT ST_AsText(ST_Transform(ST_GeomFromText('POINT(-5.86424016952515 36.5277099609375)',4326),32630)) As check;
2) To get the UTM zone from a lat/long geometry you can use this formula:
ST_X(input_geometry)+180)/6)+1
with some adjustments.
For this purpose we use this function:
CREATE OR REPLACE FUNCTION get_utmzone(input_geom geometry)
RETURNS integer AS
$BODY$
DECLARE
zone int;
pref int;
BEGIN
IF GeometryType(input_geom) != 'POINT' THEN
RAISE EXCEPTION 'Input geom must be a point. Currently is: %', GeometryType(input_geom);
END IF;
IF ST_Y(input_geom) >0 THEN
pref:=32600;
ELSE
pref:=32700;
END IF;
zone = floor((ST_X(input_geom)+180)/6)+1;
RETURN zone+pref;
END;
$BODY$
LANGUAGE plpgsql IMMUTABLE;
Use it with this query:
SELECT get_utmzone(ST_GeomFromText('POINT( -5.86424016952515 36.5277099609375)',4326));
The result should be 32630
Firs thing is that following documentation of OpenGIS WKT Point(x,y) yours POINT(36.5277099609375 -5.86424016952515) is south of equator so you have to use 29S(EPSG:32729) and 30S(EPSG:32730)
The reason is because is not POINT(36.5277099609375 -5.86424016952515), but POINT(-5.86424016952515 36.5277099609375)because longitude and latitude depends on the system. Normally X=longitude and Y=latitude, but e.g. in Google Maps X is the lat and Y is the long.