i need to find out what is the best method for what i'm looking.
Basically, based on user position (lat and lng coords.), i need to find their corresponding city..
my first attempt was gmaps (reverse geocoding):
Pros:
easy to use
Cons:
use is limited to N requests
according to their policies, using reverse geocoding without any visual in frontend is prohibited.
Each country has their own typography ( there is no standart field to obtain):
my second scenario, was mysql geospational:
After download gadm shape files, i tried to convert them para mysql using shp2mysql but no success..
Pros:
there is no need any external service to retrieve city
My Cons:
Unable to use them correctly.
So, am i on a right way? is second approach better than one? i only need cities/countries based on a user location, nothing more..
Any sugestion or maybe a good tutorial?
After hours:
CREATE TABLE geom (g GEOMETRY NOT NULL, SPATIAL INDEX(g)) ENGINE=MyISAM;
Next:
C:\Program Files\FWTools2.4.7>ogr2ogr -f MySQL MySQL:dbname,user=root
,password=root C:\shape_pt\PRT_adm1.shp -nln geom -update -overwrite -lco GEOMET
RY_NAME=g
ERROR 1: MySQL error message:The used table type doesn't support SPATIAL indexes
Description: ALTER TABLE `geom` ADD SPATIAL INDEX(`g`)
ERROR 1: Terminating translation prematurely after failed
translation of layer PRT_adm1 (use -skipfailures to skip errors)
Any suggestion?
Related
I am working on migration of 3.0 code into new 4.2 framework. I am facing a few difficulties:
How to do CDR level deduplication in new 4.2 framework? (Note: Table deduplication is already done).
Where to implement PostDedupProcessor - context or chainsink custom? In either case, do I need to remove duplicate hashcodes from the list or just reject the tuples? Here I am also doing column updating for a few tuples.
My file is not moving into archive. The temporary output file is getting generated and that too empty and outside load directory. What could be the possible reasons? - I have thoroughly checked config parameters and after putting logs, it seems correct output is being sent from transformer custom, so I don't know where it is stuck. I had printed TableRowGenerator stream for logs(end of DataProcessor).
1. and 2.:
You need to select the type of deduplication. It is not a big difference if you choose "table-" or "cdr-level-deduplication".
The ite.businessLogic.transformation.outputType does affect this. There is one Dedup only. You can not have both.
Select recordStream for "cdr-level-deduplication", do the transformation to table row format (e.g. if you like to use the TableFileWriter) in xxx.chainsink.custom::PostContextDataProcessor.
In xxx.chainsink.custom::PostContextDataProcessor you need to add custom code for duplicate-handling: reject (discard) tuples or set special column values or write them to different target tables.
3.:
Possibly reasons could be:
Missing forwarding of window punctuations or statistic tuple
error in BloomFilter configuration, you would see it easily because PE is down and error log gives hints about wrong sha2 functions be used
To troubleshoot your ITE application, I recommend to enable the following debug sinks if checking the StreamsStudio live graph is not sufficient:
ite.businessLogic.transformation.debug=on
ite.businessLogic.group.debug=on
ite.businessLogic.sink.debug=on
Run a test with a single input file only and check the flow of your record and statistic tuples. "Debug sinks" write punctuations markers also to debug files.
I have a very specific requirement where some columns need to be encrypted using aes_encrypt / aes_decrypt. We need to encrypt the information at SQL level using a eas so it can be read using another app or directly from MySQL using a query and aes_encrypt / aes_decrypt.
Our app was developed using CakePHP 3 and database is MySQL 5.6.25.
I found and carefully follow the instruction on this selected answer: Encyption/Decryption of Form Fields in CakePHP 3
Now the data is being saved encrypted on the database... the problem is that we still need to be able to use aes_decrypt on MySQL to decrypt the information and it's returning NULL.
On CakePHP 3, config/app.php:
'Security' => ['salt' => '1234567890']
Then encrypted using:
Security::encrypt($value, Security::salt());
Data is saved on MySQL but aes_decrypt() returns NULL
SELECT AES_DECRIPT(address_enc, '1234567890') FROM address;
How can I setup CakePHP 3 to correctly encrypt information so I can later decrypt it on MySQL using aes_decrypt() ?
[EDIT]
My MYSQL table:
CREATE TABLE IF NOT EXISTS `address` (
`id` int(11) NOT NULL,
`address` varchar(255) DEFAULT NULL,
`address_enc` blob,
`comment` varchar(255) DEFAULT NULL,
`comment_enc` blob
) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
Note: address and comment are just for testings.
Then, on CakePHP, I created a custom database type:
src/Database/Type/CryptedType.php
<?php
namespace App\Database\Type;
use Cake\Database\Driver;
use Cake\Database\Type;
use Cake\Utility\Security;
class CryptedType extends Type
{
public function toDatabase($value, Driver $driver)
{
return Security::encrypt($value, Security::salt());
}
public function toPHP($value, Driver $driver)
{
if ($value === null) {
return null;
}
return Security::decrypt($value, Security::salt());
}
}
src/config/bootstrap.php
Register the custom type.
use Cake\Database\Type;
Type::map('crypted', 'App\Database\Type\CryptedType');
src/Model/Table/AddressTable.php
Finally map the cryptable columns to the registered type, and that's it, from now on everything's being handled automatically.
use Cake\Database\Schema\Table as Schema;
class AddressTable extends Table
{
// ...
protected function _initializeSchema(Schema $table)
{
$table->columnType('address_enc', 'crypted');
$table->columnType('comment_enc', 'crypted');
return $table;
}
// ...
}
Do you really need to do that?
I'm not going to argue about the pros and cons of storing encrypted data in databases, but whether trying to decrypt on SQL level is a good idea, is a question that should be asked.
So ask yourself whether you really need to do that, maybe it would be better to implement the decryption at application level instead, it would probably make things easier with regards to replicating exactly what Security::decrypt() does, which is not only decrypting, but also integrity checking.
Just take a look at what Security::decrypt() does internally.
https://github.com/cakephp/cakephp/blob/3.1.7/src/Utility/Security.php#L201
https://github.com/cakephp/cakephp/blob/3.1.7/src/Utility/Crypto/OpenSsl.php#L77
https://github.com/cakephp/cakephp/blob/3.1.7/src/Utility/Crypto/Mcrypt.php#L89
It should be pretty easy to re-implement that in your other application.
Watch out, you may be about to burn your fingers!
I am by no means an encryption expert, so consider the following as just a basic example to get things started, and inform yourself about possible conceptual, and security related problems in particular!
Handling encryption/decryption of data without knowing exactly what you are doing, is a very bad idea - I can't stress that enough!
Decrypting data at SQL level
That being said, using the example code from my awful (sic) answer that you've linked to, ie using Security::encrypt(), and Security::salt() as the encryption key, will by default leave you with a value that has been encrypted in AES-256-CBC mode, using an encryption key derived from the salt concatenated with itself (first 32 bytes of its SHA256 representation).
But that's not all, additionally the encrypted value gets an HMAC hash, and the initialization vector pepended, so that you do not end up with "plain" encrypted data that you could directly pass to AES_DECRYPT().
So if you'd wanted to decrypt this on MySQL level (for whatever reason), then you'd first of all have to set the proper block encryption mode
SET block_encryption_mode = 'aes-256-cbc';
sparse out the HMAC hash (first 64 bytes) and the initialization vector (following 16 bytes)
SUBSTRING(`column` FROM 81)
and use the first 32 bytes of hash('sha256', Security::salt() . Security::salt()) as the encryption key, and the initialization vector from the encrypted value for decryption
SUBSTRING(`column`, 65, 16)
So in the end you'd be left with something like
SET block_encryption_mode = 'aes-256-cbc';
SELECT
AES_DECRYPT(
SUBSTRING(`column` FROM 81), -- the actual encryted data
'the-encryption-key-goes-here',
SUBSTRING(`column`, 65, 16) -- the intialization vector
)
FROM table;
Finally you maybe also want to cast the value (CAST(AES_DECRYPT(...) AS CHAR)), and remove possible zero padding (not sure whether AES_DECRYPT() does that automatically).
Data integrity checks
It should be noted that the HMAC hash that is prepended to the encrypted value, has a specific purpose, it is used to ensure integrity, so by just dropping it, you'll lose that. In order to keep it, you'd have to implement a (timing attack safe) HMAC256 generation/comparison on SQL level too. This leads us back to the intial question, do you really need to decrypt on SQL level?
[Solution] The solution for this particular requirement (we need to encrypt the information at SQL level using a eas so it can be read using another app or directly from MySQL using a query and aes_encrypt / aes_decryp) was to create a custom database type in CakePHP them, instead of using CakePHP encryption method, we implemented PHP Mcrypt.
Now the information is saved to the database from our CakePHP 3 app and the data be read at MySQL/phpMyAdmin level using eas_decrypt and aes_encrypt.
FOR ANYONE STRUGGLING TO DECRYPT WITH MYSQL: This generally applies to anyone using symmetric AES encryption/decryption - specifically when trying to decrypt with AES_DECRYPT.
For instance, if you are using aes-128-ecb, and your encrypted data is 16 bytes long with no padding, you need to add padding bytes to your encrypted data before trying to decrypt (because mySQL is expecting PKCS7 padding). Because MySQL uses PKCS7, you need to add 16 more bytes, in this case those pad bytes are 0x10101010101010101010101010101010. We take the left 16 bytes because when we encrypt the 0x10101010101010101010101010101010, we get 32 bytes, and we only need the first 16.
aes_decrypt(concat(<ENCRYPTED_BYTES>, left(aes_encrypt(<PAD BYTES>, <KEY>), 16)), <KEY>)
I'm working on google map search functionality. The vision of this is to find whether a (geolocation) point is located within a polygon or not as shown in the illustration below?
I'm using mysql 5.6.20 with Spatial extension, I know it has useful geometry functions built-in, so I will be allowed to query geocoded locations from a database directly.
My aim was to familiarize myself with geospatial functions, so I wrote up an experimental sql.
given a point geolocation: POINT(100.52438735961914 13.748889613522605)
and a polygon or boundary to be tested with was:
POLYGON(100.49503326416016 13.766897133254545,100.55940628051758 13.746555203977,100.56266784667969 13.72170897580617,100.48885345458984 13.739051587150175)
here is my sql example:
SELECT ST_Within(
ST_GEOMFROMTEXT('POINT(100.52438735961914 13.748889613522605)'),
ST_GEOMFROMTEXT('POLYGON(100.49503326416016 13.766897133254545,
100.55940628051758 13.746555203977,100.56266784667969 13.72170897580617,
100.48885345458984 13.739051587150175)'))
As geoFenceStatus
but after issuing the command, what I got in return seemed to be as follows:
geoFenceStatus
===============
null
I'm so unsure why it returned me 'null' value. since it was indicated in function documentation that this should return '1' in case a point resides within a given polygon
any advice would be appreciated, how to get my sql right.
The error message isn't very clear but your issue is that you have an invalid Polygon. There are two problems with it:
You have to repeat the first point at the end -- this is to differentiate it from a LINESTRING, essentially, ie, it is closed.
POLYGONS start with and end with double parenthesis, ie, POLYGON((x1 y1, x2 y2.......xn yn, x1 y1)). This is to allow for inner rings to be delineated with single parenthesis sets inside the polygon.
See the wikipedia WKT article for more information.
You should find that if you write you query as:
SELECT ST_Within(ST_GEOMFROMTEXT('POINT(100.52438735961914 13.748889613522605)'),
ST_GEOMFROMTEXT('POLYGON((100.49503326416016 13.766897133254545, 100.55940628051758 13.746555203977,100.56266784667969 13.72170897580617, 100.48885345458984 13.739051587150175,
100.49503326416016 13.766897133254545))'))
As geoFenceStatus
then is should work.
I'm trying to create a procedure which extracts data from a MySQL server (using the RODBC package), performs some statistical routines on that data in R, then saves generated plots back to the server such that they can be retrieved in a Web Browser via a little bit of php and web magic.
My plan is to save the plot in a MySQL BLOB field by using the RODBC package to execute a SQL insert into statement. I think I can insert the data directly as a string. Problem is, how do I get the data string and will this even work? My best thought is to use the savePlot function to save a temp file and then read it back in somehow.
Anybody tried this before or have suggestions on how to approach this?
Regardless of if you think this is a terrible idea, here is a working answer I was able to piece together from this post
## open connection
library(RODBC)
channel <- odbcConnect("")
## generate a plot and save it to a temp file
x <- rnorm(100,0,1)
hist(x, col="light blue")
savePlot("temp.jpg", type="jpeg")
## read back in the temp file as binary
plot_binary <- paste(readBin("temp.jpg", what="raw", n=1e6), collapse="")
## insert it into a table
sqlQuery(channel, paste("insert into test values (1, x'",plot_binary,"')", sep=""))
## close connection
odbcClose(channel)
Before implementation, I'll make sure to do some soul searching to decide if this should be used rather than using the servers file system.
Storing images in databases is often frowned upon. To create an in memory file in R you can use a textConnection as a connection. This will give you the string. It will work if you don't forget to set the proper mime type and open the connection as binary.
Save the plot to a server and write the filename into the database will work. But there's this thing called Rapache may help. Plus, Jeroen Ooms has some online demo, including a web interface for Hadley Wickham's famous R Graph package ggplot2.
I am trying to put a SHP file into my PostGIS database, the the data is just a little off. I think this is because I am using the wrong SRID. The contents of the PRJ file are as follows:
GEOGCS["GCS_North_American_1983",
DATUM["D_North_American_1983",
SPHEROID["GRS_1980",6378137.0,298.257222101]],
PRIMEM["Greenwich",0.0],
UNIT["Degree",0.0174532925199433]]
What SRID does this correlate to? And more generally, how can I look up the SRID based on the information found in the PRJ file? Is there a lookup table somewhere that lists all SRID's and their 'geogcs' equivalents?
The data imported using srid=4269 and 4326 were the exact same results.
Does this mean I'm using the wrong SRID, or is this just expected margin of error?
The shp file is from here.
To elaborate on synecdoche's answer, the SRID is sometimes called an "EPSG" code. The SRID/EPSG code is a defacto short-hand for the Well-Known-Text representations of projections.
You can do a quick search on the SRID table to see if you can find an exact or similar match:
SELECT srid, srtext, proj4text FROM spatial_ref_sys WHERE srtext ILIKE '%BLAH%'
Above was found at http://www.bostongis.com/?content_name=postgis_tut01.
You can also search on spatialreference.org for these kinds of things. The search tool is primitive so you may have to use a Google search and specify the site, but any results will show you the ESRI PRJ contents, the PostGIS SQL INSERT, and a bunch of other representations.
I think your PRJ is at: http://spatialreference.org/ref/sr-org/15/
Prj2EPSG is a small website aimed at exactly this problem; paste in the PRJ contents and it does its best to find a matching EPSG. They also have a web service API. It's not an exact science. They seem to use Lucene and the EPSG database to do text searches for matches.
The data seems to be NAD83, which has an SRID of 4269. Your PostGIS database has a spatial_ref_sys table which is the SRID lookup table.
If the data looks the same with an SRID of 4269 (NAD83) and 4326 (WGS84), then there's something wrong.
Go and download the GDAL utilities , the ogrinfo (which would spit the projection information) and ogr2ogr utilities are invaluable.
James gave already a link to spatialreference.org. That helps to find spatial reference information... I assume you did load the spatial_ref_sys.sql when you prepared your postgis instance.
And to be honest, I don't think the problem is in the PostGIS side of things.
I usually keep my data in different SRIDs in my PostGIS dbs. However, I always need to project to the output SRS. You are showing OpenStreetMap pre-rendered tiles, and I bet they have been drawn using SRID 900913 (the Google Map's modified mercator projection that now everyone uses to render).
My recommendation to you is:
1- Set the right projection in the OpenLayers code which matches whatever tiles you are reading from.
2.- Keep the data in the database in whatever SRID you want (as long as it is correct of course).
3.- Make sure the server you are using to generate the images from your data (ArcGIS Server, Mapserver, GeoServer or whatever it is) is reprojecting to that same SRS.
Everything will match.
Cheers
Use GDAL's OSR Python module to determine the code:
from osgeo import osr
srsWkt = '''GEOGCS["GCS_North_American_1983",
DATUM["D_North_American_1983",
SPHEROID["GRS_1980",6378137.0,298.257222101]],
PRIMEM["Greenwich",0.0],
UNIT["Degree",0.0174532925199433]]'''
# Load in the projection WKT
sr = osr.SpatialReference(srsWkt)
# Try to determine the EPSG/SRID code
res = sr.AutoIdentifyEPSG()
if res == 0: # success
print('SRID=' + sr.GetAuthorityCode(None))
# SRID=4269
else:
print('Could not determine SRID')
Be sure to take a look at: http://www.epsg-registry.org/
Use the Query by Filter option and enter: North American Datum 1983.
This yields -> EPSG:6269.
Hope this works for you.