I have a mysql spatial database ('mydb1') of one table ('pk'), that i obtained from a point shapefile using ogr2ogr tool. one row of this 'pk' table, is like:
OGR_FID SHAPE CODIF Position_X Position_Y Nom Status
1 [GEOMETRY-25o] PAC182854 398235.38 414569.24 G-31 Vert
I need to output the geoJSON file of this table. I downloaded geoPHP library and used MySQL to GeoJSON script which i adapted to my settings, like below:
<?php
/**
* Title: MySQL to GeoJSON (Requires https://github.com/phayes/geoPHP)
* Notes: Query a MySQL table or view and return the results in GeoJSON format, suitable for use in OpenLayers, Leaflet, etc.
* Author: Bryan R. McBride, GISP
* Contact: bryanmcbride.com
* GitHub: https://github.com/bmcbride/PHP-Database-GeoJSON
*/
# Include required geoPHP library and define wkb_to_json function
include_once('geoPHP/geoPHP.inc');
function wkb_to_json($wkb) {
$geom = geoPHP::load($wkb,'wkb');
return $geom->out('json');
}
# Connect to MySQL database
$conn = new PDO('mysql:host=localhost;dbname=mydb1','root','admin');
# Build SQL SELECT statement and return the geometry as a WKB element
$sql = 'SELECT *, AsWKB(SHAPE) AS wkb FROM pk';
# Try query or error
$rs = $conn->query($sql);
if (!$rs) {
echo 'An SQL error occured.\n';
exit;
}
# Build GeoJSON feature collection array
$geojson = array(
'type' => 'FeatureCollection',
'features' => array()
);
# Loop through rows to build feature arrays
while ($row = $rs->fetch(PDO::FETCH_ASSOC)) {
$properties = $row;
# Remove wkb and geometry fields from properties
unset($properties['wkb']);
unset($properties['SHAPE']);
$feature = array(
'type' => 'Feature',
'geometry' => json_decode(wkb_to_json($row['wkb'])),
'properties' => $properties
);
# Add feature arrays to feature collection array
array_push($geojson['features'], $feature);
}
header('Content-type: application/json');
echo json_encode($geojson, JSON_NUMERIC_CHECK);
$conn = NULL;
?>
But when i execute the code on the browser, i get totally a blank page. What's wrong should i fix to output my geoJSON file plz?
You should check the php logs for more information in case an error ocured. Make sure error reporting and display are enabled:
error_reporting(E_ALL);
ini_set('display_errors', 1);
if no error show up inspect the $geojson object easiest would be using
print_r($geojson);
Related
I am trying to import a file into a mariadb (mysql), database. A proof of concept file is in .json format on the web at this location. https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson
I know how to do this on db2 for i.
select * from JSON_TABLE(
SYSTOOLS.HTTPGETCLOB('https://earthquake.usgs.gov' ||
'/earthquakes/feed/v1.0/summary/all_week.geojson',null),
'$.features[*]'
COLUMNS( MILLISEC BIGINT PATH '$.properties.time',
MAG DOUBLE PATH '$.properties.mag',
PLACE VARCHAR(100) PATH '$.properties.place'
)) AS X;
This reads a from the web and lists 3 fields. Surrounding it with an insert clause will put it into a database file for me.
I would like to do exactly the same thing on my home server using mariadb. Ultimately a script will run unattended on a hosted server.
A .json segment of the earthquake data looks like this:
… "features":[
{"type":"Feature",
"properties":{
"mag":1.1,
"place":"58 km WNW of Anchor Point, Alaska",
"time":1640472257402,
"updated":1640472615410,
"tz":null,
"url":"https://earthquake.usgs.gov/earthquakes/eventpage/ak021gi3al5x",
"detail":"https://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/ak021gi3al5x.geojson",
"felt":null,
"cdi":null,
"mmi":null,
"alert":null,
"status":"automatic",
"tsunami":0,
"sig":19,
"net":"ak",
"code":"021gi3al5x",
"ids":",ak021gi3al5x,",
"sources":",ak,",
"types":",origin,",
"nst":null,
"dmin":null,
"rms":0.79,
"gap":null,
"magType":"ml",
"type":"earthquake",
"title":"M 1.1 - 58 km WNW of Anchor Point, Alaska"},
"geometry":{
"type":"Point",
"coordinates":[-152.8406,59.9119,89.7]
},
"id":"ak021gi3al5x"
}, ...
Just a thought, but you could maybe just write a BASH script, e.g.
#!/bin/sh
cd "$(dirname "$0")"
export PATH=/bin:/usr/bin:/usr/local/bin
TODAY=`date +"%d%b%Y-%H%M"`
MYSQL_HOST='127.0.0.1'
MYSQL_PORT='3306'
MYSQL_USER='root'
MYSQL_PASSWORD='root'
url=https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson
DATA=$(curl ${url} 2>/dev/null)
printf '%s' "$DATA" | awk '{print $0}'
exit
...
...
...
and then use a tool like https://webinstall.dev/jq/, Python and whatever other tools you have on your system to extract the data that you want and then update the DB.
I am more familiar with PHP, which would work also. You could tidy that up a bit, but seems to work actually.
earthquake.php
<?php
$database = false;
try {
$options = array(PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_OBJ, PDO::ATTR_ERRMODE => PDO::ERRMODE_WARNING, PDO::ATTR_EMULATE_PREPARES => true );
$conn = new PDO('mysql:host=127.0.0.1;dbname=test;port=3306;charset=utf8','root','root', $options);
} catch (PDOException $e) {
// Echo custom message. Echo error code gives you some info.
echo '[{"error":"Database connection can not be estabilished. Please try again later. Error code: ' . $e->getCode() . '"}]';
exit;
}
$url = "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
$features = json_decode($result)->features;
foreach ($features as $feature) {
echo 'Mag: '.$feature->properties->mag.', Place: '.$feature->properties->place.', '.gmdate("Y-m-d H:i:s", $feature->properties->time/1000).PHP_EOL;
$query = 'INSERT INTO features (mag, place, time) VALUES (?, ?, ?)';
$params = [$feature->properties->mag, $feature->properties->place, gmdate("Y-m-d H:i:s", $feature->properties->time/1000)];
$stmt = $conn->prepare($query) or die ('["status":{"error":"Prepare Statement Failure","query":"' .$query . '"}]');
$stmt->execute($params) or die('[{"error":"' . $stmt->errorInfo()[2] . '","query":"' .$query . '","params":' .json_encode($params) . '}]');
}
?>
create a local DB called features with mag, place and time columns. If you have php on your system just run it from the CLI, php earthquake.php
e.g. Insert into mysql from Bash script
I use Laravel a bit, and would probably actually build my own model and use Eloquent and a little UI to handle that, but using a script seems like an option.
i'm facing problems into cURL. here i can get the json data from another source code and also i got some ideas. but really i can't save the json data into my own server in sql database. i wanna to retrieve the data from
url: https://jamuna.tv/wp-json/wp/v2/posts
and wanna to save into my server (mysql).
here is the $url json data i wanna to save in mysql server:
id
date
link
title
content
author
categories
wp:attachment
and i want to save them into mysql database. my table name is "news" and i want to save them into my table columns.
id [id]
title [title]
description [content]
date [date]
category [categories]
thumbnail [wp:attachment]
admin [author]
here i'm mark the json from url and replaced into my sql columns name. if anyone can give me the instructions about how i can fetch the data from user and save into mysql.
thanks advance.
There is an implementation of your desired with PHP. You must check if values are valid or not or if the array members exist or not. This is just a primitive implementation of your example.
<!DOCTYPE html>
<html>
<body>
<?php
// db connection
$options = array(
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC
);
$db = new PDO('mysql:host=localhost;dbname=test;', 'user', 'pass', $options);
// download json from url
$json = file_get_contents('https://jamuna.tv/wp-json/wp/v2/posts');
echo '<br/>';
$sql = "INSERT INTO `mytable` (`id`,`title`,`description`,`date`,`category`,`thumbnail`,`admin`)
VALUES (:id, :title, :content , :date, :categories, :attachment, :author)";
$stm = $db->prepare($sql);
// parse JSON
$arr = json_decode($json, true);
$i = 0;
foreach ($arr as $record) {
echo "==================Insert record " . $i ++ . "<br>";
$data = array(
':id' => $record['id'],
':title' => $record['title']['rendered'],
':content' => $record['content']['rendered'],
':date' => $record['date'],
':categories' => $record['categories'][0],
':attachment' => $record['_links']['wp:attachment'][0]['href'],
':author' => $record['author']
);
var_dump($data);
// inserting a record
$stm->execute($data);
}
?>
</body>
</html>
I'm having trouble processing the returned results from a DB SQL Mapper into a recognizable json encoded array.
function apiCheckSupplyId() {
/*refer to the model Xrefs*/
$supply_id = $this->f3->get('GET.supply_id');
$xref = new Xrefs($this->tongpodb);
$supply = $xref->getBySupplyId( $supply_id );
if ( count( $supply ) == 0 ) {
$this->logger->write('no xref found for supply_id=' .$supply_id);
$supply = array( array('id'=>0) );
echo json_encode( $supply );
} else {
$json = array();
foreach ($supply as $row){
$item = array();
foreach($row as $key => $value){
$item[$key] = $value;
}
array_push($json, $item);
}
$this->logger->write('xref found for supply_id=' .$supply_id.json_encode( $json ) );
echo json_encode( $json );
}
}
This is the method I am using but it seems very clunky to me. Is there a better way?
Assuming the getBySupplyId returns an array of Xref mappers, you could simplify the whole thing like this:
function apiCheckSupplyId() {
$supply_id = $this->f3->get('GET.supply_id');
$xref = new Xrefs($this->tongpodb);
$xrefs = $xref->getBySupplyId($supply_id);
echo json_encode(array_map([$xref,'cast'],$xrefs));
$this->logger->write(sprintf('%d xrefs found for supply_id=%d',count($xrefs),$supply_id));
}
Explanation:
The $xrefs variable contains an array of mappers. Each mapper being an object, you have to cast it to an array before encoding it to JSON. This can be done in one line by mapping the $xref->cast() method to each record: array_map([$xref,'cast'],$xrefs).
If you're not confident with that syntax, you can loop through each record and cast it:
$cast=[];
foreach ($xrefs as $x)
$cast[]=$x->cast();
echo json_encode($cast);
The result is the same.
The advantage of using cast() other just reading each value (as you're doing in your original script) is that it includes virtual fields as well.
I can't seem to figure out the issue with my .pm file and script. I am fairly new to Perl.
I have a database with name "project" and there is table with name "mailing".
mailing table has 7 entries, which I want to display using module.
So, I have this custom module to log in to database and do a query. This module is names as DB.pm
DB.pm is stored on my FEDORA 20 at /root/mysql/GUI/DB.pm.
DB.pm is defined as follows:
package GUI::DB;
use strict;
use DBI;
use vars qw(#ISA #EXPORT);
use Exporter;
#ISA = qw(Exporter);
#EXPORT = qw(dbConnect query);
#
# dbConnect - connect to the database, get the database handle
#
sub dbConnect {
# Read database settings from config file:
print "Works";
my $dsn = "DBI:mysql:project";
my $dbh = DBI->connect( $dsn,
'root',
'mydatabasepassword',
{ RaiseError => 1 }
);
return $dbh;
}
#
# query - execute a query with parameters
# query($dbh, $sql, #bindValues)
#
sub query {
my $dbh = shift;
my $sql ="SELECT * FROM mailing";
my #bindValues = #_; # 0 or serveral parameters
my #returnData = ();
# issue query
my $sth = $dbh->prepare($sql); //**line number 39 that is giving** error
if ( #bindValues ) {
$sth->execute(#bindValues);
} else {
$sth->execute();
}
if ( $sql =~ m/^select/i ) {
while ( my $row = $sth->fetchrow_hashref ) {
push #returnData, $row;
}
}
# finish the sql statement
$sth->finish();
return #returnData;
}
1;
Now I want to use this module inside my per script. This is what I tried:
#!/usr/bin/perl
use warnings;
use strict;
use lib '/root/mysql/';
use GUI::DB qw(dbConnect query);
dbConnect();
query();
This is the error I'm getting -->
Can't call method "prepare" on an undefined value at /root/mysql/GUI/DB.pm line 39.
Please help me with this. I am not sure how to proceed. I am guessing it has something to do with argument passing. Nothing is wrong with database. It works fine from CLI.
Thanks :)
_x_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X__X_X_X_X_X__X
TILL HERE IT IS RESOLVED
_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X_X__X_X_X
FURTHER ISSUE is SQL command is not working.
In the mailing table of my database I have email id with different domains.
For example, some id's are xyz#gmail.com, 12343#gmail.com , bae#yahoo.com as so on and I am assuming new email ids will be added to mailing tables every day with different domains.
I am trying to write a scripts that updates another table which holds a daily count of email addresses by their domain name. This is what I tried:
#!/usr/bin/perl
use warnings;
use strict;
use lib '/root/mysql/';
use 5.016;
use Data::Dumper;
use GUI::DB qw(dbConnect query);
my $data = dbConnect();
my #domain = query($data, "SELECT substr(addr,locate('\#',addr)+1) as maildomain, count (*) as mailcount FROM mailing GROUP BY maildomain ORDER BY mailcount DESC");
for my $key (#domain){
say Dumper ($key);
}
But I am getting an error,
You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '*) as mailcount FROM mailing GROUP BY maildomain ORDER BY mailcount DESC' at line 1 at /root/mysql/GUI/DB.pm line 44.
Same SQL statement works from CLI with no issues.
Any help would be appreciated. :)
1) Your error is saying that $dbh is undefined here:
sub query {
my $dbh = shift;
...
# issue query
my $sth = $dbh->prepare($sql); #<***LOOK HERE***
...which means $dbh must be undefined here:
sub query {
my $dbh = shift; #<***LOOK HERE***
...
# issue query
my $sth = $dbh->prepare($sql);
2) Let's see why. Your dbConnect() method returns $dbh:
sub dbConnect {
# Read database settings from config file:
print "Works";
my $dsn = "DBI:mysql:project";
my $dbh = DBI->connect(
$dsn,
'root',
'mydatabasepassword',
{ RaiseError => 1 }
);
return $dbh; #<***LOOK HERE*****
}
3) But, you call dbConnect() like this:
dbConnect();
Because you never save the return value anywhere, $dbh is discarded.
4) Furthermore, you call query() like this:
query();
Yet, you defined query() like this:
sub query {
my $dbh = shift;
The query() sub believes that the first argument will be the database handle--but you didn't call query() with any arguments.
You need to do this:
my $data_base_handle = dbConnect();
my #results = query($data_base_handle);
#do something with #results
Response to comment:
I printed #results, this is what I see HASH(0x1d05be8)
HASH(0x1d05ba0) HASH(0x1d05b58) HASH(0x1d05b10) HASH(0x1d05ac8)
HASH(0x1d05a80) HASH(0x1d05a38)
You wrote:
my $row = $sth->fetchrow_hashref;
...which asks DBI to return each row as a reference to a hash. Then you wrote:
push #returnData, $row;
...which pushed each hash reference into an array. So query() returns an array of hash references. The notation HASH(0x1d05be8) is what perl outputs when you print a hash reference.
If you want to see what's in those hashes, then do this:
use 5.016; #enable say()
use Data::Dumper;
...
...
for my $href (#results) {
say Dumper($href);
}
To access the data in a hash reference, you can do this:
use strict;
use warnings;
use 5.016;
use Data::Dumper;
my $href = {
c => 3,
a => 1,
b => 2,
};
my %hash = %{$href}; #dereference, {}, the reference into a hash, %
for my $key ( keys %hash ) {
say "$key $hash{$key}";
}
--output:--
c 3
a 1
b 2
Response to next comment:
(Answer posted in comments under op.)
By the way, perl is pretty good at text processing, so if you couldn't figure out the problem with your query, you could process the email addresses with perl:
use strict;
use warnings;
use 5.012;
use Data::Dumper;
use DBI;
use DBD::mysql;
# CONFIG VARIABLES
my $db_type = "mysql";
my $database = "my_db";
my $host = "localhost";
my $port = "3306";
my $user = "root";
my $pword = "";
# DATA SOURCE NAME
my $dsn = "dbi:$db_type:$database:$host:$port";
# PERL DBI CONNECT
my $dbh = DBI->connect($dsn, $user, $pword);
# PREPARE THE QUERY
my $tablename = "mailing";
my $select =<<"END_OF_SELECT";
select addr from $tablename
END_OF_SELECT
my $addr_aref = $dbh->selectcol_arrayref($select); #Returns a reference to a flat array containing all the email addresses
$dbh->disconnect;
my %count_for;
for my $addr (#{$addr_aref}) {
$addr =~ s/.*#//;
$count_for{$addr}++;
}
say Dumper(\%count_for);
--output:--
$VAR1 = {
'google.com' => 2,
'gorilla.com' => 1,
'yahoo.com' => 3
};
I am trying to export my MySQL tables from my database to a JSON file, so I can list them in an array.
I can create files with this code no problem:
$sql=mysql_query("select * from food_breakfast");
while($row=mysql_fetch_assoc($sql))
{
$ID=$row['ID'];
$Consumption=$row['Consumption'];
$Subline=$row['Subline'];
$Price=$row['Price'];
$visible=$row['visible'];
$posts[] = array('ID'=> $ID, 'Consumption'=> $Consumption, 'Subline'=> $Subline, 'Price'=> $Price, 'visible'=> $visible);
}
$response['posts'] = $posts;
$fp = fopen('results.json', 'w');
fwrite($fp, json_encode($response));
fclose($fp);
Now this reads a table and draws it's info from the fields inside it.
I would like to know if it is possible to make a JSON file with the names of the tables, so one level higher in the hierarchy.
I have part of the code:
$showtablequery = "
SHOW TABLES
FROM
[database]
LIKE
'%food_%'
";
$sql=mysql_query($showtablequery);
while($row=mysql_fetch_array($sql))
{
$tablename = $row[0];
$posts[] = array('tablename'=> $tablename);
}
$response['posts'] = $posts;
But now i am stuck in the last line where is says: $ID=$row['ID']; This relates to the info inside the Table and I do not know what to put here.
Also as you can see, I need to filter the Tables to only list the tables starting with food_ and drinks_
Any help is greatly appreciated:-)
There is no 'table id' in MySQL and therefore the result set from SHOW TABLES has no index id. The only index in the resultset is named 'Tables_in_DATABASENAME'.
Also you should use the mysqli library as the good old mysql library is depreacted. Having prepared an example:
<?php
$mysqli = new mysqli(
'yourserver',
'yourusername',
'yourpassword',
'yourdatabasename'
);
if ($mysqli->connect_errno) {
echo "Failed to connect to MySQL: (" . $mysqli->connect_errno . ") "
. $mysqli->connect_error;
}
$result = $mysqli->query('SHOW TABLES FROM `yourdatabasename` LIKE \'%food_%\'');
if(!$result) {
die('Database error: ' . $mysqli->error);
}
$posts = array();
// use fetch_array instead of fetch_assoc as the column
while($row = $result->fetch_array()) {
$tablename = $row[0];
$posts []= array (
'tablename' => $tablename
);
}
var_dump($posts);