how to import json file from https web page into a mariadb or mysql database? - mysql

Question

I am trying to import a file into a mariadb (mysql), database. A proof of concept file is in .json format on the web at this location. https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson
I know how to do this on db2 for i.
select * from JSON_TABLE(
SYSTOOLS.HTTPGETCLOB('https://earthquake.usgs.gov' ||
'/earthquakes/feed/v1.0/summary/all_week.geojson',null),
'$.features[*]'
COLUMNS( MILLISEC BIGINT PATH '$.properties.time',
MAG DOUBLE PATH '$.properties.mag',
PLACE VARCHAR(100) PATH '$.properties.place'
)) AS X;
This reads a from the web and lists 3 fields. Surrounding it with an insert clause will put it into a database file for me.
I would like to do exactly the same thing on my home server using mariadb. Ultimately a script will run unattended on a hosted server.
A .json segment of the earthquake data looks like this:
… "features":[
{"type":"Feature",
"properties":{
"mag":1.1,
"place":"58 km WNW of Anchor Point, Alaska",
"time":1640472257402,
"updated":1640472615410,
"tz":null,
"url":"https://earthquake.usgs.gov/earthquakes/eventpage/ak021gi3al5x",
"detail":"https://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/ak021gi3al5x.geojson",
"felt":null,
"cdi":null,
"mmi":null,
"alert":null,
"status":"automatic",
"tsunami":0,
"sig":19,
"net":"ak",
"code":"021gi3al5x",
"ids":",ak021gi3al5x,",
"sources":",ak,",
"types":",origin,",
"nst":null,
"dmin":null,
"rms":0.79,
"gap":null,
"magType":"ml",
"type":"earthquake",
"title":"M 1.1 - 58 km WNW of Anchor Point, Alaska"},
"geometry":{
"type":"Point",
"coordinates":[-152.8406,59.9119,89.7]
},
"id":"ak021gi3al5x"
}, ...

Answer 1

Just a thought, but you could maybe just write a BASH script, e.g.
#!/bin/sh
cd "$(dirname "$0")"
export PATH=/bin:/usr/bin:/usr/local/bin
TODAY=`date +"%d%b%Y-%H%M"`
MYSQL_HOST='127.0.0.1'
MYSQL_PORT='3306'
MYSQL_USER='root'
MYSQL_PASSWORD='root'
url=https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson
DATA=$(curl ${url} 2>/dev/null)
printf '%s' "$DATA" | awk '{print $0}'
exit
...
...
...
and then use a tool like https://webinstall.dev/jq/, Python and whatever other tools you have on your system to extract the data that you want and then update the DB.
I am more familiar with PHP, which would work also. You could tidy that up a bit, but seems to work actually.
earthquake.php
<?php
$database = false;
try {
$options = array(PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_OBJ, PDO::ATTR_ERRMODE => PDO::ERRMODE_WARNING, PDO::ATTR_EMULATE_PREPARES => true );
$conn = new PDO('mysql:host=127.0.0.1;dbname=test;port=3306;charset=utf8','root','root', $options);
} catch (PDOException $e) {
// Echo custom message. Echo error code gives you some info.
echo '[{"error":"Database connection can not be estabilished. Please try again later. Error code: ' . $e->getCode() . '"}]';
exit;
}
$url = "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_week.geojson";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
$features = json_decode($result)->features;
foreach ($features as $feature) {
echo 'Mag: '.$feature->properties->mag.', Place: '.$feature->properties->place.', '.gmdate("Y-m-d H:i:s", $feature->properties->time/1000).PHP_EOL;
$query = 'INSERT INTO features (mag, place, time) VALUES (?, ?, ?)';
$params = [$feature->properties->mag, $feature->properties->place, gmdate("Y-m-d H:i:s", $feature->properties->time/1000)];
$stmt = $conn->prepare($query) or die ('["status":{"error":"Prepare Statement Failure","query":"' .$query . '"}]');
$stmt->execute($params) or die('[{"error":"' . $stmt->errorInfo()[2] . '","query":"' .$query . '","params":' .json_encode($params) . '}]');
}
?>
create a local DB called features with mag, place and time columns. If you have php on your system just run it from the CLI, php earthquake.php
e.g. Insert into mysql from Bash script
I use Laravel a bit, and would probably actually build my own model and use Eloquent and a little UI to handle that, but using a script seems like an option.