Writing GPS string to MySQL database on Ubuntu Server - mysql

Question

I have an app that writes a set of GPS strings to a text file like this:
[{"date":"02/13/2017 19:26:00","time":1486974360428,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:13","time":1486974373496,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:23","time":1486974383539,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:33","time":1486974393449,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:43","time":1486974403423,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:53","time":1486974413483,"longitude":151.209900,"latitude":-33.865143}]
the file always starts and ends with [].
This file gets uploaded to an Ubuntu server at
'filepath'/uploads/gps/'device ID'/'year-month-day'/'UTC download time'.txt
for example
/uploads/gps/12/2017-02-12/1486940878.txt
The text files get created when the file gets uploaded to the server, so there are multiple files written per day.
I would like a method to write the values to a MySQL database with the headings DEVICE (obtained from the filepath), DATE, TIME, LONGITUDE, LATITUDE.
Initially, just a command I can run on the server would be preferable, which I can eventually run from a PHP command on an admin panel.
Where do I start?

Answer 1

Instead of uploading, you could easily submit the text to a PHP program on the server. It would use JSON decode to convert it to an array, and then save each record to a table. The device ID would be one of the parameters to the script.
Using this type of approach would eliminate a lot of issues such as not importing a file twice, renaming/moving the files after import, finding the file(s), etc.
It would also mean your data is up to date every time the data is sent.
A script like that would be pretty trivial to write, but it should have some type of security built in to prevent data from being sent by an unauthorized entity.
Here's some sample code that will process the files and store them to the DB. I've removed certain info (user ID/password database name) that you will need to edit. It's a little longer that I guessed, but still pretty short. If you need more info, PM me.
<?php
/* ===============================================================
Locate and parse GPS files, then store to MySQL DB.
Presumes a folder stucture of gps/device_id/date:YYYY-MM-DD.
After a file is processed and stored in the DB table, the
file is renamed with a leading "_" so it will be ignored later.
===============================================================
*/
$DS = '/'; // Directory separator character. Use '/' for Linux, '\' for windows.
// Path to folder containing device folders.
$base_folder = "./gps";
// Today's date foratted like the folders under the devices. If parameter "date" has a value, use it instead of today's date. Parameter MUST be formatted correctly.
$today = isset($_REQUEST['date']) && $_REQUEST['date'] != '' ? $_REQUEST['date'] : date('Y-m-d');
// Get a list of device folders
$device_folders = get_folders($base_folder);
// Loop through all of the device folders
$num_file_processed = 0;
foreach($device_folders as $dev_folder) {
// Check to see if there is a folder in the device folder for today.
$folder_path = $base_folder.$DS.$dev_folder.$DS.$today;
// Check if the device/date folder exists.
if(file_exists($folder_path) && is_dir($folder_path)) {
// Folder exists, get a list of files that haven't been processed.
$file_list = get_files($folder_path);
// Process the files (if any)
foreach($file_list as $filename) {
$f_path = $folder_path.$DS.$filename;
$json = file_get_contents($f_path);
// Fix the JSON -- missing "," between records.
$json = str_replace("}{","},{",$json);
$data = json_decode($json);
// Process each row of data and save to DB.
$num_saved = 0;
$rec_num = 0;
foreach($data as $recno => $rec_data) {
if(save_GPS($dev_folder,$rec_data->date,$rec_data->time,$rec_data->longitude,$rec_data->latitude)) {
$num_saved++;
}
$rec_num++;
}
// Rename file so we can ignore it if processing is done again.
if($num_saved > 0) {
$newName = $folder_path.$DS."_".$filename;
rename($f_path,$newName);
$num_file_processed++;
}
}
} else {
echo "<p>" . $folder_path . " not found.</p>\n";
}
}
echo "Processing Complete. ".$num_file_processed." files processed. ".$num_saved." records saved to db.\n";
function save_GPS($dev_id,$rec_date,$rec_time,$long,$lat) {
$server = "localhost";
$uid = "your_db_user_id";
$pid = "your_db_password";
$db_name = "your_database_name";
$qstr = "";
$qstr .= "INSERT INTO `gps_log`\n";
$qstr .= "(`device`,`date`,`time`,`longitude`,`latitude`)\n";
$qstr .= "VALUES\n";
$qstr .= "('".$dev_id."','".$rec_date."','".$rec_time."','".$long."','".$lat."');\n";
$db = mysqli_connect($server,$uid,$pid,$db_name);
if(mysqli_connect_errno()) {
echo "Failed to connect to MySQL server: " . mysqli_connect_errno() . " " . mysqli_connect_error() . "\n";
return false;
}
// Connected to DB, so save the record
mysqli_query($db,$qstr);
mysqli_close($db);
return true;
}
function get_folders($base_folder) {
$rslts = array();
$folders = array_map("htmlspecialchars", scandir($base_folder));
foreach($folders as $folder) {
// Ignore files and folders that start with "." (ie. current folder and parent folder references)
if(is_dir($base_folder."/".$folder) && substr($folder,0,1) != '.') {
$rslts[] = $folder;
}
}
return $rslts;
}
function get_files($base_folder) {
$rslts = array();
$files = array_map("htmlspecialchars", scandir($base_folder));
foreach($files as $file) {
// Ignore files and folders that start with "." (ie. current folder and parent folder references), or "_" (files already processed).
if(!is_dir($file) && substr($file,0,1) != '.' && substr($file,0,1) != '_') {
$rslts[] = $file;
}
}
return $rslts;
}