I want to remove duplicate email addresses based on the domain name. For example:
aa#example.com
bb#example.com
cc#bla.com
Should become:
aa#example.com
cc#bla.com
Can anyone help with this? I tried using sort / uniq and awk, but haven't got it working yet.
In php:
<?php
$domains = []; // list of domains we have already included
$cleanList = []; // "clean" email list
$list = file('/path/to/email-list.txt'); // load the raw list
// loop over the raw list
foreach($list as $email) {
// extract the domain from the email
$domain = preg_replace('/^.*#/', '', $email);
// if the domain has not been taken yet
if(!in_array($domain, $domains)) {
// add it to the list of taken domains
array_push($domains, $domain);
// add the email to the clean list
array_push($cleanList, $email);
}
}
// write the clean list out to a file
file_put_contents('/tmp/clean-emails.txt', implode("\n", $cleanList));
Related
I want to send a newsletter through email and I would like to see who opened my email.
I send HTML in the content of the email, so I can not add javascript in there. (see here )
Is there any way to send a post request (to my server) only through HTML, every time the HTML is opened and not by pressing a button?
No.
The only HTTP requests that can be triggered by simply opening an HTML document without any JS in it are GET requests.
Tracking of HTML emails is usually achieved using GET requests from images (and usually blocked by email clients because it is intrusive).
I think you are wanting to track emails which are opened? If you are using a PHP server you can create a simple "pixel" but would need to hook it up to your database.
Inside your email you can load the pixel as an image and replace the %pixel_id%
<img src="https://yoururl/pixel.php?tid=%pixel_id%" style="width:1px;height:1px;" title="pixel">
Pixel code:
<?php
//YOU NEED TO INCLUDE YOUR DATABASE CONNECTION FILE HERE
$conn_cms=get_dbc();
$stmt = $conn_cms->prepare("SELECT * FROM `pixel_tracker` WHERE `pixel_id` = ?");
$stmt->bind_param("s", $tid);
$tid = $_GET['tid'];
$stmt->execute();
$result = $stmt->get_result();
$assoc = $result->fetch_assoc();// get the mysqli result
$stmt = $conn_cms->prepare("UPDATE `pixel_tracker` SET `seen` = ?,`seen_count` = ?,
`seen_when`= ?, `header_track`= ? WHERE `pixel_id` = ?");
$stmt->bind_param("sssss", $seen, $seen_count, $seen_when, $header_track, $pixel_id);
$seen = 1;
$seen_count = $assoc['seen_count']+1;
$seen_when = date("Y-m-d H:i:s");
if(isset($_SERVER['HTTP_REFERER'])){
$header_track = $_SERVER['HTTP_REFERER'];
} else {
$header_track = "none";
}
$pixel_id = $tid;
$stmt->execute();
$result = $stmt->get_result(); // get the mysqli result
$pixel = imagecreate(1,1);
$color = imagecolorallocate($pixel,255,255,255);
imagesetpixel($pixel,1,1,$color);
header("content-type:image/jpg");
imagejpeg($pixel);
imagedestroy($pixel);
?>
EDIT: If you are sending emails automatically from your server you can dynamically insert the pixel reference in to your database, otherwise for now you can do this manually.
I have an app that writes a set of GPS strings to a text file like this:
[{"date":"02/13/2017 19:26:00","time":1486974360428,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:13","time":1486974373496,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:23","time":1486974383539,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:33","time":1486974393449,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:43","time":1486974403423,"longitude":151.209900,"latitude":-33.865143}{"date":"02/13/2017 19:26:53","time":1486974413483,"longitude":151.209900,"latitude":-33.865143}]
the file always starts and ends with [].
This file gets uploaded to an Ubuntu server at
'filepath'/uploads/gps/'device ID'/'year-month-day'/'UTC download time'.txt
for example
/uploads/gps/12/2017-02-12/1486940878.txt
The text files get created when the file gets uploaded to the server, so there are multiple files written per day.
I would like a method to write the values to a MySQL database with the headings DEVICE (obtained from the filepath), DATE, TIME, LONGITUDE, LATITUDE.
Initially, just a command I can run on the server would be preferable, which I can eventually run from a PHP command on an admin panel.
Where do I start?
Instead of uploading, you could easily submit the text to a PHP program on the server. It would use JSON decode to convert it to an array, and then save each record to a table. The device ID would be one of the parameters to the script.
Using this type of approach would eliminate a lot of issues such as not importing a file twice, renaming/moving the files after import, finding the file(s), etc.
It would also mean your data is up to date every time the data is sent.
A script like that would be pretty trivial to write, but it should have some type of security built in to prevent data from being sent by an unauthorized entity.
Here's some sample code that will process the files and store them to the DB. I've removed certain info (user ID/password database name) that you will need to edit. It's a little longer that I guessed, but still pretty short. If you need more info, PM me.
<?php
/* ===============================================================
Locate and parse GPS files, then store to MySQL DB.
Presumes a folder stucture of gps/device_id/date:YYYY-MM-DD.
After a file is processed and stored in the DB table, the
file is renamed with a leading "_" so it will be ignored later.
===============================================================
*/
$DS = '/'; // Directory separator character. Use '/' for Linux, '\' for windows.
// Path to folder containing device folders.
$base_folder = "./gps";
// Today's date foratted like the folders under the devices. If parameter "date" has a value, use it instead of today's date. Parameter MUST be formatted correctly.
$today = isset($_REQUEST['date']) && $_REQUEST['date'] != '' ? $_REQUEST['date'] : date('Y-m-d');
// Get a list of device folders
$device_folders = get_folders($base_folder);
// Loop through all of the device folders
$num_file_processed = 0;
foreach($device_folders as $dev_folder) {
// Check to see if there is a folder in the device folder for today.
$folder_path = $base_folder.$DS.$dev_folder.$DS.$today;
// Check if the device/date folder exists.
if(file_exists($folder_path) && is_dir($folder_path)) {
// Folder exists, get a list of files that haven't been processed.
$file_list = get_files($folder_path);
// Process the files (if any)
foreach($file_list as $filename) {
$f_path = $folder_path.$DS.$filename;
$json = file_get_contents($f_path);
// Fix the JSON -- missing "," between records.
$json = str_replace("}{","},{",$json);
$data = json_decode($json);
// Process each row of data and save to DB.
$num_saved = 0;
$rec_num = 0;
foreach($data as $recno => $rec_data) {
if(save_GPS($dev_folder,$rec_data->date,$rec_data->time,$rec_data->longitude,$rec_data->latitude)) {
$num_saved++;
}
$rec_num++;
}
// Rename file so we can ignore it if processing is done again.
if($num_saved > 0) {
$newName = $folder_path.$DS."_".$filename;
rename($f_path,$newName);
$num_file_processed++;
}
}
} else {
echo "<p>" . $folder_path . " not found.</p>\n";
}
}
echo "Processing Complete. ".$num_file_processed." files processed. ".$num_saved." records saved to db.\n";
function save_GPS($dev_id,$rec_date,$rec_time,$long,$lat) {
$server = "localhost";
$uid = "your_db_user_id";
$pid = "your_db_password";
$db_name = "your_database_name";
$qstr = "";
$qstr .= "INSERT INTO `gps_log`\n";
$qstr .= "(`device`,`date`,`time`,`longitude`,`latitude`)\n";
$qstr .= "VALUES\n";
$qstr .= "('".$dev_id."','".$rec_date."','".$rec_time."','".$long."','".$lat."');\n";
$db = mysqli_connect($server,$uid,$pid,$db_name);
if(mysqli_connect_errno()) {
echo "Failed to connect to MySQL server: " . mysqli_connect_errno() . " " . mysqli_connect_error() . "\n";
return false;
}
// Connected to DB, so save the record
mysqli_query($db,$qstr);
mysqli_close($db);
return true;
}
function get_folders($base_folder) {
$rslts = array();
$folders = array_map("htmlspecialchars", scandir($base_folder));
foreach($folders as $folder) {
// Ignore files and folders that start with "." (ie. current folder and parent folder references)
if(is_dir($base_folder."/".$folder) && substr($folder,0,1) != '.') {
$rslts[] = $folder;
}
}
return $rslts;
}
function get_files($base_folder) {
$rslts = array();
$files = array_map("htmlspecialchars", scandir($base_folder));
foreach($files as $file) {
// Ignore files and folders that start with "." (ie. current folder and parent folder references), or "_" (files already processed).
if(!is_dir($file) && substr($file,0,1) != '.' && substr($file,0,1) != '_') {
$rslts[] = $file;
}
}
return $rslts;
}
I am experimenting with a Firefox extension that will load an arbitrary URL (only via HTTP or HTTPS) when certain conditions are met.
With certain conditions, I just want to display a message instead of requesting a URL from the internet.
I was thinking about simply hosting a local webpage that would display the message. The catch is that the message needs to include a variable.
Is there a simple way to craft a local web page so that it can display a variable passed to it in the URL? I would prefer to just use HTML and CSS, but adding a little inline javascript would be okay if absolutely needed.
As a simple example, when the extension calls something like:
folder/messageoutput.html?t=Text%20to%20display
I would like to see:
Message: Text to display
shown in the browser's viewport.
You can use the "search" property of the Location object to extract the variables from the end of your URL:
var a = window.location.search;
In your example, a will equal "?t=Text%20to%20display".
Next, you will want to strip the leading question mark from the beginning of the string. The if statement is just in case the browser doesn't include it in the search property:
var s = a.substr(0, 1);
if(s == "?"){s = substr(1);}
Just in case you get a URL with more than one variable, you may want to split the query string at ampersands to produce an array of name-value pair strings:
var R = s.split("&");
Next, split the name-value pair strings at the equal sign to separate the name from the value. Store the name as the key to an array, and the value as the array value corresponding to the key:
var L = R.length;
var NVP = new Array();
var temp = new Array();
for(var i = 0; i < L; i++){
temp = R[i].split("=");
NVP[temp[0]] = temp[1];
}
Almost done. Get the value with the name "t":
var t = NVP['t'];
Last, insert the variable text into the document. A simple example (that will need to be tweaked to match your document structure) is:
var containingDiv = document.getElementById("divToShowMessage");
var tn = document.createTextNode(t);
containingDiv.appendChild(tn);
getArg('t');
function getArg(param) {
var vars = {};
window.location.href.replace( location.hash, '' ).replace(
/[?&]+([^=&]+)=?([^&]*)?/gi, // regexp
function( m, key, value ) { // callback
vars[key] = value !== undefined ? value : '';
}
);
if ( param ) {
return vars[param] ? vars[param] : null;
}
return vars;
}
On my page for an add-on i am creating works perfectly well unless i try to update an existing row. `public function actionUpdate()
{
$visitor = XenForo_Visitor::getInstance();
$userName = $visitor['username'];
//Get the text that user wrote in the text box
$text3 = $this->_input->filterSingle('simple_text2', XenForo_Input::STRING);
$text4 = $this->_input->filterSingle('simple_text3', XenForo_Input::STRING);
//Create a instance of our DataWriter
$dwSimpleText = XenForo_DataWriter::create('MinecraftAdder_DataWriter_MinecraftAdder');
//Set the field with the data we filtered
$dwSimpleText->setExistingData('Name');
$dwSimpleText->set('Name', $text3);
$dwSimpleText->setExistingData('Rank');
$dwSimpleText->set('Rank', XenForo_Visitor::getUserId());
$dwSimpleText->setExistingData('UUID');
$dwSimpleText->set('UUID', $text4);
$dwSimpleText->setExistingData('UserID');
$dwSimpleText->set('UserID', $userName);
//Save in the database, please!
$dwSimpleText->save();
//Send a response to the user, so he know that everything went fine with this action
return $this->responseRedirect(
XenForo_ControllerResponse_Redirect::SUCCESS,
$this->getDynamicRedirect()
);
}`
I get the error The existing data required by the data writer could not be found. Does anyone know how to fix this?
My Addon page
Your usage of setExistingData is incorrect. That function must be given one of these two values:
The value of the Primary Key column(s) for the row you want to update
An array with all the data for the row you want to update
So in your case, seeing as you don't select the row in advance, you'd use option 1. Assuming UserID is your Primary Key column, your code would be:
public function actionUpdate() {
$visitor = XenForo_Visitor::getInstance();
$userName = $visitor['username'];
//Get the text that user wrote in the text box
$text3 = $this->_input->filterSingle('simple_text2', XenForo_Input::STRING);
$text4 = $this->_input->filterSingle('simple_text3', XenForo_Input::STRING);
//Create a instance of our DataWriter
$dwSimpleText = XenForo_DataWriter::create('MinecraftAdder_DataWriter_MinecraftAdder');
//Set the field with the data we filtered
$dwSimpleText->setExistingData($userName);
$dwSimpleText->set('Name', $text3);
$dwSimpleText->set('Rank', XenForo_Visitor::getUserId());
$dwSimpleText->set('UUID', $text4);
//Save in the database, please!
$dwSimpleText->save();
//Send a response to the user, so he know that everything went fine with this action
return $this->responseRedirect(
XenForo_ControllerResponse_Redirect::SUCCESS,
$this->getDynamicRedirect()
);
}
Some other tips:
If this actionUpdate may be accessed through a form only, you should add $this->_assertPostOnly(); on the top of the function to make sure the request was a POST request.
You may want to check if the Visitor's user_id is 0 so you don't save the information of guests. (Unless that's what you want of course)
I have a bunch of html files in a site that were created in the year 2000 and have been maintained to this day. We've recently began an effort to replace illegal characters with their html entities. Going page to page looking for copyright symbols and trademark tags seems like quite a chore. Do any of you know of an app that will take a bunch of html files and tell me where I need to replace illegal characters with html entities?
You could write a PHP script (if you can; if not, I'd be happy to help), but I assume you already converted some of the "special characters", so that does make the task a little harder (although I still think it's possible)...
Any good text editor will do a file contents search for you and return a list of matches.
I do this with EditPlus. There are several editors like Notepad++, TextPad, etc that will easily help you do this.
You do not have to open the files. You just specify a path where the files are stored and the Mask (*.html) and the contents to search for "©" and the editor will come back with a list of matches and when you double click, it opens the file and brings up the matching line.
I also have a website that needs to regularly convert large numbers of file names back and forth between character sets. While a text editor can do this, a portable solution using 2 steps in php was preferrable. First, add the filenames to an array, then do the search and replace. An extra piece of code in the function excludes certain file types from the array.
Function listdir($start_dir='.') {
$nonFilesArray=array('index.php','index.html','help.html'); //unallowed files & subfolders
$filesArray = array() ; // $filesArray holds new records and $full[$j] holds names
if (is_dir($start_dir)) {
$fh = opendir($start_dir);
while (($tmpFile = readdir($fh)) !== false) { // get each filename without its path
if (strcmp($tmpFile, '.')==0 || strcmp($tmpFile, '..')==0) continue; // skip . & ..
$filepath = $start_dir . '/' . $tmpFile; // name the relative path/to/file
if (is_dir($filepath)) // if path/to/file is a folder, recurse into it
$filesArray = array_merge($filesArray, listdir($filepath));
else // add $filepath to the end of the array
$test=1 ; foreach ($nonFilesArray as $nonfile) {
if ($tmpFile == $nonfile) { $test=0 ; break ; } }
if ( is_dir($filepath) ) { $test=0 ; }
if ($test==1 && pathinfo($tmpFile, PATHINFO_EXTENSION)=='html') {
$filepath = substr_replace($filepath, '', 0, 17) ; // strip initial part of $filepath
$filesArray[] = $filepath ; }
}
closedir($fh);
} else { $filesArray = false; } # no such folder
return $filesArray ;
}
$filesArray = listdir($targetdir); // call the function for this directory
$numNewFiles = count($filesArray) ; // get number of records
for ($i=0; $i<$numNewFiles; $i++) { // read the filenames and replace unwanted characters
$tmplnk = $linkpath .$filesArray[$i] ;
$outname = basename($filesArray[$i],".html") ; $outname = str_replace('-', ' ', $outname);
}