I have a hosted site with access to cPanel where I have a daily cron job executing a PHP script. This script exports a MySQL table to CSV. I then have a scheduled job on my Windows here at the office that FTPs this CSV to my local machine. So far so good.
But the MySQL table has mixed English and Hebrew data in it. Via the FileManager of the cPanel I see the Hebrew in the created CSV correctly, but after FTPing it to my local machine the Hebrew is unreadable.
EDIT :
When opening the downloaded CSV in Office-2016 the problem persists. When opening it with Notepad++ or MS-Notepad - the Hebrew appears ok.
This means that the file is downloaded correctly and the problem lays in the MS-Office.
The thing is that this CSV is to be used as input to an Excel macro (XLSM) which runs automatically nightly. I found that in Excel I can manually "Import" the CSV to a sheet and the encoding is fine and the Hebrew is ok. I recorded a macro and the VBA now does the job nicely. I then found it has already been mentioned in Opening tsv file via Notepad++ and save it in text format
END OF EDIT
The PHP script (notice the 'SET NAMES utf8') :
<?php
error_reporting(E_ALL);
ini_set('display_errors', 1);
$username = "XXX";$password ="YYY";$dbname = "ZZZ";
try {
$conn = new PDO('mysql:host=localhost;dbname='.$dbname, $username, $password);
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$conn->query('SET NAMES utf8');
$stmt = $conn->prepare("SELECT comp_id, comp_name FROM companies");
$stmt->execute();
$file_export = '/home/darushnisayon/public_html/vehadarta/Exported_tables_from_DB/AA_companies.csv';
$data = fopen($file_export, 'w');
$csv_fields = array();
$csv_fields[] = 'comp_id';
$csv_fields[] = 'comp_name';
fputcsv($data, $csv_fields);
while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
fputcsv($data, $row);
}
} catch(PDOException $e) {
echo 'ERROR: ' . $e->getMessage();
}
?>
My Windows FTP job (notice the BINARY option) :
#Echo Off
Set _FTPServerName=nn.nn.nnn.nnn
Set _UserName=XXX
Set _Password=YYY
Set _LocalFolder=C:\Dropbox\GADI\Vehadarta\Routine_Tasks\T002_Daily_Check_if_Synced
Set _RemoteFolder=public_html/vehadarta/Exported_tables_from_DB/
Set _Filename=AA_companies.csv
Set _ScriptFile=ftp1
:: Create script
>"%_ScriptFile%" Echo verbose
>>"%_ScriptFile%" Echo open %_FTPServerName%
>>"%_ScriptFile%" Echo %_UserName%
>>"%_ScriptFile%" Echo %_Password%
>>"%_ScriptFile%" Echo lcd %_LocalFolder%
>>"%_ScriptFile%" Echo cd %_RemoteFolder%
>>"%_ScriptFile%" Echo prompt
>>"%_ScriptFile%" Echo binary
>>"%_ScriptFile%" Echo get %_Filename%
>>"%_ScriptFile%" Echo quit
:: Run script
ftp -s:"%_ScriptFile%"
Del "%_ScriptFile%"
The CSV file seen at the cPanel :
comp_id,comp_name
1,"קשרי עסקים בע""מ"
2,ASK
3,DCL
4,"אסטרטגיה וליווי עסקי S.M.C"
The CSV file on my local dir after FTP :
comp_id comp_name
1 ׳§׳©׳¨׳™ ׳¢׳¡׳§׳™׳ ׳‘׳¢"׳
2 ASK
3 DCL
4 ׳׳¡׳˜׳¨׳˜׳’׳™׳” ׳•׳׳™׳•׳•׳™ ׳¢׳¡׳§׳™ S.M.C
Thank you for any idea.
As I mentioned in my EDIT, the problem was in the way MS-Office reads the CSV. When opening the file in Notepad the encoding is correct. Since the CSV is to be copied into am Excel spreadsheet, all I had to do was to 'Import' the CSV into the Excel.
Many thanks to Martin Prikryl who gave me the first hint for the solution.
Related
I´m trying to import about 3gb csv files to phpmyadmin. Some of them contains more terminated chars and then importing stops because of wrong fields.
I have two colums which i want to fill. Im using : as terminanting char but when there is more of them in line it just stops. I cannot manage csv files they are too big. I want to skip error lines or look for other solutions. How can i do this ?
csv files looks like this
ahoj123:dublin
cat:::dog
pes::lolko
As a solution to your problem, I have written a simple PHP file that will "fix" your file for you ..
It will open "test.csv" with contents of:
ahoj123:dublin
cat:::dog
pes::lolko
And convert it to the following and save to "fixed_test.csv"
ahoj123:dublin
cat:dog
pes:lolko
Bear in mind that I am basing this on your example, so I am letting $last keep it's EOL character since there is no reason to remove or edit it.
PHP file:
<?php
$filename = "test.csv";
$handle = fopen($filename, "r+") or die("Could not open $filename" . PHP_EOL);
$keep = '';
while(!feof($handle)) {
$line = fgets($handle);
$elements = explode(':', $line);
$first = $elements[0];
$key = (count($elements) - 1);
$last = $elements[$key];
$keep .= "$first:$last";
}
fclose($handle);
$new_filename = "better_test.csv";
$new_handle = fopen("fixed_test.csv", "w") or die("Could not open $new_filename" . PHP_EOL);
fwrite($new_handle, $keep);
fclose($new_handle);
One of the websites I host still runs on PHP 5.2 with MySQL 4.0. Its text is in Hebrew (displays just fine on the site), but in the DB the text appears as gibberish containing question marks—but not exclusively. It looks some like: ?????£ ?§???¥ ???£ ?×???
I am trying to move this DB to MySQL 5.x with the same website, with no luck so far. I have tried using the MYSQL 40 compatibility mode, as well as other compatibility modes. I have made sure that the destination DB has the hebrew_bin collation as the old one, and I've played around with SET NAMES. The problem is, this is a type of gibberish I am not yet familiar with and therefore have no idea how to convert it to readable text.
I didn't get another answer here, and therefore had no choice but to write a simple script that manually does the migration process through PHP and not through MySQL dumps.
Here is the script, with some obvious modifications. Please note that the script is "dirty code", it might not be secure and it's not the most efficient, but it should get the job done if you're using it internally. Do not use in an environment where the script is accessible to the public without further modifications.
<?php
$local = mysqli_connect([source server], [source username], [source password], [source DB name]) or die('No local connection');
mysqli_query($local, "SET NAMES 'hebrew'");
$remote = mysqli_connect([destination server], [destination username], [destination password], [destination DB name]) or die('No remote connection');
mysqli_query($remote, "SET NAMES 'utf8'");
$table_list = array('table1', 'table2', 'table3'); // you can get a list of tables automatically too if the list is very long
foreach ($table_list as $table)
{
$query_local = "SELECT * FROM `{$table}`";
$result_local = mysqli_query($local, $query_local) or die('Error in q1: '.mysqli_error($local));
$delete_remote = mysqli_query($remote, "DELETE FROM `{$table}` WHERE 1") or die('Error deleting table: '.$table); // necessary only if you plan to run it more than once
$i = 0;
while ($row_local = mysqli_fetch_assoc($result_local))
{
foreach ($row_local as $key => $value)
$row_local[$key] = mysqli_real_escape_string($remote, $value);
$query_remote = "INSERT INTO `{$table}` (`".implode('`, `', array_keys($row_local))."`) VALUES ('".implode("', '", $row_local)."')";
$result_remote = mysqli_query($remote, $query_remote)
or die('Error in remote q'.$i.' in table '.$table.':<br /> '.mysqli_error($remote).'<br /> Query: '.$query_remote);
echo 'Successfully transferred row '.$i.' in table '.$table;
echo '<br />'.PHP_EOL;
$i++;
}
}
?>
so I have a Folder called DATA, and it includes the following: part1.html, part2.html, part3.html, HTML.htm, plain.html, and jojo.jsp.
Now i use the following commmand to open the DATA folder and extract the files containing .htm
opendir(DIR,'DATA');
my(#dir) = grep /\.htm/, readdir (DIR);
closedir(DIR);
It successfully prints out the name of the files containing the extensions .html . Now i wish to use the html file that are filtered and print the data out into the cygwin terminal. I tried to use the files and stored it to a variable, and use a foreach loop to open the first html file using Filehandler and printing out the data init. The loop will repeat itself and do the same for all the other html files. But i seemed to run into the error! Please help!
my $value = join(#dir);
print "$value\n";
foreach(#dir){
my $movies = my $value;
open (FHD, $movies) || die " could not open $movies\n";
my #movies = <FHD>;
my $value2 = join(', ', #movies);
print "$value2\n";
What's with this line?
my $movies = my $value;
You're making this a lot harder than it needs to be.
Just use glob to read the directory as that will automatically include the path information on your found files.
use strict;
use warnings;
use autodie;
for my $html (glob('DATA/*.htm*')) {
print "File: $html\n";
open my $fh, '<', $html;
print <$fh>;
}
I want to create a HTML page that will have a table that will populate itself with info from 2 .txt files that are on a remote Linux Server.
or populate a html page on that remote server with the same info from those 2 .txt files and then access that html page using apache's webserver.
something as basic as possible would be nice but I can understand if it's complicated to do with html
honestly, any help at all would be nice.
I would personally do it in PHP. You can read the file and echo it into a table. You can then use the lines of the file for anything you want. I put comments in explaining each step. All you have to do is change $filepath to point at your text file:
Edited: Edited the code to add constraints mentioned by OG poster in comments. There is probably a more optimized way of performing your task, but this works and should introduce some new concepts to you if you are new to PHP
<?php
$filepath = 'files/the_file.txt';
if (file_exists($filepath)) {
$file = fopen($filepath, 'r');
echo '<table border=1>';
while (!feof($file)) {
$line = fgets($file);
$first_char = $line[0];
if ($first_char != '*' && $first_char != '^' && trim($line) != '') {
if (strstr($line, '|')) {
$split = explode('|', $line);
echo '<tr>';
foreach($split as $line) {
echo '<td>'.$line.'</td>';
}
echo '</tr>';
} else {
echo '<tr><td>'.$line.'</td></tr>';
}
}
}
echo '</table>';
} else {
echo 'the file does not exist';
}
?>
I'll do my best to explain it line by line instead of flooding the scrip with comments:
set your file path
If the file exists, continue on. If not, throw the error located at the bottom of the script
open the file
create the table ('<table>')
while the text file is being read, do a series of things: First, get the line. If the first character of the line is a * or ^, or when the line is trimmed there are no characters, skip it completely. Otherwise, continue on
if the line contains a | character, split (explode) the line at all of the | characters. Use this array of split up content and for each piece of content, echo out a new column in the existing row with the current content. Otherwise, there is not | found and you can just echo the line into a row normally
once you are finished up, end the table ('</table>')
Edit #2: The original solution I posted:
<?php
$filepath = '/var/www/files/the_file.txt';
if (file_exists($filepath)) {
$file = fopen($filepath, 'r');
echo '<table border=1>';
while (!feof($file)) {
$line = fgets($file);
echo '<tr><td>'.$line.'</td></tr>';
}
echo '</table>';
} else {
echo 'the file does not exist';
}
?>
HTML can't do anything, HTML is a presentation format.
PHP, Javascript, BASH could do the job in very different ways :
PHP : the server calls the 2 remote files and output the assembled html file into a webpage, then send it to the client
Javascript : the page itself calls the 2 files and add them in itself.
Bash + CURL : a BASH (or PHP, Python...) script creates a .html file containing the data of the 2 files.
One of these might help you, if you can precreate the HTML rather than doing it dynamically. These scripts take CSV as input and output an HTML table:
http://stromberg.dnsalias.org/svn/to-table/
http://stromberg.dnsalias.org/svn/to-table2/
I have the following script to update one of my FTP passwords every 15 days through a cronjob and e-mail the appropriate people after the attempt has been made. It randomly will fail and so I will run it again manually and it will work. I can't seem to find where it's going wrong.
The script is connecting to a local mysql database grabbing the login and password for an account and then changing that password on FTP. Everything is successful up until the changing the password part. Again it's random, sometimes it works, sometime it doesn't.
Thanks!
#!/usr/bin/perl -w
#
use DBI;
use Net::FTP;
our $dbh = DBI->connect('DBI:mysql:database:127.0.0.1','user','password') or die "Aargh $!\n";
$transquery=q{SELECT dest_login,dest_password FROM list where id=123};
$sth=$dbh->prepare($transquery);
$sth->execute();
while($co=$sth->fetchrow_hashref){
$login=$co->{'dest_login'};
$pass=$co->{'dest_password'};
}
$changeresult='FAIL';
$actionlog='';
$newstring='';
$upperchars='ABCDEFGHIJKLMNOPQRSTUVWXYZ';
$lowerchars='abcdefghijklmnopqrstuvwxyz';
$allowedchars='ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789##$';
$l=length($upperchars);
$newstring.=substr($upperchars,int(rand($l)),1);
$newstring.=substr($lowerchars,int(rand($l)),1);
$l=length($allowedchars);
for ($i=0;$i<6;$i++){
$newstring.=substr($allowedchars,int(rand($l)),1);
}
print "$newstring\n";
$actionlog .= "Setting Password for $login from $pass to $newstring\n";
$username=
eval{
$ftp=Net::FTP->new('x.x.x.x',Timeout=>480,Debug=>1) or die "Error connecting FTP $!\n";
$changepassword="$pass/$newstring/$newstring";
$ftp->login($login,$changepassword) or die "Error changing password $!\n";
#If we are here, time to update the password
$changeresult='SUCCESS';
$actionlog .= "Password successfully updated\n";
$transquery=q{UPDATE list set dest_password=(?) where id=123};
$sth=$dbh->prepare($transquery);
$sth->execute($newstring);
};
if ($#) {
$actionlog = $actionlog . "$#\n";
};
if($actionlog ne ""){
#print $actionlog;
#my $send_to = "To: someone\#example.com\n";
my $send_to = "To: databaseusers\#example.com\n";
my $sendmail = "/usr/sbin/sendmail -t";
open(SENDMAIL, "|$sendmail") or die "Cannot open $sendmail: $!";
print SENDMAIL "Reply-to: databasepassword\#example.com\n";
print SENDMAIL "Subject: Password Change Information [$changeresult]\n";
print SENDMAIL $send_to;
print SENDMAIL "Content-type: text/plain\n\n";
print SENDMAIL $send_to;
print SENDMAIL "Content-type: text/plain\n\n";
print SENDMAIL $actionlog;
close(SENDMAIL);
$actionlog='';
}
else{
#print "Nothing done this session\n";
USUW might tell you something. ( use strict; use warnings; )
Does anything print?
You don't do much error checking in the DBI part at the beginning, perhaps you're getting a connect error. AIX boxes used to have this problem of getting a client port that the system was unsure about whether or not it was in use. When that happened, it would just fail to connect to the database.
I finally fixed that problem for our scripts by examining the $OS_ERROR ( aka $! ) for that particular code ( Errno::EADDRINUSE ) and then waiting and retrying, with an exponential falloff ( wait 2 seconds, then 4, then 8 ... ).
If your script "dies for some reason" then it's important the script can tell you that reason. I would investigate the topic of error reporting in the various modules you are using.
For example Net::FTP allows you to pass a Debug => 1 switch, and then you'll see the whole conversation.
And I know that there is a whole lot more with DBI where you can get error reporting.