How to remove non-printable character ^# in perl - csv

Question

I have a simple perl script as below
use strict;
use warnings 'all';
use Text::CSV;
use open ":std", ":encoding(UTF-8)";
my $in_file = $ARGV[0] or die "Usage: $0 filename\n";
my $out_file = $ARGV[1] or die "Usage: $0 filename\n";
my $csv = Text::CSV->new( { binary => 1, auto_diag => 1 });
my $out_csv = Text::CSV->new ( { binary => 1,
auto_diag => 1,
eol => $/,
sep_char => ',',
quote_char => '"',
always_quote => 1});
open my $data, '<:encoding(UTF-8)', $in_file or die $!;
open my $fh_out, '>:encoding(UTF-8)', $out_file or die $!;
while (my $words = $csv->getline($data))
{
tr/\r\n//d for #$words;
tr/,/;/ for #$words;
tr/"/'/ for #$words;
$out_csv->print($fh_out, $words);
}
All it does is basically convert a csv file into more structured one which is easily understandable by a external application which has limitation. The script removes unwanted new line, commas and double quote which are entered as part user text entry. This has been working well . However recently in one of the input file we have non-printable character ^# , he script doesn't fail but the out put file have additional characters "0
Not sure if I can give an example of input file since the charcater is non-printable. Please see the command output below.
cat Input.csv
ObjectId,OrgId,Title,ObjectType,Type,ObjectId,Location,StartDate,EndDate,DueDate,ObjectId1,ObjectId2,ObjectId3,LastModified,IsDeleted,SortOrder,Depth,DummyId,IsHidden,ResultId,DeletedDate,CreatedBy,LastModifiedBy,DeletedBy
3386484,532947,Test,Topic,Auto,3386415,http://www.test.com ,,,,,,,2016-06-27T05:08:26.3070000Z,1,443,3,,False,,2017-02-16T00:31:39.4870000Z,,,
With cat -e option you can see the non printable character.
cat -e Input.csv
M-oM-;M-?ObjectId,OrgId,Title,ObjectType,Type,ObjectId,Location,StartDate,EndDate,DueDate,ObjectId1,ObjectId2,ObjectId3,LastModified,IsDeleted,SortOrder,Depth,DummyId,IsHidden,ResultId,DeletedDate,CreatedBy,LastModifiedBy,DeletedBy^M$
3386484,532947,Test,Topic,Auto,3386415,http://www.test.com ^#,,,,,,,2016-06-27T05:08:26.3070000Z,1,443,3,,False,,2017-02-16T00:31:39.4870000Z,,,^M$
Once the file goes through the script the output looks like this. With additional "0 at the end of http://www.test.com
cat -e Output.csv
"M-oM-;M-?ObjectId","OrgId","Title","ObjectType","Type","ObjectId","Location","StartDate","EndDate","DueDate","ObjectId1","ObjectId2","ObjectId3","LastModified","IsDeleted","SortOrder","Depth","DummyId","IsHidden","ResultId","DeletedDate","CreatedBy","LastModifiedBy","DeletedBy"$
"3386484","532947","Test","Topic","Auto","3386415","http://www.test.com "0","","","","","","","2016-06-27T05:08:26.3070000Z","1","443","3","","False","","2017-02-16T00:31:39.4870000Z","","",""$
Adding command output as per Dave's request
od -ch Input.csv
0000000 357 273 277 O b j e c t I d , O r g I
bbef 4fbf 6a62 6365 4974 2c64 724f 4967
0000020 d , T i t l e , O b j e c t T y
2c64 6954 6c74 2c65 624f 656a 7463 7954
0000040 p e , T y p e , O b j e c t I d
6570 542c 7079 2c65 624f 656a 7463 6449
0000060 , L o c a t i o n , S t a r t D
4c2c 636f 7461 6f69 2c6e 7453 7261 4474
0000100 a t e , E n d D a t e , D u e D
7461 2c65 6e45 4464 7461 2c65 7544 4465
0000120 a t e , O b j e c t I d 1 , O b
7461 2c65 624f 656a 7463 6449 2c31 624f
0000140 j e c t I d 2 , O b j e c t I d
656a 7463 6449 2c32 624f 656a 7463 6449
0000160 3 , L a s t M o d i f i e d , I
2c33 614c 7473 6f4d 6964 6966 6465 492c
0000200 s D e l e t e d , S o r t O r d
4473 6c65 7465 6465 532c 726f 4f74 6472
0000220 e r , D e p t h , D u m m y I d
7265 442c 7065 6874 442c 6d75 796d 6449
0000240 , I s H i d d e n , R e s u l t
492c 4873 6469 6564 2c6e 6552 7573 746c
0000260 I d , D e l e t e d D a t e , C
6449 442c 6c65 7465 6465 6144 6574 432c
0000300 r e a t e d B y , L a s t M o d
6572 7461 6465 7942 4c2c 7361 4d74 646f
0000320 i f i e d B y , D e l e t e d B
6669 6569 4264 2c79 6544 656c 6574 4264
0000340 y \r \n 3 3 8 6 4 8 4 , 5 3 2 9 4
0d79 330a 3833 3436 3438 352c 3233 3439
0000360 7 , T e s t , T o p i c , A u t
2c37 6554 7473 542c 706f 6369 412c 7475
0000400 o , 3 3 8 6 4 1 5 , h t t p : /
2c6f 3333 3638 3134 2c35 7468 7074 2f3a
0000420 / w w w . t e s t . c o m \0 ,
772f 7777 742e 7365 2e74 6f63 206d 2c00
0000440 , , , , , , 2 0 1 6 - 0 6 - 2 7
2c2c 2c2c 2c2c 3032 3631 302d 2d36 3732
0000460 T 0 5 : 0 8 : 2 6 . 3 0 7 0 0 0
3054 3a35 3830 323a 2e36 3033 3037 3030
0000500 0 Z , 1 , 4 4 3 , 3 , , F a l s
5a30 312c 342c 3334 332c 2c2c 6146 736c
0000520 e , , 2 0 1 7 - 0 2 - 1 6 T 0 0
2c65 322c 3130 2d37 3230 312d 5436 3030
0000540 : 3 1 : 3 9 . 4 8 7 0 0 0 0 Z ,
333a 3a31 3933 342e 3738 3030 3030 2c5a
0000560 , , \r \n
2c2c 0a0d
0000564
How can I handle this in the script , I do not want the additional "0 in the output csv.
Thanks

Answer 1

Once you find out the numerical value for that character, you can use \x{} to specify it by that code number:
s/\x{....}//g;
For example, if the character is 🐱, I can find its ordinal value. I typically do that with a hex dump, but there are a variety of ways. It's U+1F431:
s/\x{1F431}//g;
You can also specify the code number in octal for the non-wide characters:
s/\200//g;
Or use those in a range in a character class:
s/[\200-\377]//g;
s/[\000-\037]//g;
s/[\000-\037\200-\377]//g;
But, there may be another thing you want to do. You might match it by a property it has (see perluniprops):
s/\p{XPosixCntrl}//g
Or, with an uppercase P, a property it doesn't have:
s/\P{Print}//g

Answer 2

So, we've learned two things from your hex dump of the file:
It's definitely a UTF8 file - as the first three characters are the UTF8 Byte Order Marker (or BOM).
Your problematic character is actually a null (with a codepoint of zero).
So, following brian's advice, you'll be able to remove the character with:
s/\0//g;

Answer 3

I'm not sure how you discovered it was ^#, but you should be able to refer to any such thing by using \c#, where the \c stands for "CTRL-". So, if it were a SOH, \x01, and you saw it displayed as ^A, you could specify s/\cA//g to get rid of it