Can someone tell me how to remove special character like '\r\n' withing double quotes in a csv fiel?
I basically need a unix command or script which can do the job.
My csv file looks like this:-
ID,Name,Age,Address,Street
1,Raul,45,"NewYork",1211
2,David,55,"New
York",4576
3,Charles,32,"Boston",656
4,"Julia
Jones",31,"Boston",7567
I want my file output to be like this:-
ID,Name,Age,Address,Street
1,Raul,45,"NewYork",1211
2,David,55,"New York",4576
3,Charles,32,"Boston",656
4,"Julia Jones",31,"Boston",7567
I tries some commands like sed,awk but somehow couldnt find a solution.
Thanks in Advance.
Use the -z flag to consume the line as one line, allowing you to process the carriage returns:
sed -zi 's/\n//' file
I have the following csv where I have to replace the thousand comma separator with nothing. In example below, when I have the amount "1,000.00" I should have 1000.00 (no comma, no quotes) instead.
I use JREPL to remove header from my csv
jrepl "(?:.*\n){1,1}([\s\S]*)" "$1" /m /f "csv/Transactions.csv" /o "csv/Transactionsfeed.csv")
I was wondering if I could do the process of removing header + dealing with the thousand comma in one step.
I am also opened to the option of doing it with another command in a second step...
Tnx ID,Trace ID - Gateway,Profile,Customer PIN,Customer,Ext. ID,Identifier,Amount,Chrg,Curr,Processor,Type,Status,Created By,Date Created,RejectReason
1102845,3962708,SL,John,Mohammad Alo,NA,455015*****9998,900.00,900.00,$,Un,Credit Card,Rejected,Internet,2016-05-16 06:54:10,"-330: Fail by bank, try again later(refer to acquirer)"
1102844,3962707,SL,John,Mohammad Alo,NA,455015*****9998,"1,000.00","1,000.00",$,Un,Credit Card,Rejected,Internet,2016-05-16 06:52:26,"-330: Fail by bank, try again later(refer to acquirer)"
Yes, there is a very efficient and fairly compact and straight-forward solution:
jrepl "\q(\d{1,3}(?:,\d{3})*(?:\.\d*)*)\q" "$1.replace(/,/g,'')" /x /j /jendln "if (ln==1) $txt=false" /f "csv/Transactions.csv" /o "csv/Transactionsfeed.csv"
The /JENDLN JScript expression strips the header line by setting $txt to false if it is the first line.
The search string matches any quoted number that contains commas as thousand separators, and $1 is the number without the quotes.
The replace string is a JScript expression that replaces all commas in the matching $1 number with nothing.
EDIT
Note that the above will likely work with any CSV that you are likely to have. However, it would fail if you have a quoted field that contains a quoted number string literal. Something like the following would yield a corrupted CSV with the code above:
...,"some text ""123,456.78"" more text",...
This issue can be fixed with a bit more regex code. You only want to modify a quoted number if the opening quote is preceded by a comma or the beginning of the line, and the closing quote should be followed by a comma or the end of line.
A look-ahead assertion can be used for the trailing comma/EOL. But JREPL does not support look-behind. So the leading comma/BOL must be captured and preserved in the replacement
jrepl "(^|,)\q(\d{1,3}(?:,\d{3})*(?:\.\d*)*)\q(?=$|,)" "$1+$2.replace(/,/g,'')" /x /j /jendln "if (ln==1) $txt=false" /f "csv/Transactions.csv" /o "csv/Transactionsfeed.csv"
EDIT in response to changing requirement in comment
The following will simply remove all quotes and commas from quoted CSV fields. I don't like this concept, and I suspect there is a much better way to handle this for import into mysql, but this is what the OP is asking for.
jrepl "(^|,)(\q(?:[^\q]|\q\q)*\q)(?=$|,)" "$1+$2.replace(/,|\x22/g,'')" /x /j /jendln "if (ln==1) $txt=false" /f "csv/Transactions.csv" /o "csv/Transactionsfeed.csv"
May I suggest you a different, simpler solution? The 5-lines Batch file below do what you want; save it with .bat extension:
#set #a=0 /*
#cscript //nologo //E:JScript "%~F0" < "csv/Transactions.csv" > "csv/Transactionsfeed.csv"
#goto :EOF */
WScript.Stdin.ReadLine();
WScript.Stdout.Write(WScript.Stdin.ReadAll().replace(/(\"(\d{1,3}),(\d{3}\.\d{2})\")/g,"$2$3"));
JREPL.BAT is a large and complex program capable of advanced replacement tasks; however, your request is very simple. This code is also a Batch-JScript hybrid script that use the replace method in the same way as JREPL.BAT, but that is tailored to your specific request.
The first ReadLine() read the header line of the input file, so the posterior ReadAll() read and process the rest of lines.
The regexp (\"(\d{1,3}),(\d{3}\.\d{2})\") define 3 submatches enclosed in parentheses: the first one is the whole number enclosed in quotes, like "1,000.00"; the second submatch is the digits before the comma and the third submatch is the digits after the comma, including the decimal point.
The .replace method change the previous regexp, that is, the whole number enclosed in quotes by just the second and third submatches.
I try to import some large csv dataset into neo4j using the neo4j-import tool. Quotation is not used anywhere, and therefore i get errors when parsing using --quote " --quote ' --quote ยด and alike. even choosing very rare unicode chars doesnt help with this multi-gig csv because it also contains arabic letters, math symbols and everything you can imagine.
So: Is there a way to disable the quotation checking completely?
Perhaps it would be useful to have the import tool able to accept character configuration values specifying ASCII codes. If so then you could specify --quote \0 and no character would match. That would also be useful for specifying other special characters in general I'd guess.
You need to make sure the CSV file uses quotation marks, since they allow the tool to reliably determine when strings end.
Any string in your data file might contain the delimiter character (a comma, by default). Even if there were a way to turn off quotation checking, the tool would treat every delimiter character as the end of a field. Therefore, any string field that happened to contain the delimiter character would be terminated prematurely, causing errors.
i am creating csv files with php. To write the data into my csv file, i use the php function "fputcsv".
this is the issue:
i can open the created file normally with Excel. But i cant import the file into a shopsystem (in this case "shopware"). It says something like "the data could not be read".
And now comes the clue:
If i open the created file and choose "save as" and select "CSV (comma delimited)" in type, this file can be imported into shopware. I read something about the php function "mb_convert_encoding" which i used to encode the data, but it could not fix the problem.
I will be very glad if you can help me.
thanks.
Thanks for your input.
I solved this problem by replacing fputcsv with fwrite. Then i just needed to add "\r\n" (thanks wmil) to the end of the line and the generated file can be read by shopware.
Obviously the fputcsv function uses \n and not \r\n as EOL character.
I think you cannot set the encode using fputcsv. However fputcsv looks to the locale setting, wich you can change with setlocale.
Maybe you could send your file directly to the users browser and use changing contenttype and charset with header function.
This can't be answered without knowing more about your system. Most likely it has nothing to do with character encoding. It's probably a problem with wrong number of columns or column headers being incorrect.
If it is a character encoding issue, your best bet is:
$new_str = mb_convert_encoding($str, 'Windows-1252', 'auto');
Also end newlines with \r\n, not just \n.
If that doesn't work you'll need to check the software docs.