When I try to read a csv file in Octave I realize that the very first value from it is converted to zero. I tried both csvread and dlmread and I'm receiving no errors. I am able to open the file in a plain text editor and I can see the correct value there. From what I can tell, there are no funny hidden characters, spacings, or similar in the csv file. Files also contain only numbers. The only thing that I feel might be important is that I have five columns/groups that each have different number of values in them.
I went through the commands' documentation on Octave Forge and I do not know what may be causing this. Does anyone have an idea what I can troubleshoot?
To try to illustrate the issue, if I try to load a file with the contents:
1.1,2.1,3.1,4.1,5.1
,2.2,3.2,4.2,5.2
,2.3,3.3,4.3,
,,3.4,4.4
,,3.5,
Command window will return:
0.0,2.1,3.1,4.1,5.1
,2.2,3.2,4.2,5.2
,2.3,3.3,4.3,
,,3.4,4.4
,,3.5,
( with additional trailing zeros after the decimal point).
Command syntaxes I'm using are:
dt = csvread("FileName.csv")
and
dt = dlmread("FileName.csv",",")
and they both return the same.
Your csv file contains a Byte Order Mark right before the first number. You can confirm this if you open the file in a hex editor, you will see the sequence EF BB BF before the numbers start.
This causes the first entry to be interpreted as a 'string', and since strings are parsed based on whether there are numbers in 'front' of the string sequence, this is parsed as the number zero. (see also this answer for more details on how csv entries are parsed).
In my text editor, if I start at the top left of the file, and press the right arrow key once, you can tell that the cursor hasn't moved (meaning I've just gone over the invisible byte order mark, which takes no visible space). Pressing backspace at this point to delete the byte order mark allows the csv to be read properly. Alternatively, you may have to fix your file in a hex editor, or find some other way to convert it to a proper Ascii file (or UTF without the byte order mark).
Also, it may be worth checking how this file was produced; if you have any control in that process, perhaps you can find why this mark was placed in the first place and prevent it. E.g., if this was exported from Excel, you can choose plain 'csv' format instead of 'utf-8 csv'.
UPDATE
In fact, this issue seems to have already been submitted as a bug and fixed in the development branch of octave. See #58813 :)
I've got a very large .csv file, which contains 10 million lines of data. The file size is around 250 MB. Each line contains three values and looks like this:
-9.8199980e-03,183,-4.32
I want to delete every 2nd line or e.g. copy every 10th line straight into a new file. Which program should I use and can you also post the code?
I tried it with Scilab and Excel; they couldn't open the file or just a small part of it. I can open the file in Notepad++, but when I tried to record and run a macro, which deletes every 2nd line, it crashed.
I would recommend you install gawk/awk from here and harness the power of this brilliant tool.
If you want every other line:
gawk "NR%2" original.csv > new.csv
If you want every 10th line:
gawk 'NR%10==0" original.csv > new.csv
Is there an efficient command-line tool for prepending lines to a file inside a ZIP archive?
I have several large ZIP files containing CSV files missing their header, and I need to insert the header line. It's easy enough to write a script to extract them, prepend the header, and then re-compress, but the files are so large, it takes about 15 minutes to extract each one. Is there some tool that can edit the ZIP in-place without extracting?
Fast answer, no.
A zip file contains 1 to N file entries inside and all of them works as un splitable units, meaning that if you want to do something on an entry, you need to process this entry completely (i.e. extracting).
The only fast operation you can do is adding a new file to your archive. It will create a new entry and append it to the file, but this is probably not what you need
i'm trying to find out who did the last change on a certain line in a large xml file ( ~100.00 lines ).
I tried to use the log file viewer of thg but it does not give me any results.
Using hg annote file.xml -l 95000 .. took forever and eventually died with an error message.
Is there a way to annote a single line in a large file that does not take forever ?
You can use hg grep to dig into a file if you have interest in very specific text:
hg grep "text-to-search" file.txt
You will likely need to add the --all switch to get every change that matches and then limit your results to a specific changeset range with -r firstchage:lastchange syntax.
I don't have a file on hand of the size you are working with, so this may also have trouble past a certain point, particularly if the search string matches many many lines in the file. But if you can get as specific as possible with your search string, you should be able to track it down.
I'm new to mercurial, i have a certain revision with me and i would like to switch to that particular revision and save the change-set of a particular file with line number. Thank You
I don't know a simple way to do this. Mercurial has a method where it calculates the diff between changesets and then it applies a formatter to this to print the data.
But your requirement is more complex than it looks. Imagine you have two changes in a file. In version 2, a couple of lines at the beginning have been deleted and then a line near the end has been changed.
Questions:
How do you plan to assign line numbers to the deleted lines? Omit them or use the original line numbers from version 1?
How about the lines after the deleted lines? Do you want to show the new line numbers or the original ones?
Which line numbers are you going to show for the changes near the end?
Of course, you could show both but that would need a lot of parsing in your head.
Some HTML-based changeset viewers use this approach: https://bitbucket.org/digulla/ts-html/commits/62fc23841ff7e7cce95eefa85244a2b821f92ba2
But I haven't see something similar for the command line since it would waste 15-20 columns of text.