Big CSV parsing [closed] - csv

Question

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 5 years ago.
Improve this question
I'm having quite a hard time to figure out a robust and light algorithm for post-processing some big CSV files. Here's a minimal example of how they look like:
Time a b c
0 2.9 1.6 4.1
0 3.6 1.1 0.5
0 3.4 0.2 1.7
1.2 0.1 4.2 1.9
1.201 2.3 3.1 4.8
9.99 0.2 0.8 1.2
10 3.1 3.3 2.3
10 3.6 3.5 3.0
10.01 1.1 4.5 3.9
10.01 2.2 3.0 2.3
17 4.3 2.3 3.8
20 1.0 3.2 3.0
30 4.1 3.0 4.9
40 3.8 3.3 1.6
I need to postprocess my CSV based on those rules:
only lines whose time is a multiple of 10 need to be considered
if multiple line have the same time tamp, take the average value of each column across the different rows
Here's the output I'd like to get:
Time a b c
0 3.3 0.97 2.1
10 2.04 3.02 2.54
20 1.0 3.2 3.0
30 4.1 3.0 4.9
40 3.8 3.3 1.6
Now the constraint: my script needs to handle pretty big CSV (up to few hundreds MB) on a Windows machine without much memory available for this. Because of that, I'm not keen on storing all the CSV in a big array of dictionary, but I'd prefer to do this row by row.
Here's my first naive attempt. It's very poor and not properly working. (Small margin note: the average is not a true average but a kind of weird "running average". Bear with me here, I was trying to assess the workflow and don't really care for the numbers at this stage.)
filename = "test"
sampling_time = 10.0
tolerance = 1e-1
Dim FSO, input, output
Const ForReading = 1
Const ForWriting = 2
'Create the objects
Set FSO = CreateObject("Scripting.FileSystemObject")
Set input = FSO.OpenTextFile(filename & ".csv", ForReading, False)
Set output = FSO.OpenTextFile(filename & "_output.csv", ForWriting, True)
'First line: write headers
s = input.ReadLine()
output.WriteLine s
'Second line: initialize sSplit_old
s = input.ReadLine()
sSplit = Split(s, ",")
sSplit_old = sSplit
'Keep reading...
Do Until input.AtEndOfStream
'read new line and split it into its components
'this is needed to read the first element of the line, i.e. the time
s = input.ReadLine()
sSplit = Split(s, ",")
'If the remainder of time/sampling_time is below the tolerance then the
'line has to be processed.
'Here the "\" operator (i.e. the integer division: 5\2=2, while 5/2=2.5)
'is used as the "Mod" operator return integer remainders.
If CDbl(sSplit(0))-sampling_time*(CDbl(sSplit(0))\sampling_time) < tolerance Then
'If the current time is close to the previous one (within a tolerance)...
If Abs(CDbl(sSplit(0))-CDbl(sSplit_old(0))) < tolerance Then
'... cycle through the arrays and store the average
For i = 0 To UBound(sSplit)
sSplit_old(i) = (CDbl(sSplit(i)) + CDbl(sSplit_old(i))) / 2.0
Next
Else
'... otherwise just write the previous time and save the current
'one to compare it to the next one
s = Join(sSplit_old, ",")
output.WriteLine s
sSplit_old = sSplit
End If
End If
Loop
output.WriteLine s
input.Close
output.Close

Answer 1

When you paid (too much) for your Windows OS, you also paid for a SQL engine. So use it:
Option Explicit
Dim db : Set db = CreateObject("ADODB.Connection")
Dim dd : dd = "E:\work\proj\soa\47155733\data"
Dim cs
If "AMD64" = CreateObject("WScript.Shell").ExpandEnvironmentStrings("%PROCESSOR_ARCHITECTURE%") Then
cs = "Driver=Microsoft Access Text Driver (*.txt, *.csv);Dbq=" & dd & ";Extensions=asc,csv,tab,txt;"
WScript.Echo "64 Bit:", cs
Else
cs = "Driver={Microsoft Text Driver (*.txt; *.csv)};Dbq=" & dd & ";Extensions=asc,csv,tab,txt;"
WScript.Echo "32 Bit:", cs
End If
db.Open cs
Dim ss : ss = "SELECT * FROM [47155733.txt]"
WScript.Echo ss
WScript.Echo db.Execute(ss).GetString(2,,vbTab,vbCrlf,"*")
ss = "SELECT t, avg(a), avg(b), avg(c) FROM [47155733.txt]" _
& " WHERE t = Int(t) And 0.0 = t Mod 10 GROUP BY t"
WScript.Echo ss
WScript.Echo db.Execute(ss).GetString(2,,vbTab,vbCrlf,"*")
ss = "SELECT Round(1/3, 3)"
WScript.Echo ss
WScript.Echo db.Execute(ss).GetString(2,,vbTab,vbCrlf,"*")
output:
cscript 47155733.vbs
SELECT * FROM [47155733.txt]
0 2,9 1,6 4,1
0 3,6 1,1 0,5
0 3,4 0,2 1,7
1,2 0,1 4,2 1,9
1,201 2,3 3,1 4,8
9,99 0,2 0,8 1,2
10 3,1 3,3 2,3
10 3,6 3,5 3
10,01 1,1 4,5 3,9
10,01 2,2 3 2,3
17 4,3 2,3 3,8
20 1 3,2 3
30 4,1 3 4,9
40 3,8 3,3 1,6
SELECT t, avg(a), avg(b), avg(c) FROM [47155733.txt] WHERE t = Int(t) And 0.0 = t Mod 10 GROUP BY t
0 3,3 0,966666666666667 2,1
10 3,35 3,4 2,65
20 1 3,2 3
30 4,1 3 4,9
40 3,8 3,3 1,6
SELECT Round(1/3, 3)
0,333
Tested for 32 and 64 bit on Windows 10; German locale. I prefer to specify the file format in a schema.ini file:
[47155733.txt]
Format=Delimited(,)
ColNameHeader=True
DecimalSymbol=.
Col1=t Double
Col2=a Double
Col3=b Double
Col4=c Double
Background:
Connectionstrings, odbc Connectionstrings, Driver download.