Parsing CSV file which includes a header block of "comments" - csv

Question

I'm trying to parse a CSV file hosted in a remote location, but annoyingly the file contains some readme comments atop the file, in the following format:
######
# some readme text,
# some more, comments
######
01-02-03,123,foo,http://example.com
04-05-06,789,baz,http://another.com
I'm attempting to use the following code in order to extract the URLs within the data, but it throws an error saying wrong number of fields due to the comments at the top, presumably it's trying to parse them as CSV content.
type myData struct {
URL string `json:"url"`
}
func doWork() ([]myData, error) {
rurl := "https://example.com/some.csv"
out := make([]myData, 0)
resp, err := http.Get(rurl)
if err != nil {
return []myData{}, err
}
defer resp.Body.Close()
reader := csv.NewReader(resp.Body)
reader.Comma = ','
data, err := reader.ReadAll()
if err != nil {
return []myData{}, err
}
for _, row := range data {
out = append(out, myData{URL: row[4]})
}
return out, nil
}
func main() {
data, err := doWork()
if err != nil {
panic(err)
}
// do something with data
}
Is there a way to skip over the first N lines of the remote file, or have it ignore lines which start with a #

Answer 1

Oh, actually I just realised I can add this:
reader.Comment = '#' // ignores the line starting with '#'
Which works perfectly with my current code, but appreciate the other suggestions.

Answer 2

Well, the approach is simple: do not try to interpret the lines starting with '#' as part of the CSV stream; instead, consider the whole stream of data as a concatenation of two streams: the header and the actual CSV payload.
The easiest approach probably is to employ the fact bufio.Reader is able to read lines from its underlying stream and it itself is an io.Reader so you can make a csv.Reader to read from it instead of the source stream.
So, you could roll like this (not real code, untested):
import (
"bufio"
"encoding/csv"
"io"
"strings"
)
func parse(r io.Reader) ([]MyData, error) (
br := bufio.NewReader(r)
var line string
for {
s, err := br.ReadString('\n')
if err != nil {
return nil, err
}
if len(s) == 0 || s[0] != '#' {
line = s
break
}
}
// At this point the line variable contains the 1st line of the CSV stream.
// Let's create a "multi reader" which reads first from that line
// and then — from the rest of the CSV stream.
cr := csv.NewReader(io.MultiReader(strings.NewReader(line), br))
cr.Comma = ','
data, err := cr.ReadAll()
if err != nil {
return nil, err
}
for _, row := range data {
out = append(out, myData{URL: row[4]})
}
return out, nil
}