How do I solve Golang filepath.walkfunc problem? - csv

I'm trying to solve a task where I must to find one file with data in CSV format among other files with similar names and same size and print a number on 5th row 3rd column (indexes 4 and 2)
So I wrote this code
package main
import (
"encoding/csv"
"fmt"
"os"
"path/filepath"
)
var s [][]string
func walkfunc(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
buf, err1 := os.Open(path)
if err1 == nil {
var err2 error
r := csv.NewReader(buf)
s, err2 = r.ReadAll()
if err2 == nil {
fmt.Printf("found: %v", s[4][2])
}
}
defer buf.Close()
return nil
}
func main() {
const root = "./task/"
if err := filepath.Walk(root, walkfunc); err != nil {
fmt.Printf("error: %v", err)
}
}
And I got this in output
GOROOT=/usr/local/go #gosetup
GOPATH=/usr/local/go/bin #gosetup
/usr/local/go/bin/go build -o /private/var/folders/j2/ybr0drz13yq31dc67zmvkb1w0000gn/T/GoLand/___go_build_qwasd3_go /Users/user/Downloads/zadacha/qwasd3.go #gosetup
/private/var/folders/j2/ybr0drz13yq31dc67zmvkb1w0000gn/T/GoLand/___go_build_qwasd3_go
panic: runtime error: index out of range [4] with length 3
goroutine 1 [running]:
main.walkfunc({0x14000018120?, 0x0?}, {0x14000098d88?, 0x10247fe40?}, {0x0?, 0x0?})
/Users/user/Downloads/zadacha/qwasd3.go:23 +0x28c
path/filepath.walk({0x14000018120, 0xe}, {0x1024c9cf8, 0x140000685b0}, 0x1024c9338)
/usr/local/go/src/path/filepath/path.go:433 +0xd0
path/filepath.walk({0x10248d4a8, 0x7}, {0x1024c9cf8, 0x140000684e0}, 0x1024c9338)
/usr/local/go/src/path/filepath/path.go:457 +0x1fc
path/filepath.Walk({0x10248d4a8, 0x7}, 0x1024c9338)
/usr/local/go/src/path/filepath/path.go:520 +0x6c
main.main()
/Users/user/Downloads/zadacha/qwasd3.go:37 +0x30
Process finished with the exit code 2
What am I doing wrong?
I was trying to run this code on MacBook.
The needed file contains table with numbers and I need to print a number on 5th row and 3rd column.

As other comments have pointed out, you need to check each CSV to make sure it's actually as big as you expect it to be. You could also add a simple check to try and make sure it's a CSV file before opening it by looking for a ".csv" extension.
Though, to directly address your error... The CSV reader may be able to interpret a plain txt file as CSV and not return an err, like:
buf := strings.NewReader(`A regular text file with 3 lines.
Line2
Line3
`)
r := csv.NewReader(buf)
records, err := r.ReadAll()
if err != nil {
fmt.Println("could not read all of CSV file!")
return err
}
fmt.Println(records)
prints:
[[A regular text file with 3 lines.] [Line2] [Line3]]
Just assuming that it's a CSV with the correct number of rows and columns:
fmt.Println("found", records[4][2])
gives the panic message you shared:
panic: runtime error: index out of range [4] with length 3
You at least need to check that your CSV has 5 rows, and if it does, then check if the 5th row has 3 columns before you try to read that field:
if len(records) < 5 {
fmt.Println(path, "does not have 5 rows")
return nil
}
if len(records[4]) < 3 {
fmt.Println(path, "5th row does not have 3 columns")
return nil
}
fmt.Println("found", records[4][2])
You could also do, inside your walkfunc, a basic check of the file path itself to see if it looks like a CSV:
if strings.ToLower(path[len(path)-4:]) != ".csv" {
fmt.Println(path, "is not a CSV")
return nil
}
I show all this code, plus a fully worked/integrated example in this Playground.

Related

Strange number when using fmt.Println in Golang

I'm new to Golang and have been doing alright but I have a strange issue that I have not encountered before when using fmt. This strange behavior is when I'm printing a string. At the end of the string (which has sub-strings) it is also printing out what appears to be the len() of each string although the number don't add up. Can anyone explain why this is happening and how to stop it?
Any help is greatly appreciated
Here is the code:
package main
import (
"fmt"
//"log"
"strings"
)
var e = "[{8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888 localhost:3303 4d50f447-7c93-42df-a03e-89c09626950a}]"
func main() {
tl := strings.Trim(e, "[{")
tr := strings.Trim(tl, "}]")
r := strings.TrimSpace(tr)
s := strings.Fields(r)
V_PK := s[0]
SERVER_ADDR := s[1]
A_KEY := s[2]
vv, _ := fmt.Printf("[{\"v_pk\": %q", V_PK)
pp, _ := fmt.Printf(",\"server_addr\": %q", SERVER_ADDR)
kk, _ := fmt.Printf(",\"a_key\": %q}] ", A_KEY)
rstr, _ := fmt.Println(vv, pp, kk)
stringc := string(rstr)
fmt.Println(stringc)
}
Expected output:
[{"v_pk": "8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888","server_addr": "localhost:3303","a_key": "4d50f447-7c93-42df-a03e-89c09626950a"}]
Actual output:
[{"v_pk": "8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888","server_addr": "localhost:3303","a_key": "4d50f447-7c93-42df-a03e-89c09626950a"}] 82 36 53
Why on earth would it be printing these string lengths on the end? It's probably obvious that I'm trying to build a JSON string so these numbers on the end are problematic when trying to import the string into a JSON interpreter.
Again, any help is appreciated!
Take a look at the documentation for fmt.Printf and its friends fmt.Println. The documentation reads:
Printf formats according to a format specifier and writes to standard output. It returns the number of bytes written and any write error encountered.
The line in your code
vv, _ := fmt.Printf("[{\"v_pk\": %q", V_PK)
prints the formatted string to standard output, then return the number of bytes written and stores that in vv. If you want to print the formatted string to standard output, just call fmt.Printf and ignore the output:
package main
import (
"fmt"
//"log"
"strings"
)
var e = "[{8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888 localhost:3303 4d50f447-7c93-42df-a03e-89c09626950a}]"
func main() {
tl := strings.Trim(e, "[{")
tr := strings.Trim(tl, "}]")
r := strings.TrimSpace(tr)
s := strings.Fields(r)
V_PK := s[0]
SERVER_ADDR := s[1]
A_KEY := s[2]
fmt.Printf("[{\"v_pk\": %q, \"server_addr\": %q, \"a_key\": %q}]\n", V_PK, SERVER_ADDR, A_KEY)
}
Or, if you want to store the formatted string to a new string variable, call fmt.Sprintf:
stringc := fmt.Sprintf("[{\"v_pk\": %q, \"server_addr\": %q, \"a_key\": %q}]", V_PK, SERVER_ADDR, A_KEY)
fmt.Println(stringc)
You can check out a working version at the playground.
You might also want to checkout the json package, which can do the parsing and serializing for you with properly defined structs:
package main
import (
"encoding/json"
"fmt"
)
func main() {
type Datum struct {
VPK string `json:"v_pk"`
Server string `json:"server_addr"`
AKey string `json:"a_key"`
}
data := []Datum{
{VPK: "8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888",
Server: "localhost:3303",
AKey: "4d50f447-7c93-42df-a03e-89c09626950a",
}}
json, err := json.MarshalIndent(data, "", " ")
if err != nil {
// deal with error
}
fmt.Println(string(json))
}
Check it out at the go playground.
fmt.Printf returns the number of bytes written. The variables vv, pp, kk are the number of bytes written by those three Printf calls, and the three numbers printed are those numbers.

How to set the output of the aws-sdk-go to "text"?

Although the output setting has been set to text
~/.aws/config
[default]
output=text
the aws-sdk-go returns json. The question is whether the output could be switched to text.
When:
aws route53 get-hosted-zone --id some-id
is run, the output looks as follows:
NAMESERVERS some-ns
NAMESERVERS some-ns1
NAMESERVERS some-ns2
NAMESERVERS some-ns3
According to the this AWS documentation one could set the configuration:
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-2")},
)
One attempt was to consult this Config struct, but an Output option seems to be omitted.
How to set the output to text?
Note: an issue has added to the github page of the aws-sdk-go as well.
Example
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/route53"
)
func main() {
session, err := session.NewSession()
if err != nil {
log.Fatal(err)
}
r53 := route53.New(session)
listParams := &route53.ListResourceRecordSetsInput{
HostedZoneId: aws.String("some-id"),
}
records, err := r53.ListResourceRecordSets(listParams)
if err != nil {
log.Fatal(err)
}
fmt.Println(records)
}
returns:
{
IsTruncated: false,
MaxItems: "100",
ResourceRecordSets: [
{
Name: "some-domain.",
ResourceRecords: [{
Value: "some-ip"
}],
TTL: 7200,
Type: "A"
}
}
while aws route53 list-resource-record-sets --hosted-zone-id some-id, results in:
RESOURCERECORDSETS some-domain. 7200 A
RESOURCERECORDS some-ip
Problem
While it is possible to set the format of the aws-cli to output, it does not seem to be possible to do the same for the SDK.
Question
How to let the go-aws-sdk return text rather than json?
I have all of the information you need, you just have to unravel it from the response (records).
To get similar results from the last cli command:
for _, recordSet := range records.ResourceRecordSets {
log.Println("RESOURCERECORDSETS " + *recordSet.Name + strconv.Itoa(int(*recordSet.TTL)) + *recordSet.Type)
for _, record := range recordSet.ResourceRecords {
log.Println("RESOURCERECORDS " + *record.Value)
}
log.Println("")
}

Golang empty Location on Mac OSX when parsing time

When decoding a timestamp field from JSON into a struct on my local OS X machine, the Location of the time.Time field is "empty" rather than UTC. This is problematic for me running unit tests locally (vs. on a CI server where the Location is being set correctly to be UTC).
Here's the example code: https://play.golang.org/p/pb3eMbjSmv
package main
import (
"fmt"
"time"
)
func main() {
// Ignoring the err just for this example's sake!
parsed, _ := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
which outputs
String(): 2017-08-15 22:30:00 +0000 +0000
Location():
So while the offset of the time.Time's Location appears to be correct, its timezone name is just an empty string. Running in on other machines (and The Go Playground) give the expected "UTC" location.
When I run that on my machine, I see
TimeField.String(): 2017-08-15 22:30:00 +0000 +0000
TimeField.Location():
So while the offset of the time.Time's Location appears to be correct, its timezone name is just an empty string. This is using Go 1.5:
go version go1.5 darwin/amd64
I find same behavior using my current setup on Mac and I suspect it will be same behavior on Linux (not sure through)
$ go version
go version devel +31ad583 Wed Aug 10 19:44:08 2016 +0000 darwin/amd64
To make it more deterministic, I suggest using a custom json Unmarshal like so:
package main
import (
"encoding/json"
"fmt"
"strings"
"time"
)
type Time struct {
*time.Time
}
func (t *Time) UnmarshalJSON(b []byte) error {
const format = "\"2006-01-02T15:04:05+00:00\""
t_, err := time.Parse(format, string(b))
if err != nil {
return err
}
*t = Time{&t_}
return nil
}
type Example struct {
TimeField *Time `json:"time_field"`
}
func main() {
inString := "{\"time_field\": \"2017-08-15T22:30:00+00:00\"}"
var ex Example
decoder := json.NewDecoder(strings.NewReader(inString))
decoder.Decode(&ex)
fmt.Printf("TimeField.String(): %v\n", ex.TimeField.String())
fmt.Printf("TimeField.Location(): %v\n", ex.TimeField.Location())
}
Yes, You are right. On The Go Playground the Local is set to UTC inside that sandbox:
Try this working sample code on The Go Playground:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
output on The Go Playground:
go1.7 amd64p32 nacl
String(): 2017-08-15 22:30:00 +0000 UTC
Location(): UTC
And try it on your local system, output Location() is empty.
You may use utc := parsed.UTC() with the location set to UTC, like this working sample code The Go Playground:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
utc := parsed.UTC()
fmt.Printf("String(): %v\n", utc.String())
fmt.Printf("Location(): %v\n", utc.Location())
}
Also You may use time.ParseInLocation(time.RFC3339, "2017-08-15T22:30:00+00:00", time.UTC), like this working sample code:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.ParseInLocation(time.RFC3339, "2017-08-15T22:30:00+00:00", time.UTC)
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
So the Location() will be UTC.
Thanks to #djd for pointing out that we can skip all the JSON/struct decoding business; the key issue is with time.Parse.
The same issue comes up here where the Location is "empty" rather than UTC (I would've expected UTC based on the docs: https://golang.org/pkg/time/#Parse
In the absence of a time zone indicator, Parse returns a time in UTC.
This answer was taken from the question as of revision 6.

Go panic: extra delimiter at end of line

I'm reading the MaxMind GeoIP Lite City locations CSV file using Go:
csvFile, err := os.Open("/path/GeoLiteCity_20130702/GeoLiteCity-Location.csv")
defer csvFile.Close()
if err != nil {
panic(err)
}
csvf := csv.NewReader(csvFile)
csvf.Read() // skip header row
for {
fields, err := csvf.Read()
if err == io.EOF {
break
} else if err != nil {
panic(err)
}
// does nothing yet
}
The error I'm getting is:
panic: line 2, column 22: extra delimiter at end of line
goroutine 1 [running]: main.main()
/path/myprogram.go:239
+0x108f
goroutine 2 [runnable]: exit status 2
The file is quite long, but starts with these lines:
locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode
1,O1,,,,0.0000,0.0000,,
2,AP,,,,35.0000,105.0000,,
3,EU,,,,47.0000,8.0000,,
4,AD,,,,42.5000,1.5000,,
5,AE,,,,24.0000,54.0000,,
6,AF,,,,33.0000,65.0000,,
7,AG,,,,17.0500,-61.8000,,
8,AI,,,,18.2500,-63.1667,,
9,AL,,,,41.0000,20.0000,,
It appears to be properly formatted. Each row has 9 fields.
Line 239 is my line invoking the panic, panic(err). As you can see, it's failing on line 2 of the CSV file, which happens in the first iteration of the loop (line 1 is read before the loop, to skip the header row). Column 22 of line 2 is the second-to-last comma.
Am I missing something here? I don't see any trailing comma... (clarification: the commas at the end of each line must be there to indicate empty field values, so they're not trailing, as in, extra.)
UPDATE: The gophers have resolved this issue and the fix ships with Go 1.1.2.
There are even two trailing commas on each line.
Try setting csv.Reader.TrailingComma = true.
It really often helps taking a look at the source or at least the package documentation :-)
Here is a complete example for you. The key is csvf.TrailingComma = true.
package main
import (
"bytes"
"encoding/csv"
"fmt"
"io"
)
var csvData = `locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode
1,O1,,,,0.0000,0.0000,,
2,AP,,,,35.0000,105.0000,,
3,EU,,,,47.0000,8.0000,,
4,AD,,,,42.5000,1.5000,,
5,AE,,,,24.0000,54.0000,,
6,AF,,,,33.0000,65.0000,,
7,AG,,,,17.0500,-61.8000,,
8,AI,,,,18.2500,-63.1667,,
9,AL,,,,41.0000,20.0000,,
`
func main() {
csvFile := bytes.NewBufferString(csvData)
csvf := csv.NewReader(csvFile)
csvf.TrailingComma = true
csvf.Read() // skip header row
for {
fields, err := csvf.Read()
if err == io.EOF {
break
} else if err != nil {
panic(err)
}
// does nothing yet
fmt.Println(fields)
}
}

sending JSON with go

I'm trying to send a JSON message with Go.
This is the server code:
func (network *Network) Join(
w http.ResponseWriter,
r *http.Request) {
//the request is not interesting
//the response will be a message with just the clientId value set
log.Println("client wants to join")
message := Message{-1, -1, -1, ClientId(len(network.Clients)), -1, -1}
var buffer bytes.Buffer
enc := json.NewEncoder(&buffer)
err := enc.Encode(message)
if err != nil {
fmt.Println("error encoding the response to a join request")
log.Fatal(err)
}
fmt.Printf("the json: %s\n", buffer.Bytes())
fmt.Fprint(w, buffer.Bytes())
}
Network is a custom struct. In the main function, I'm creating a network object and registering it's methods as callbacks to http.HandleFunc(...)
func main() {
runtime.GOMAXPROCS(2)
var network = new(Network)
var clients = make([]Client, 0, 10)
network.Clients = clients
log.Println("starting the server")
http.HandleFunc("/request", network.Request)
http.HandleFunc("/update", network.GetNews)
http.HandleFunc("/join", network.Join)
log.Fatal(http.ListenAndServe("localhost:5000", nil))
}
Message is a struct, too. It has six fields all of a type alias for int.
When a client sends an http GET request to the url "localhost:5000/join", this should happen
The method Join on the network object is called
A new Message object with an Id for the client is created
This Message is encoded as JSON
To check if the encoding is correct, the encoded message is printed on the cmd
The message is written to the ResponseWriter
The client is rather simple. It has the exact same code for the Message struct. In the main function it just sends a GET request to "localhost:5000/join" and tries to decode the response. Here's the code
func main() {
// try to join
var clientId ClientId
start := time.Now()
var message Message
resp, err := http.Get("http://localhost:5000/join")
if err != nil {
log.Fatal(err)
}
fmt.Println(resp.Status)
dec := json.NewDecoder(resp.Body)
err = dec.Decode(&message)
if err != nil {
fmt.Println("error decoding the response to the join request")
log.Fatal(err)
}
fmt.Println(message)
duration := time.Since(start)
fmt.Println("connected after: ", duration)
fmt.Println("with clientId", message.ClientId)
}
I've started the server, waited a few seconds and then ran the client. This is the result
The server prints "client wants to join"
The server prints "the json: {"What":-1,"Tag":-1,"Id":-1,"ClientId":0,"X":-1,"Y":-1}"
The client prints "200 OK"
The client crashes "error decoding the response to the join request"
The error is "invalid character "3" after array element"
This error message really confused me. After all, nowhere in my json, there's the number 3. So I imported io/ioutil on the client and just printed the response with this code
b, _ := ioutil.ReadAll(resp.Body)
fmt.Printf("the json: %s\n", b)
Please note that the print statement is the same as on the server. I expected to see my encoded JSON. Instead I got this
"200 OK"
"the json: [123 34 87 104 97 116 ....]" the list went on for a long time
I'm new to go and don't know if i did this correctly. But it seems as if the above code just printed the slice of bytes. Strange, on the server the output was converted to a string.
My guess is that somehow I'm reading the wrong data or that the message was corrupted on the way between server and client. But honestly these are just wild guesses.
In your server, instead of
fmt.Fprint(w, buffer.Bytes())
you need to use:
w.Write(buffer.Bytes())
The fmt package will format the Bytes() into a human-readable slice with the bytes represented as integers, like so:
[123 34 87 104 97 116 ... etc
You don't want to use fmt.Print to write stuff to the response. Eg
package main
import (
"fmt"
"os"
)
func main() {
bs := []byte("Hello, playground")
fmt.Fprint(os.Stdout, bs)
}
(playground link)
Produces
[72 101 108 108 111 44 32 112 108 97 121 103 114 111 117 110 100]
Use the Write() method of the ResponseWriter instead
You could have found this out by telneting to your server as an experiment - always a good idea when you aren't sure what is going on!