Golang empty Location on Mac OSX when parsing time - json

When decoding a timestamp field from JSON into a struct on my local OS X machine, the Location of the time.Time field is "empty" rather than UTC. This is problematic for me running unit tests locally (vs. on a CI server where the Location is being set correctly to be UTC).
Here's the example code: https://play.golang.org/p/pb3eMbjSmv
package main
import (
"fmt"
"time"
)
func main() {
// Ignoring the err just for this example's sake!
parsed, _ := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
which outputs
String(): 2017-08-15 22:30:00 +0000 +0000
Location():
So while the offset of the time.Time's Location appears to be correct, its timezone name is just an empty string. Running in on other machines (and The Go Playground) give the expected "UTC" location.
When I run that on my machine, I see
TimeField.String(): 2017-08-15 22:30:00 +0000 +0000
TimeField.Location():
So while the offset of the time.Time's Location appears to be correct, its timezone name is just an empty string. This is using Go 1.5:
go version go1.5 darwin/amd64

I find same behavior using my current setup on Mac and I suspect it will be same behavior on Linux (not sure through)
$ go version
go version devel +31ad583 Wed Aug 10 19:44:08 2016 +0000 darwin/amd64
To make it more deterministic, I suggest using a custom json Unmarshal like so:
package main
import (
"encoding/json"
"fmt"
"strings"
"time"
)
type Time struct {
*time.Time
}
func (t *Time) UnmarshalJSON(b []byte) error {
const format = "\"2006-01-02T15:04:05+00:00\""
t_, err := time.Parse(format, string(b))
if err != nil {
return err
}
*t = Time{&t_}
return nil
}
type Example struct {
TimeField *Time `json:"time_field"`
}
func main() {
inString := "{\"time_field\": \"2017-08-15T22:30:00+00:00\"}"
var ex Example
decoder := json.NewDecoder(strings.NewReader(inString))
decoder.Decode(&ex)
fmt.Printf("TimeField.String(): %v\n", ex.TimeField.String())
fmt.Printf("TimeField.Location(): %v\n", ex.TimeField.Location())
}

Yes, You are right. On The Go Playground the Local is set to UTC inside that sandbox:
Try this working sample code on The Go Playground:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
output on The Go Playground:
go1.7 amd64p32 nacl
String(): 2017-08-15 22:30:00 +0000 UTC
Location(): UTC
And try it on your local system, output Location() is empty.
You may use utc := parsed.UTC() with the location set to UTC, like this working sample code The Go Playground:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
utc := parsed.UTC()
fmt.Printf("String(): %v\n", utc.String())
fmt.Printf("Location(): %v\n", utc.Location())
}
Also You may use time.ParseInLocation(time.RFC3339, "2017-08-15T22:30:00+00:00", time.UTC), like this working sample code:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.ParseInLocation(time.RFC3339, "2017-08-15T22:30:00+00:00", time.UTC)
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
So the Location() will be UTC.

Thanks to #djd for pointing out that we can skip all the JSON/struct decoding business; the key issue is with time.Parse.
The same issue comes up here where the Location is "empty" rather than UTC (I would've expected UTC based on the docs: https://golang.org/pkg/time/#Parse
In the absence of a time zone indicator, Parse returns a time in UTC.
This answer was taken from the question as of revision 6.

Related

How do I solve Golang filepath.walkfunc problem?

I'm trying to solve a task where I must to find one file with data in CSV format among other files with similar names and same size and print a number on 5th row 3rd column (indexes 4 and 2)
So I wrote this code
package main
import (
"encoding/csv"
"fmt"
"os"
"path/filepath"
)
var s [][]string
func walkfunc(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
buf, err1 := os.Open(path)
if err1 == nil {
var err2 error
r := csv.NewReader(buf)
s, err2 = r.ReadAll()
if err2 == nil {
fmt.Printf("found: %v", s[4][2])
}
}
defer buf.Close()
return nil
}
func main() {
const root = "./task/"
if err := filepath.Walk(root, walkfunc); err != nil {
fmt.Printf("error: %v", err)
}
}
And I got this in output
GOROOT=/usr/local/go #gosetup
GOPATH=/usr/local/go/bin #gosetup
/usr/local/go/bin/go build -o /private/var/folders/j2/ybr0drz13yq31dc67zmvkb1w0000gn/T/GoLand/___go_build_qwasd3_go /Users/user/Downloads/zadacha/qwasd3.go #gosetup
/private/var/folders/j2/ybr0drz13yq31dc67zmvkb1w0000gn/T/GoLand/___go_build_qwasd3_go
panic: runtime error: index out of range [4] with length 3
goroutine 1 [running]:
main.walkfunc({0x14000018120?, 0x0?}, {0x14000098d88?, 0x10247fe40?}, {0x0?, 0x0?})
/Users/user/Downloads/zadacha/qwasd3.go:23 +0x28c
path/filepath.walk({0x14000018120, 0xe}, {0x1024c9cf8, 0x140000685b0}, 0x1024c9338)
/usr/local/go/src/path/filepath/path.go:433 +0xd0
path/filepath.walk({0x10248d4a8, 0x7}, {0x1024c9cf8, 0x140000684e0}, 0x1024c9338)
/usr/local/go/src/path/filepath/path.go:457 +0x1fc
path/filepath.Walk({0x10248d4a8, 0x7}, 0x1024c9338)
/usr/local/go/src/path/filepath/path.go:520 +0x6c
main.main()
/Users/user/Downloads/zadacha/qwasd3.go:37 +0x30
Process finished with the exit code 2
What am I doing wrong?
I was trying to run this code on MacBook.
The needed file contains table with numbers and I need to print a number on 5th row and 3rd column.
As other comments have pointed out, you need to check each CSV to make sure it's actually as big as you expect it to be. You could also add a simple check to try and make sure it's a CSV file before opening it by looking for a ".csv" extension.
Though, to directly address your error... The CSV reader may be able to interpret a plain txt file as CSV and not return an err, like:
buf := strings.NewReader(`A regular text file with 3 lines.
Line2
Line3
`)
r := csv.NewReader(buf)
records, err := r.ReadAll()
if err != nil {
fmt.Println("could not read all of CSV file!")
return err
}
fmt.Println(records)
prints:
[[A regular text file with 3 lines.] [Line2] [Line3]]
Just assuming that it's a CSV with the correct number of rows and columns:
fmt.Println("found", records[4][2])
gives the panic message you shared:
panic: runtime error: index out of range [4] with length 3
You at least need to check that your CSV has 5 rows, and if it does, then check if the 5th row has 3 columns before you try to read that field:
if len(records) < 5 {
fmt.Println(path, "does not have 5 rows")
return nil
}
if len(records[4]) < 3 {
fmt.Println(path, "5th row does not have 3 columns")
return nil
}
fmt.Println("found", records[4][2])
You could also do, inside your walkfunc, a basic check of the file path itself to see if it looks like a CSV:
if strings.ToLower(path[len(path)-4:]) != ".csv" {
fmt.Println(path, "is not a CSV")
return nil
}
I show all this code, plus a fully worked/integrated example in this Playground.

Parse dpkg command output to JSON in Go

I'm trying to convert exec output to json format. I just want to read a single string from the output, what would be the right way to do it. I thought about jq also but didn't work for me.
Here is the code snippet:
package main
import (
"fmt"
"encoding/json"
"os/exec"
)
type pkg struct {
Package string `json:"package"`
Version string `json:"version:"`
}
func main() {
fmt.Println("Hello, 世界")
cmd := exec.Command("dpkg", "-s", "tar")
output, err := cmd.CombinedOutput()
if err != nil {
fmt.Printf("%v", err)
}
var xyz pkg
if err := json.Unmarshal([]byte(output), &xyz); err != nil {
fmt.Printf("\n%v %v\n", string(output), err)
}
fmt.Println(xyz)
}
Command output:
$ dpkg -s tar
Package: tar
Essential: yes
Status: install ok installed
Priority: required
Section: utils
Installed-Size: 3152
Maintainer: Janos Lenart <ocsi#debian.org>
Architecture: amd64
Multi-Arch: foreign
Version: 1.34+dfsg-1
Replaces: cpio (<< 2.4.2-39)
Pre-Depends: libacl1 (>= 2.2.23), libc6 (>= 2.28), libselinux1 (>= 3.1~)
Suggests: bzip2, ncompress, xz-utils, tar-scripts, tar-doc
Breaks: dpkg-dev (<< 1.14.26)
Conflicts: cpio (<= 2.4.2-38)
Description: GNU version of the tar archiving utility
Tar is a program for packaging a set of files as a single archive in tar
format. The function it performs is conceptually similar to cpio, and to
things like PKZIP in the DOS world. It is heavily used by the Debian package
management system, and is useful for performing system backups and exchanging
sets of files with others.
Homepage: https://www.gnu.org/software/tar/
I'm getting the following error on json.Unmarshal:
invalid character 'P' looking for beginning of value
A simple solution could be use dpkg-query to format your desired output in json format:
package main
import (
"fmt"
"encoding/json"
"os/exec"
)
type pkg struct {
Package string `json:"package"`
Version string `json:"version"`
}
func main() {
cmd := exec.Command("dpkg-query", "-W", "-f={ \"package\": \"${Package}\", \"version\": \"${Version}\" }", "tar")
output, err := cmd.CombinedOutput()
if err != nil {
fmt.Printf("%v", err)
}
fmt.Println(string(output))
var xyz pkg
if err := json.Unmarshal([]byte(output), &xyz); err != nil {
fmt.Printf("\n%v %v\n", string(output), err)
}
fmt.Println(xyz.Package)
fmt.Println(xyz.Version)
}
{ "package": "tar", "version": "1.34+dfsg-1" }
tar
1.34+dfsg-1

Error on importing go-sql-driver/sql from github repository

As the title says, I have an error when importing go-mysql-driver package. I have installed the go-my-sql driver in my machine but the error still persists. I use XAMPP for local hosting and here’s the block of program.
package model
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
type Table interface {
Name() string
Field() ([]string, []interface{})
}
func Connect(username string, password string, host string, database string) (*sql.DB, error) {
conn := fmt.Sprintf("%s:%s#tcp(%s:3306)/%s", username, password, host, database)
db, err := sql.Open("mysql", conn)
return db, err
}
func CreateDB(db *sql.DB, name string) error {
query := fmt.Sprintf("CREATE DATABASE %v", name)
_, err := db.Exec(query)
return err
}
func CreateTable(db *sql.DB, query string) error {
_, err := db.Exec(query)
return err
}
func DropDB(db *sql.DB, name string) error {
query := fmt.Sprintf("DROP DATABASE %v", name)
_, err := db.Exec(query)
return err
}
could not import github.com/go-sql-driver/mysql (no required modules provides package "github.com/go-sql-driver/mysql")
screenshot of what's happening
It seems that you read the tutorial for an older go version.
Go 1.17 requires dependencies must be explicitly in go.mod.
Maybe you could try go module first (https://go.dev/blog/using-go-modules)
Your IDE is not showing you the whole picture. By running go run main.go (or whatever main file you have) on the command line, you can see the same error as you're seeing on your IDE with some extra:
$ go run main.go
main.go:7:5: no required module provides package github.com/go-sql-driver/mysql; to add it:
go get github.com/go-sql-driver/mysql
By issuing the suggested command go get github.com/go-sql-driver/mysql, you'll get the dependency added to your go.mod file and the contents of the package will be downloaded to your machine.
Your next execution will work:
$ go run main.go
Hello world
I've made some small modifications to your code to work, I'll add them here for completeness:
I've used the same source, but changed the package name to main.
I've added a main function to the bottom of the file:
func main() {
fmt.Println("Hello world")
_, err := Connect("username", "password", "localhost", "db")
if err != nil {
panic(err)
}
}
I've saved to a file named main.go
I've initialized the go.mod file by running go mod init test and go mod tidy, then I took the steps described on the beginning of the answer.

Strange number when using fmt.Println in Golang

I'm new to Golang and have been doing alright but I have a strange issue that I have not encountered before when using fmt. This strange behavior is when I'm printing a string. At the end of the string (which has sub-strings) it is also printing out what appears to be the len() of each string although the number don't add up. Can anyone explain why this is happening and how to stop it?
Any help is greatly appreciated
Here is the code:
package main
import (
"fmt"
//"log"
"strings"
)
var e = "[{8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888 localhost:3303 4d50f447-7c93-42df-a03e-89c09626950a}]"
func main() {
tl := strings.Trim(e, "[{")
tr := strings.Trim(tl, "}]")
r := strings.TrimSpace(tr)
s := strings.Fields(r)
V_PK := s[0]
SERVER_ADDR := s[1]
A_KEY := s[2]
vv, _ := fmt.Printf("[{\"v_pk\": %q", V_PK)
pp, _ := fmt.Printf(",\"server_addr\": %q", SERVER_ADDR)
kk, _ := fmt.Printf(",\"a_key\": %q}] ", A_KEY)
rstr, _ := fmt.Println(vv, pp, kk)
stringc := string(rstr)
fmt.Println(stringc)
}
Expected output:
[{"v_pk": "8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888","server_addr": "localhost:3303","a_key": "4d50f447-7c93-42df-a03e-89c09626950a"}]
Actual output:
[{"v_pk": "8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888","server_addr": "localhost:3303","a_key": "4d50f447-7c93-42df-a03e-89c09626950a"}] 82 36 53
Why on earth would it be printing these string lengths on the end? It's probably obvious that I'm trying to build a JSON string so these numbers on the end are problematic when trying to import the string into a JSON interpreter.
Again, any help is appreciated!
Take a look at the documentation for fmt.Printf and its friends fmt.Println. The documentation reads:
Printf formats according to a format specifier and writes to standard output. It returns the number of bytes written and any write error encountered.
The line in your code
vv, _ := fmt.Printf("[{\"v_pk\": %q", V_PK)
prints the formatted string to standard output, then return the number of bytes written and stores that in vv. If you want to print the formatted string to standard output, just call fmt.Printf and ignore the output:
package main
import (
"fmt"
//"log"
"strings"
)
var e = "[{8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888 localhost:3303 4d50f447-7c93-42df-a03e-89c09626950a}]"
func main() {
tl := strings.Trim(e, "[{")
tr := strings.Trim(tl, "}]")
r := strings.TrimSpace(tr)
s := strings.Fields(r)
V_PK := s[0]
SERVER_ADDR := s[1]
A_KEY := s[2]
fmt.Printf("[{\"v_pk\": %q, \"server_addr\": %q, \"a_key\": %q}]\n", V_PK, SERVER_ADDR, A_KEY)
}
Or, if you want to store the formatted string to a new string variable, call fmt.Sprintf:
stringc := fmt.Sprintf("[{\"v_pk\": %q, \"server_addr\": %q, \"a_key\": %q}]", V_PK, SERVER_ADDR, A_KEY)
fmt.Println(stringc)
You can check out a working version at the playground.
You might also want to checkout the json package, which can do the parsing and serializing for you with properly defined structs:
package main
import (
"encoding/json"
"fmt"
)
func main() {
type Datum struct {
VPK string `json:"v_pk"`
Server string `json:"server_addr"`
AKey string `json:"a_key"`
}
data := []Datum{
{VPK: "8888a8558921d75ec8bc362efbe9a76b82ec002337534e9f06ce92cbf8c27c8888",
Server: "localhost:3303",
AKey: "4d50f447-7c93-42df-a03e-89c09626950a",
}}
json, err := json.MarshalIndent(data, "", " ")
if err != nil {
// deal with error
}
fmt.Println(string(json))
}
Check it out at the go playground.
fmt.Printf returns the number of bytes written. The variables vv, pp, kk are the number of bytes written by those three Printf calls, and the three numbers printed are those numbers.

How to set the output of the aws-sdk-go to "text"?

Although the output setting has been set to text
~/.aws/config
[default]
output=text
the aws-sdk-go returns json. The question is whether the output could be switched to text.
When:
aws route53 get-hosted-zone --id some-id
is run, the output looks as follows:
NAMESERVERS some-ns
NAMESERVERS some-ns1
NAMESERVERS some-ns2
NAMESERVERS some-ns3
According to the this AWS documentation one could set the configuration:
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-2")},
)
One attempt was to consult this Config struct, but an Output option seems to be omitted.
How to set the output to text?
Note: an issue has added to the github page of the aws-sdk-go as well.
Example
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/route53"
)
func main() {
session, err := session.NewSession()
if err != nil {
log.Fatal(err)
}
r53 := route53.New(session)
listParams := &route53.ListResourceRecordSetsInput{
HostedZoneId: aws.String("some-id"),
}
records, err := r53.ListResourceRecordSets(listParams)
if err != nil {
log.Fatal(err)
}
fmt.Println(records)
}
returns:
{
IsTruncated: false,
MaxItems: "100",
ResourceRecordSets: [
{
Name: "some-domain.",
ResourceRecords: [{
Value: "some-ip"
}],
TTL: 7200,
Type: "A"
}
}
while aws route53 list-resource-record-sets --hosted-zone-id some-id, results in:
RESOURCERECORDSETS some-domain. 7200 A
RESOURCERECORDS some-ip
Problem
While it is possible to set the format of the aws-cli to output, it does not seem to be possible to do the same for the SDK.
Question
How to let the go-aws-sdk return text rather than json?
I have all of the information you need, you just have to unravel it from the response (records).
To get similar results from the last cli command:
for _, recordSet := range records.ResourceRecordSets {
log.Println("RESOURCERECORDSETS " + *recordSet.Name + strconv.Itoa(int(*recordSet.TTL)) + *recordSet.Type)
for _, record := range recordSet.ResourceRecords {
log.Println("RESOURCERECORDS " + *record.Value)
}
log.Println("")
}