How can you define the timezone of a timestamp when loading into BigQuery from a CSV file?
none of these seem to work:
2018-07-31 11:55:00 Europe/Rome
2018-07-31 11:55:00 CET
I get the following error:
{Location: "query"; Message: "Unrecognized timezone: Europe/Rome;
Could not parse '2018-07-31 11:55:00 Europe/Rome' as datetime for
field ts (position 0) starting at location 0"; Reason: "invalidQuery"}
I am running an import from Google Cloud Storage, using this Go code:
gcsRef := bigquery.NewGCSReference(gcsFilename)
gcsRef.SourceFormat = bigquery.CSV
gcsRef.FieldDelimiter = "|"
gcsRef.Schema = bigquery.Schema{
{Name: "ts", Type: bigquery.TimestampFieldType},
{Name: "field2", Type: bigquery.StringFieldType},
{Name: "field3", Type: bigquery.StringFieldType},
}
loader := bigqueryClient.Dataset("events").Table("mytable").LoaderFrom(gcsRef)
loader.WriteDisposition = bigquery.WriteAppend
job, err := loader.Run(ctx)
if err != nil {
log.Fatalln("loader.Run", err.Error())
}
status, err := job.Wait(ctx)
if err != nil {
log.Fatalln("job.Wait", err.Error())
}
if status.Err() != nil {
log.Fatalln("Job completed with error: %v", status.Err(), status.Errors)
}
to make it work - try to declare ts field as string and then you will be able to resolve it into timestamp in whatever query you will then use - using already mentioned (in comment) approach - like SELECT TIMESTAMP(ts)
Related
I need to add entry to a yaml file from struct value config.Sif["snk_prod"] (I need to fill it on runtime) and try the following , I tried the following but I got an error when filling the struct, any idea?
package main
import "gopkg.in/yaml.v3"
const document = `
spec:
mec:
customConfig:
sif:
prom_exporter:
type: prometheus_exporter
snk_dev:
type: sk_hc_logs
inputs:
- tesslt
ep: ${NT}
dken: ${SN}
`
type YamlObject map[string]any
type CustomConfig struct {
Sif map[string]interface{} `yaml:"sif,omitempty"`
}
type ecfg struct {
Type string `yaml:"type,omitempty"`
Inputs []string `yaml:"inputs,omitempty"`
Ep string `yaml:"ep,omitempty"`
Dken string `yaml:"dken,omitempty"`
}
func main() {
t := CustomConfig{}
config := &t
// -------I need to add it as struct values as I got the values on runtime dynamically
config.Sif["snk_prod"] = ecfg{
Type: "sk_hc_ls",
Inputs: []string{"tesslt"},
Ep: "${NT}",
}
yamlBytes, err := yaml.Marshal(t)
doc := make(YamlObject)
if err := yaml.Unmarshal([]byte(document), &doc); err != nil {
panic(err)
}
addon := make(YamlObject)
if err := yaml.Unmarshal(yamlBytes, &addon); err != nil {
panic(err)
}
node := findChild(doc, "spec", "mec", "customConfig", "sif")
if node == nil {
panic("Must not happen")
}
for key, val := range addon {
(*node)[key] = val
}
outDoc, err := yaml.Marshal(doc)
if err != nil {
panic(err)
}
println(string(outDoc))
}
func findChild(obj YamlObject, path ...string) *YamlObject {
if len(path) == 0 {
return &obj
}
key := path[0]
child, ok := obj[key]
if !ok {
return nil
}
obj, ok = child.(YamlObject)
if !ok {
return nil
}
return findChild(obj, path[1:]...)
}
https://go.dev/play/p/6CHsOJPXqpw
After searching I found this answer https://stackoverflow.com/a/74089724/6340176
which is quite similar to mine, the change is the that I need to add it as struct value
at the end I need to add new entry under sif
At the end The output should be like following
spec:
mec:
customConfig:
sif:
prom_exporter:
type: prometheus_exporter
snk_dev:
type: sk_hc_logs
inputs:
- tesslt
ep: ${NT}
dken: ${SN}
snk_prod:
type: sk_hc_ls
inputs:
- tesslt
ep: ${NT}
Replace the creation of the yamlBytes as follows:
t := make(map[string]interface{})
t["snk_prod"] = ecfg{
Type: "sk_hc_ls",
Inputs: []string{"tesslt"},
Ep: "${NT}",
}
yamlBytes, err := yaml.Marshal(t)
Then you will get the expected result.
BTW: The panic is triggered because you are trying to insert a value into an uninitialized map (config.Sif is nil). You could simply create an empty map using, for example, the following line of code before assigning values:
config.Sif = make(map[string]interface{})
but there would be an additional unwanted sif node in the code, e.g. something like this:
...
sif:
prom_exporter:
type: prometheus_exporter
sif:
snk_prod:
...
Therefore, the yaml snippet to be added dynamically should be generated as shown at the beginning.
I'm currently trying to create a CloudScheduler job from golang.
However, it is stuck with an error and I would like to know how to fix it.
The current code looks like this, and is to be run from CloudRun.
const projectID = "hogehoge-project"
const locationID = "asia-northeast1"
const jobBaseUrl = "https://example.com/notify"
func StartToCheckRunnning(jobID string) error {
ctx := context.Background()
cloudschedulerService, err := cloudscheduler.NewCloudSchedulerClient(ctx)
if err != nil {
log.Fatalf("cloudscheduler.NewCloudSchedulerClient: %v", err)
return fmt.Errorf("cloudscheduler.NewCloudSchedulerClient: %v", err)
}
defer cloudschedulerService.Close()
queuePath := fmt.Sprintf("projects/%s/locations/%s", projectID, locationID)
req := &schedulerpb.CreateJobRequest{
Parent: queuePath,
Job: &schedulerpb.Job{
Name: jobID,
Description: "managed by the system",
Target: &schedulerpb.Job_HttpTarget{
HttpTarget: &schedulerpb.HttpTarget{
Uri: createJobUrl(jobBaseUrl, jobID),
HttpMethod: schedulerpb.HttpMethod_POST,
},
},
Schedule: "* * * * *",
TimeZone: "jst",
},
}
resp, err := cloudschedulerService.CreateJob(ctx, req)
if err != nil {
log.Fatalf("cloudschedulerService.CreateJob: %v", err)
return fmt.Errorf("cloudschedulerService.CreateJob: %v", err)
}
// TODO: Use resp.
_ = resp
return nil
}
When I run it, I will get the following error.
cloudschedulerService.CreateJob: rpc error: code = InvalidArgument desc = Job name must be formatted: "projects/<PROJECT_ID>/locations/<LOCATION_ID>/jobs/<JOB_ID>".
However, when I change the queuePath to the following, I now get the following error.
queuePath := fmt.Sprintf("projects/%s/locations/%s/jobs/%s", projectID, locationID, jobID)
cloudschedulerService.CreateJob: rpc error: code = PermissionDenied desc = The principal (user or service account) lacks IAM permission "cloudscheduler.jobs.create" for the resource "projects/hogehoge-project/locations/asia-northeast1/jobs/029321cb-467f-491e-852e-0c3df3d49db3" (or the resource may not exist).
Since I am using the CloudRun default service account, there should be no lack of permissions.
By the way, here is what it says to write in this format: projects/PROJECT_ID/locations/LOCATION_ID.
https://pkg.go.dev/google.golang.org/genproto/googleapis/cloud/scheduler/v1beta1#CreateJobRequest
How can I correctly execute the create request?
Thanks.
I solved myself with this code.
It was an error in the Name of the Job, not the Parent.
queuePath := fmt.Sprintf("projects/%s/locations/%s", projectID, locationID)
namePath := fmt.Sprintf("projects/%s/locations/%s/jobs/%s", projectID, locationID, jobID)
req := &schedulerpb.CreateJobRequest{
Parent: queuePath,
Job: &schedulerpb.Job{
Name: namePath,
Description: "managed by system",
Target: &schedulerpb.Job_HttpTarget{
HttpTarget: &schedulerpb.HttpTarget{
Uri: createJobUrl(jobBaseUrl, jobID),
HttpMethod: schedulerpb.HttpMethod_POST,
},
},
Schedule: "* * * * *",
TimeZone: "Asia/Tokyo",
},
}
I am using the golang library go-jmespath to query a json file. My code looks like the following:
package main
import (
"encoding/json"
"log"
"github.com/jmespath/go-jmespath"
)
func main() {
var jsonBlob = []byte(`[
{
"oeeId": 3162396,
"oeeDate": "2019-03-06T00:00:00",
"oee": 21.2
}]`)
var d interface{}
err := json.Unmarshal(jsonBlob, &d)
if err != nil {
log.Printf("1: %s", err)
}
res, err := jmespath.Search("{ labels: ['example'], datasets: [{label: 'Verfügbarkeit', data: [?oeeDate == `2019-03-06T00:00:00`].oee}]}", d)
if err != nil {
log.Printf("2: %s", err)
}
log.Println(res)
}
Here is also a link to an example in the Playground.
When I execute this code I get the following error:
invalid character '-' after top-level value
I am wondering what's my issue with this code since this example works with other Jmespath implementations like the javascript jmespath.js library.
The error seems to be in the query rather than the input data. Can anybody help me on this one?
Replacing the backticks in the search string with single quote removes the error.
Use this instead: "{ labels: ['example'], datasets: [{label: 'Verfügbarkeit', data: [?oeeDate == '2019-03-06T00:00:00'].oee}]}".
No more error.
EDIT: I Made a mistake and corrected it
I'm trying to map a CSV file to a struct that defines fields of time.Datetime (later I want to change the datetime format to another format).
This is the output of my program
❯ go run issue.go ✗ master
record: 05/02/2019 15:02:31
record: 05/02/2019 16:02:40
record:
record on line 0; parse error on line 4, column 2: parsing time "" as "02/01/2006 15:04:05": cannot parse "" as "02"
1936 - 2019-02-05 15:02:31 +0000 UTC
1937 - 2019-02-05 16:02:40 +0000 UTC
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x10e58cc]
goroutine 1 [running]:
main.main()
/Users/gtheys/Code/go/src/github.axa.com/KTAXA/claims-headers-poc/issue.go:49 +0x1bc
exit status 2
I have tried to execute time.Parse using an empty string, it Didn't work. So the problem should be here. But I have no idea how to handle it graceful in following code:
package main
import (
"encoding/csv"
"fmt"
"io"
"os"
"time"
"github.com/gocarina/gocsv"
)
type DateTime struct {
time.Time
}
type Issue struct {
NotifyNo string
NotifyDate DateTime
}
// Convert the CSV string as internal date
func (date *DateTime) UnmarshalCSV(csv string) (err error) {
fmt.Println("record: ", csv)
date.Time, err = time.Parse("02/01/2006 15:04:05", csv)
return err
}
func main() {
//const layout = "02/05/2006 15:04:00"
issuesFile, err := os.OpenFile("ISSUES.txt", os.O_RDWR|os.O_CREATE, os.ModePerm)
if err != nil {
panic(err)
}
defer issuesFile.Close()
issues := []*Issue{}
gocsv.SetCSVReader(func(in io.Reader) gocsv.CSVReader {
r := csv.NewReader(in)
r.Comma = '|'
return r // Allows use pipe as delimiter
})
if err := gocsv.UnmarshalFile(issuesFile, &issues); err != nil {
fmt.Println(err)
}
for _, issue := range issues {
fmt.Println(issue.NotifyNo, " - ", issue.NotifyDate)
}
if _, err := issuesFile.Seek(0, 0); err != nil { // Go to the start of the file
panic(err)
}
}
the csv input file:
NotifyNo|NotifyDate|
1936|05/02/2019 15:02:31|
1937|05/02/2019 16:02:40|
1938||
1951|05/02/2019 17:02:19|
I'm fetching and decoding a large JSON response that has an error in it. Now I need to find where the error is! I read about json.SyntaxError but I am struggling to find out how to use it.
package main
import (
"encoding/json"
"fmt"
"net/http"
"os"
"text/template"
"time"
)
type Movie struct {
Title string `json:"title"`
PublishedAt time.Time `json:"published_at"`
}
func main() {
req, _ := http.NewRequest("GET", "https://s.natalian.org/2016-12-07/debugme2.json", nil)
resp, err := http.DefaultClient.Do(req)
defer resp.Body.Close()
dec := json.NewDecoder(resp.Body)
_, err = dec.Token()
for dec.More() {
var m Movie
if err = dec.Decode(&m); err != nil {
fmt.Println(err)
fmt.Println("Bad", m)
// https://blog.golang.org/error-handling-and-go
if serr, ok := err.(*json.SyntaxError); ok {
fmt.Println("Syntax error", serr)
}
} else {
fmt.Println("Good", m)
}
tmpl := template.Must(template.New("test").Parse("OUTPUT: {{ if .Title }}{{.Title}}{{ if .PublishedAt }} was published at {{.PublishedAt}} {{ end }}{{end}}\n"))
tmpl.Execute(os.Stdout, m)
}
}
What am I missing? Any tools or strategies or suggestions would be much appreciated. My output currently looks like:
Good {foobar 2016-11-24 16:17:12 +0800 SGT}
OUTPUT: foobar was published at 2016-11-24 16:17:12 +0800 SGT
parsing time ""null"" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null"" as "2006"
Bad {barbar 0001-01-01 00:00:00 +0000 UTC}
OUTPUT: barbar was published at 0001-01-01 00:00:00 +0000 UTC
Good { 1999-12-24 16:11:12 +0200 +0200}
OUTPUT:
Good {Something else entirely 2000-01-24 16:11:12 +0200 +0200}
OUTPUT: Something else entirely was published at 2000-01-24 16:11:12 +0200 +0200
But I need something like this in my stderr to better debug the issue:
Line 8: published_at is invalid
And maybe some context of the Title so I can tell the API backend team they have an error in their JSON response.
BONUS question: Furthermore I don't want to print the value 0001-01-01 00:00:00 +0000 UTC as it's actually really empty. I don't actually mind it being missing.
I found some solution:
if err := json.Unmarshal([]byte(data), &myStruct); err != nil {
if jsonErr, ok := err.(*json.SyntaxError); ok {
problemPart := data[jsonErr.Offset-10 : jsonErr.Offset+10]
err = fmt.Errorf("%w ~ error near '%s' (offset %d)", err, problemPart, jsonErr.Offset)
}
}
It will print something like
invalid character 'n' after object key:value pair ~ error near 'rence\","numberOfBil' (offset 14557)
One way to both accept null values, and to not print anything if published_at is null, is to set PublishedAt field to a pointer value :
type Movie struct {
Title string `json:"title"`
PublishedAt *time.Time `json:"published_at"`
}
The input string is valid JSON, so the json package does not raise a SyntaxError.
The json package has some other error types, such as UnmarshalTypeError, which is raised when an error occurs when the json does not match a nuilt-in type (e.g : string, int, array ...).
Unfortunately, when it calls a custom UnmarshalJSON() function, it looks like the json package returns the raw error :
package main
import (
"bytes"
"encoding/json"
"fmt"
"time"
)
// check the full type of an error raised when Unmarshaling a json string
func main() {
var test struct {
Clock time.Time
}
buf := bytes.NewBufferString(`{"Clock":null}`)
dec := json.NewDecoder(buf)
// ask to decode an invalid null value into a flat time.Time field :
err := dec.Decode(&test)
// print the details of the returned error :
fmt.Printf("%#v\n", err)
}
// Output :
&time.ParseError{Layout:"\"2006-01-02T15:04:05Z07:00\"", Value:"null", LayoutElem:"\"", ValueElem:"null", Message:""}
https://play.golang.org/p/fhZxVpOflb
The final error comes straight from the time package, it is not some kind of UnmarshalError from the json package which could at least tell you "this error occured when trying to Unmarshal value at this offset", and the error alone will not give you the context.
You can look specifically for type *time.ParseError in the error :
if terr, ok := err.(*time.ParseError); ok {
// in the example : Movie has one single time.Time field ;
// if a time.ParseError occured, it was while trying to read that field
fmt.Println("Error when trying to read 'published_at' value", terr)
// you can leave the field to its zero value,
// or if you switched to a pointer field :
m.PublishedAt = nil
}
If you happen to have several time fields (e.g : ProducedAt and PublishedAt), you can still look which field was left with its zero value :
if terr, ok := err.(*time.ParseError); ok {
if m.ProducedAt.IsZero() {
fmt.Println("Error when trying to read 'produced_at' value", terr)
}
if m.PublishedAt == zero {
fmt.Println("Error when trying to read 'published_at' value", terr)
}
}
By the way : as specified in the docs, "0001-01-01 00:00:00 UTC" is the zero value that the go team chose for go's time.Time zero value.
Your data for published_at is "null", it is string type, so I think you can define the PublishedAt as string, and you can use code to parse it to time.Time.
This is my test code:
package main
import (
"encoding/json"
"github.com/swanwish/go-common/logs"
"github.com/swanwish/go-common/utils"
)
func main() {
url := `https://s.natalian.org/2016-12-07/debugme2.json`
_, content, err := utils.GetUrlContent(url)
if err != nil {
logs.Errorf("Failed to get content from url %s, the error is %v", url, err)
return
}
movies := []struct {
Title string `json:"title"`
PublishedAt string `json:"published_at"`
}{}
err = json.Unmarshal(content, &movies)
if err != nil {
logs.Errorf("Failed to unmarshal content %s, the error is %v", string(content), err)
return
}
logs.Debugf("The movies are %v", movies)
}
The result is:
The movies are [{foobar 2016-11-24T16:17:12.000+08:00} {barbar null} { 1999-12-24T16:11:12.000+02:00} {Something else entirely 2000-01-24T16:11:12.000+02:00}]
It looks like madness, but it should work:
rawBody := []byte(`{"title":"test", "published_at":"2017-08-05T15:04:05Z", "edited_at":"05.08.2017"}`)
type Movie struct {
Title string `json:"title"`
PublishedAt time.Time `json:"published_at"`
EditedAt time.Time `json:"edited_at"`
}
var msg Movie
if err = json.Unmarshal(rawBody, &msg); err != nil {
if _, ok := err.(*time.ParseError); ok {
value := reflect.ValueOf(msg).Elem()
if value.Kind().String() != "struct" {
return err
}
for i := 0; i < value.NumField(); i++ {
field := value.Field(i)
if t, ok := field.Interface().(time.Time); ok {
if t.IsZero() {
name := value.Type().Field(i).Name
return fmt.Errorf("field: %s, message: %s", strings.ToLower(name), "time is not in RFC 3339 format.")
}
}
}
}
return err
}
This code will return first error happened. If PublishedAt is invalid we will know nothing about EditedAt even if it is valid.