I am using the following sample program:
func getEnv(appName string, env string) {
svc := elasticbeanstalk.New(session.New(), &aws.Config{Region: aws.String("us-east-1")})
params := &elasticbeanstalk.DescribeConfigurationSettingsInput{
ApplicationName: aws.String(appName), // Required
EnvironmentName: aws.String(env),
}
resp, err := svc.DescribeConfigurationSettings(params)
if err != nil {
fmt.Println(err.Error())
return
}
v := resp.ConfigurationSettings
fmt.Printf("%s", v)
}
It's printing out the following response; this looks like a valid json except for the missing quote makes. ex: ApplicationName and not "ApplicationName".
How do I parse this? or get a valid json from AWS?
ConfigurationSettings: [{
ApplicationName: "myApp",
DateCreated: 2016-01-12 00:10:10 +0000 UTC,
DateUpdated: 2016-01-12 00:10:10 +0000 UTC,
DeploymentStatus: "deployed",
Description: "Environment created from the EB CLI using \"eb create\"",
EnvironmentName: "stag-myApp-app-s1",
OptionSettings: [
...
resp.ConfigurationSettings is not in JSON format any more, the aws-sdk-go package handled that for you. When you do,
v := resp.ConfigurationSettings
v contains an instance []*ConfigurationSettingsDescription that was parsed from the JSON response, and you don't have to parse it yourself. What you are seeing when you print it out is the Go struct representation. You can just go ahead and use it:
if len(v) > 0 {
log.Println(v[0].ApplicationName)
}
This should print out myApp
Related
This question already has answers here:
Runtime error when parsing JSON array and map elements with trailing commas
(2 answers)
Closed 1 year ago.
user golang , write json unmarshal, error happened, Because of comma before "]".
import (
"encoding/json"
"github.com/c2h5oh/datasize"
xdsboot "github.com/envoyproxy/go-control-plane/envoy/config/bootstrap/v2"
"github.com/golang/protobuf/jsonpb"
)
err = json.Unmarshal(content, cfg)
if err != nil {
log.StartLogger.Fatalf("[config] [default load] json unmarshal config failed, error: %v", err)
}
return cfg
return error:
2021-07-14 06:53:39,637 [FATAL] [config] [default load] json unmarshal config failed, error: invalid character ']' looking for beginning of value
I use unit test case to run, find file input
func TestMosnEnvoyMode(t *testing.T) {
content, _ := ioutil.ReadFile("./test.json") // this file is input file
cfg := &MOSNConfig{}
if err := json.Unmarshal([]byte(content), cfg); err != nil {
t.Fatal(err)
}
if cfg.Mode() != Mix {
t.Fatalf("config mode is %d", cfg.Mode())
}
}
file content:
{
"stats_matcher": {
"inclusion_list": {
"patterns": [
{
"prefix": "cluster.xds-grpc"
},
{
"suffix": "ssl_context_update_by_sds"
}, // the error cause by here, ","
]
}
}
},
}
if I delete comma here
{
"suffix": "ssl_context_update_by_sds"
}, // the error cause by here, ","
It is OK!
Now, why and which json lib should I use? Because the input file can not change.
It's better if you can fix the JSON as it's invalid JSON to have trailing commas.
That being said, some languages support trailing commas natively, notably JavaScript, so you may see it in your data.
If you cannot change your data, switch to a JSON parser that supports trailing commas like HuJSON (aka Human JSON) which supports trailing commas and comments in JSON. It's a soft fork of encoding/json and the last 3 commits are from noted Xoogler and Ex-Golang team member Brad Fitzpatrick.
repo: https://github.com/tailscale/hujson
docs: https://pkg.go.dev/github.com/tailscale/hujson
The Unmarshal syntax is the same as encoding/json, just use:
err := hujson.Unmarshal(data, v)
I've used it and it works as described.
Although the output setting has been set to text
~/.aws/config
[default]
output=text
the aws-sdk-go returns json. The question is whether the output could be switched to text.
When:
aws route53 get-hosted-zone --id some-id
is run, the output looks as follows:
NAMESERVERS some-ns
NAMESERVERS some-ns1
NAMESERVERS some-ns2
NAMESERVERS some-ns3
According to the this AWS documentation one could set the configuration:
sess, err := session.NewSession(&aws.Config{
Region: aws.String("us-east-2")},
)
One attempt was to consult this Config struct, but an Output option seems to be omitted.
How to set the output to text?
Note: an issue has added to the github page of the aws-sdk-go as well.
Example
package main
import (
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/route53"
)
func main() {
session, err := session.NewSession()
if err != nil {
log.Fatal(err)
}
r53 := route53.New(session)
listParams := &route53.ListResourceRecordSetsInput{
HostedZoneId: aws.String("some-id"),
}
records, err := r53.ListResourceRecordSets(listParams)
if err != nil {
log.Fatal(err)
}
fmt.Println(records)
}
returns:
{
IsTruncated: false,
MaxItems: "100",
ResourceRecordSets: [
{
Name: "some-domain.",
ResourceRecords: [{
Value: "some-ip"
}],
TTL: 7200,
Type: "A"
}
}
while aws route53 list-resource-record-sets --hosted-zone-id some-id, results in:
RESOURCERECORDSETS some-domain. 7200 A
RESOURCERECORDS some-ip
Problem
While it is possible to set the format of the aws-cli to output, it does not seem to be possible to do the same for the SDK.
Question
How to let the go-aws-sdk return text rather than json?
I have all of the information you need, you just have to unravel it from the response (records).
To get similar results from the last cli command:
for _, recordSet := range records.ResourceRecordSets {
log.Println("RESOURCERECORDSETS " + *recordSet.Name + strconv.Itoa(int(*recordSet.TTL)) + *recordSet.Type)
for _, record := range recordSet.ResourceRecords {
log.Println("RESOURCERECORDS " + *record.Value)
}
log.Println("")
}
I've been trying to get the go yaml package to parse a file with jsonlines entries.
Below is a simple example with three options of data to be parsed.
Option one is a multi-doc yaml example. Both docs parse ok.
Option two is a two jsonline example. The first line parses ok, but the second is missed.
Option three is a two jsonline example, but I've put yaml doc separators in between, to force the issue. Both of these parse ok.
From reading the yaml and json specs, I believe the second option, multiple jsonlines, ought to be handled by a yaml parser.
My questions are:
Should a YAML parser cope with jsonlines?
Am I using the go yaml package correctly?
package main
import (
"bytes"
"fmt"
"reflect"
"strings"
"gopkg.in/yaml.v2"
)
var testData = []string{
`
---
option_one_first_yaml_doc: ok_here
---
option_one_second_yaml_doc: ok_here
`,
`
{option_two_first_jsonl: ok_here}
{option_two_second_jsonl: missing}
`,
`
---
{option_three_first_jsonl: ok_here}
---
{option_three_second_jsonl: ok_here}
`}
func printVal(v interface{}, depth int) {
typ := reflect.TypeOf(v)
if typ == nil {
fmt.Printf(" %v\n", "<null>")
} else if typ.Kind() == reflect.Int || typ.Kind() == reflect.String {
fmt.Printf("%s%v\n", strings.Repeat(" ", depth), v)
} else if typ.Kind() == reflect.Slice {
fmt.Printf("\n")
printSlice(v.([]interface{}), depth+1)
} else if typ.Kind() == reflect.Map {
fmt.Printf("\n")
printMap(v.(map[interface{}]interface{}), depth+1)
}
}
func printMap(m map[interface{}]interface{}, depth int) {
for k, v := range m {
fmt.Printf("%sKey: %s Value(s):", strings.Repeat(" ", depth), k.(string))
printVal(v, depth+1)
}
}
func printSlice(slc []interface{}, depth int) {
for _, v := range slc {
printVal(v, depth+1)
}
}
func main() {
m := make(map[interface{}]interface{})
for _, data := range testData {
yamlData := bytes.NewReader([]byte(data))
decoder := yaml.NewDecoder(yamlData)
for decoder.Decode(&m) == nil {
printMap(m, 0)
m = make(map[interface{}]interface{})
}
}
}
jsonlines is newline delimited JSON. That means the individual lines are JSON, but not multiple lines and certainly not a whole file of multiple lines.
You will need to read the jsonlines input a line at a time, and those lines you should be able to process with go yaml, since YAML is a superset of JSON.
Since you also seem to have YAML end of indicator (---) lines in your test, you
need to process those as well.
When decoding a timestamp field from JSON into a struct on my local OS X machine, the Location of the time.Time field is "empty" rather than UTC. This is problematic for me running unit tests locally (vs. on a CI server where the Location is being set correctly to be UTC).
Here's the example code: https://play.golang.org/p/pb3eMbjSmv
package main
import (
"fmt"
"time"
)
func main() {
// Ignoring the err just for this example's sake!
parsed, _ := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
which outputs
String(): 2017-08-15 22:30:00 +0000 +0000
Location():
So while the offset of the time.Time's Location appears to be correct, its timezone name is just an empty string. Running in on other machines (and The Go Playground) give the expected "UTC" location.
When I run that on my machine, I see
TimeField.String(): 2017-08-15 22:30:00 +0000 +0000
TimeField.Location():
So while the offset of the time.Time's Location appears to be correct, its timezone name is just an empty string. This is using Go 1.5:
go version go1.5 darwin/amd64
I find same behavior using my current setup on Mac and I suspect it will be same behavior on Linux (not sure through)
$ go version
go version devel +31ad583 Wed Aug 10 19:44:08 2016 +0000 darwin/amd64
To make it more deterministic, I suggest using a custom json Unmarshal like so:
package main
import (
"encoding/json"
"fmt"
"strings"
"time"
)
type Time struct {
*time.Time
}
func (t *Time) UnmarshalJSON(b []byte) error {
const format = "\"2006-01-02T15:04:05+00:00\""
t_, err := time.Parse(format, string(b))
if err != nil {
return err
}
*t = Time{&t_}
return nil
}
type Example struct {
TimeField *Time `json:"time_field"`
}
func main() {
inString := "{\"time_field\": \"2017-08-15T22:30:00+00:00\"}"
var ex Example
decoder := json.NewDecoder(strings.NewReader(inString))
decoder.Decode(&ex)
fmt.Printf("TimeField.String(): %v\n", ex.TimeField.String())
fmt.Printf("TimeField.Location(): %v\n", ex.TimeField.Location())
}
Yes, You are right. On The Go Playground the Local is set to UTC inside that sandbox:
Try this working sample code on The Go Playground:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
output on The Go Playground:
go1.7 amd64p32 nacl
String(): 2017-08-15 22:30:00 +0000 UTC
Location(): UTC
And try it on your local system, output Location() is empty.
You may use utc := parsed.UTC() with the location set to UTC, like this working sample code The Go Playground:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.Parse(time.RFC3339, "2017-08-15T22:30:00+00:00")
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
utc := parsed.UTC()
fmt.Printf("String(): %v\n", utc.String())
fmt.Printf("Location(): %v\n", utc.Location())
}
Also You may use time.ParseInLocation(time.RFC3339, "2017-08-15T22:30:00+00:00", time.UTC), like this working sample code:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.Version(), runtime.GOARCH, runtime.GOOS) //go1.7 amd64p32 nacl
parsed, err := time.ParseInLocation(time.RFC3339, "2017-08-15T22:30:00+00:00", time.UTC)
if err != nil {
panic(err)
}
fmt.Printf("String(): %v\n", parsed.String())
fmt.Printf("Location(): %v\n", parsed.Location())
}
So the Location() will be UTC.
Thanks to #djd for pointing out that we can skip all the JSON/struct decoding business; the key issue is with time.Parse.
The same issue comes up here where the Location is "empty" rather than UTC (I would've expected UTC based on the docs: https://golang.org/pkg/time/#Parse
In the absence of a time zone indicator, Parse returns a time in UTC.
This answer was taken from the question as of revision 6.
I'm trying to send a JSON message with Go.
This is the server code:
func (network *Network) Join(
w http.ResponseWriter,
r *http.Request) {
//the request is not interesting
//the response will be a message with just the clientId value set
log.Println("client wants to join")
message := Message{-1, -1, -1, ClientId(len(network.Clients)), -1, -1}
var buffer bytes.Buffer
enc := json.NewEncoder(&buffer)
err := enc.Encode(message)
if err != nil {
fmt.Println("error encoding the response to a join request")
log.Fatal(err)
}
fmt.Printf("the json: %s\n", buffer.Bytes())
fmt.Fprint(w, buffer.Bytes())
}
Network is a custom struct. In the main function, I'm creating a network object and registering it's methods as callbacks to http.HandleFunc(...)
func main() {
runtime.GOMAXPROCS(2)
var network = new(Network)
var clients = make([]Client, 0, 10)
network.Clients = clients
log.Println("starting the server")
http.HandleFunc("/request", network.Request)
http.HandleFunc("/update", network.GetNews)
http.HandleFunc("/join", network.Join)
log.Fatal(http.ListenAndServe("localhost:5000", nil))
}
Message is a struct, too. It has six fields all of a type alias for int.
When a client sends an http GET request to the url "localhost:5000/join", this should happen
The method Join on the network object is called
A new Message object with an Id for the client is created
This Message is encoded as JSON
To check if the encoding is correct, the encoded message is printed on the cmd
The message is written to the ResponseWriter
The client is rather simple. It has the exact same code for the Message struct. In the main function it just sends a GET request to "localhost:5000/join" and tries to decode the response. Here's the code
func main() {
// try to join
var clientId ClientId
start := time.Now()
var message Message
resp, err := http.Get("http://localhost:5000/join")
if err != nil {
log.Fatal(err)
}
fmt.Println(resp.Status)
dec := json.NewDecoder(resp.Body)
err = dec.Decode(&message)
if err != nil {
fmt.Println("error decoding the response to the join request")
log.Fatal(err)
}
fmt.Println(message)
duration := time.Since(start)
fmt.Println("connected after: ", duration)
fmt.Println("with clientId", message.ClientId)
}
I've started the server, waited a few seconds and then ran the client. This is the result
The server prints "client wants to join"
The server prints "the json: {"What":-1,"Tag":-1,"Id":-1,"ClientId":0,"X":-1,"Y":-1}"
The client prints "200 OK"
The client crashes "error decoding the response to the join request"
The error is "invalid character "3" after array element"
This error message really confused me. After all, nowhere in my json, there's the number 3. So I imported io/ioutil on the client and just printed the response with this code
b, _ := ioutil.ReadAll(resp.Body)
fmt.Printf("the json: %s\n", b)
Please note that the print statement is the same as on the server. I expected to see my encoded JSON. Instead I got this
"200 OK"
"the json: [123 34 87 104 97 116 ....]" the list went on for a long time
I'm new to go and don't know if i did this correctly. But it seems as if the above code just printed the slice of bytes. Strange, on the server the output was converted to a string.
My guess is that somehow I'm reading the wrong data or that the message was corrupted on the way between server and client. But honestly these are just wild guesses.
In your server, instead of
fmt.Fprint(w, buffer.Bytes())
you need to use:
w.Write(buffer.Bytes())
The fmt package will format the Bytes() into a human-readable slice with the bytes represented as integers, like so:
[123 34 87 104 97 116 ... etc
You don't want to use fmt.Print to write stuff to the response. Eg
package main
import (
"fmt"
"os"
)
func main() {
bs := []byte("Hello, playground")
fmt.Fprint(os.Stdout, bs)
}
(playground link)
Produces
[72 101 108 108 111 44 32 112 108 97 121 103 114 111 117 110 100]
Use the Write() method of the ResponseWriter instead
You could have found this out by telneting to your server as an experiment - always a good idea when you aren't sure what is going on!