I use json file to configure my program arguments and use flag package to configure the same arguments.
When some argument parsed by json file and flag at the same time, I wish use the arguments parsed through flag.
The trouble is that the json file path is also parsed through flag.
The json path can be obtained after calling flag.parse(), but the arguments is also parsed, then Unmarshal json will overwrite the arguments of flag parsing.
example json:
{
"opt1": 1,
"opt2": "hello"
}
example code:
var Config = struct {
Opt1 int `json:"opt1"`
Opt2 string `json:"opt2"`
}{
Opt1: 0,
Opt2: "none",
}
func main() {
// parse config file path
var configFile string
flag.StringVar(&configFile, "config", "", "config file path")
// parse options
flag.IntVar(&Config.Opt1, "opt1", Config.Opt1, "")
flag.StringVar(&Config.Opt2, "opt2", Config.Opt2, "")
// parse flags
flag.Parse()
// load config options from config.json file
if configFile != "" {
if data, err := ioutil.ReadFile(configFile); err != nil {
fmt.Printf("read config file error: %v\n", err)
} else if err = json.Unmarshal(data, &Config); err != nil {
fmt.Printf("parse config file error: %v\n", err)
}
}
fmt.Printf("%+v", Config)
}
program example output:
./foo.exe -opt2 world
out: {Opt1:0 Opt2:world}
./foo.exe -config config.json
out: {Opt1:1 Opt2:hello}
./foo.exe -config config.json -opt2 world
real out: {Opt1:1 Opt2:hello}
hope out: {Opt1:1 Opt2:world}
One easy solution would be to first parse the JSON config file, and once that's done, then proceed to parse CLI arguments.
The problem with this is that the JSON config file is also a CLI arg.
The simplest solution would be to call flag.Parse() again after parsing the config file:
// load config options from config.json file
if configFile != "" {
if data, err := ioutil.ReadFile(configFile); err != nil {
fmt.Printf("read config file error: %v\n", err)
} else if err = json.Unmarshal(data, &Config); err != nil {
fmt.Printf("parse config file error: %v\n", err)
}
// parse flags again to have precedence
flag.Parse()
}
The second flag.Parse() will override and thus have precedence over data coming from the JSON file.
With this addition, running
./foo.exe -config config.json -opt2 world
Output will be what you desire:
{Opt1:1 Opt2:world}
Related
I am trying to develop a web application that parses JSON file AWS S3, Finally, save my local database. I don't save JSON files on my local machine. I want to read the JSON files directly from S3. Now I'm struggling to read JSON files, I cant read that parse JSON file. My workflow is 1.First read JSON file from amazon S3 withdout saving JSON file reading local machine 2.save on my DB
func (s *Server) awsDataDownload(res http.ResponseWriter, req *http.Request) {
AWSRegion := "eu-west-2"
AccessKeyID := "XXXXXXXXXXXX"
SecretAccessKey := "XXXXXXXXXXXXXXXXXX"
AWSBucketName := "sample-prod"
_, err := envcfg.New()
if err != nil {
s.logger.WithError(err).Infof("Failed to initalize AWS configuration")
http.Error(res, "Failed to initalize AWS configuration", http.StatusInternalServerError)
return
}
userS3, err := s3.NewS3(s3.Config{
Region: AWSRegion,
AccessKeyID: AccessKeyID,
SecretAccessKey: SecretAccessKey,
Buckets: map[string]string{
"userS3": AWSBucketName,
},
})
if err != nil {
s.logger.WithError(err).Infof("Failed to connect with AWS")
http.Error(res, "Failed to connect with AWS", http.StatusInternalServerError)
return
}
files, err := userS3.GetAllFiles(req.Context(), "userS3", "")
if err != nil {
s.logger.WithError(err).Infof("cannot load files from AWS")
http.Error(res, "Failed to download with AWS", http.StatusInternalServerError)
return
}
for _, f := range files {
awsUploadFile := AWSUploadedFile{
FileName: f.FileName,
URL: f.URL,
Created: time.Time{},
Size: f.Size,
}
// json file match
awsURLmatchJson := regexp.MustCompile(`.+?(.json)`).FindStringSubmatch(awsUploadFile.URL)[0]
//companiesNameMatch := regexp.MustCompile(`.+?(.companies)`).FindStringSubmatch(awsUploadFile.URL)
// date match
awsURLmatchDateJson := regexp.MustCompile(`\d{4}-\d{2}-\d{2}`).FindStringSubmatch(awsUploadFile.URL)[0]
//s.logger.Infof("%+v", awsURLmatchJson)
//s.logger.Infof("%+v", awsURLmatchDateJson)
if awsURLmatchDateJson == "2021-09-30" {
s.fctDataDownload(res,req,awsURLmatchJson,awsUploadFile.URL, awsUploadFile.FileName)
}
}
//s.logger.Info(":::::::::::::::::::::::AWS::::::::::::::::::::::::::::")
//s.logger.Info(files)
//s.logger.Infof("%+v", files)
}
Now, I got all files name, URL from this. Now I want to read JSON from getting this JSON file. this is my code
func (s *Server) fctDataDownload(res http.ResponseWriter, req *http.Request, awsURLmatchJson, URL, FileName string) {
s.fctOrgProfile(res, req, awsURLmatchJson, URL, FileName)
// s.fctOrgPeople(res, req)
// s.fctOrgFundings(res, req)
}
func (s *Server) fctOrgProfile(res http.ResponseWriter, req *http.Request, awsURLmatchJson, URL, FileName string) {
s.logger.Infof("awsURLmatchJson %+v", awsURLmatchJson)
//orgProfile := "/home/asad/go/src/codemen.org/json-dump-data/companies/companies.json"
companiesNameMatch := regexp.MustCompile(`.+?(.people)`).FindStringSubmatch(awsURLmatchJson)
s.logger.Infof("%+v", companiesNameMatch)
if len(companiesNameMatch) < 1 {
return
}
s.logger.Errorf(" companiesNameMatch %+v", companiesNameMatch)
s.logger.Errorf(" FileName %+v", FileName)
//orgProfile := companiesNameMatch
jsonFile, err := os.Open(FileName)
if err != nil {
fmt.Printf("failed to open json file: %s, error: %v", FileName, err)
s.logger.Errorf("%+v", companiesNameMatch)
return
}
//Rest code
But I'm getting this error
ERRO[0005] FileName part-00000-c6c78231-f8ca-498c-b949-472b53074a57-c000.json
failed to open json file: part-00000-c6c78231-f8ca-498c-b949-472b53074a57-c000.json, error: open part-00000-c6c78231-f8ca-498c-b949-472b53074a57-c000.json: no such file or directoryERRO[0005] [https://fct-apix-data-dump-prod.s3.eu-west-2.amazonaws.com/date%3D2021-09-30/people /people]
JSON file is stored in S3, you can't read it with os.Open
jsonFile, err := os.Open(FileName)
replace it with s3.GetObject
This question already has answers here:
Parsing multiple JSON objects in Go
(4 answers)
Closed 2 years ago.
I have a json file (file.json) with following content:
file.json:
{"job": "developer"}
{"job": "taxi driver"}
{"job": "police"}
The contents of the file are exactly as above (not valid json file)
I want use data in my code but I can not Unmarshal This
What you have is not a single JSON object but a series of (unrelated) JSON objects. You can't use json.Unmarshal() to unmarshal something that contains multiple (independent) JSON values.
Use json.Decoder to decode multiple JSON values (objects) from a source one-by-one.
For example:
func main() {
f := strings.NewReader(file)
dec := json.NewDecoder(f)
for {
var job struct {
Job string `json:"job"`
}
if err := dec.Decode(&job); err != nil {
if err == io.EOF {
break
}
panic(err)
}
fmt.Printf("Decoded: %+v\n", job)
}
}
const file = `{"job": "developer"}
{"job": "taxi driver"}
{"job": "police"}`
Which outputs (try it on the Go Playground):
Decoded: {Job:developer}
Decoded: {Job:taxi driver}
Decoded: {Job:police}
This solution works even if your JSON objects take up multiple lines in the source file, or if there are multiple JSON objects in the same line.
See related: I was getting output of exec.Command output in the following manner. from that output I want to get data which I needed
You can read string line by line and unmarshal it:
package main
import (
"bufio"
"encoding/json"
"fmt"
"strings"
)
type j struct {
Job string `json:"job"`
}
func main() {
payload := strings.NewReader(`{"job": "developer"}
{"job": "taxi driver"}
{"job": "police"}`)
fscanner := bufio.NewScanner(payload)
for fscanner.Scan() {
var job j
err := json.Unmarshal(fscanner.Bytes(), &job)
if err != nil {
fmt.Printf("%s", err)
continue
}
fmt.Printf("JOB %+v\n", job)
}
}
Output:
JOB {Job:developer}
JOB {Job:taxi driver}
JOB {Job:police}
Example
I'm trying to parse JSON files by using dynamically created structs, but apparently I'm doing something wrong. Can somebody please tell we what am I doing wrong here:
structured := make(map[string][]reflect.StructField)
structured["Amqp1"] = []reflect.StructField{
reflect.StructField{
Name: "Test",
Type: reflect.TypeOf(""),
Tag: reflect.StructTag(`json:"test"`),
},
reflect.StructField{
Name: "Float",
Type: reflect.TypeOf(5.5),
Tag: reflect.StructTag(`json:"float"`),
},
reflect.StructField{
Name: "Connections",
Type: reflect.TypeOf([]Connection{}),
Tag: reflect.StructTag(`json:"connections"`),
},
}
sections := []reflect.StructField{}
for sect, params := range structured {
sections = append(sections,
reflect.StructField{
Name: sect,
Type: reflect.StructOf(params),
},
)
}
parsed := reflect.New(reflect.StructOf(sections)).Elem()
if err := json.Unmarshal([]byte(JSONConfigContent), &parsed); err != nil {
fmt.Printf("unable to parse data from provided configuration file: %s\n", err)
os.Exit(1)
}
https://play.golang.org/p/C2I4Pduduyg
Thanks in advance.
You want to use .Interface() to return the actual, underlying value, which should be a pointer to the concrete anonymous struct.
Note that the reflect.New function returns a reflect.Value representing a pointer to a new zero value for the specified type. The Interface method, in this case, returns that pointer as interface{} which is all you need for json.Unmarshal.
If, after unmarshaling, you need a non-pointer of the struct you can turn to reflect again and use reflect.ValueOf(parsed).Elem().Interface() to effectively dereference the pointer.
parsed := reflect.New(reflect.StructOf(sections)).Interface()
if err := json.Unmarshal([]byte(JSONConfigContent), parsed); err != nil {
fmt.Printf("unable to parse data from provided configuration file: %s\n", err)
os.Exit(1)
}
https://play.golang.org/p/Bzu1hUyKjvM
reflect.New returns a value representing a pointer.
Change line 69:
fmt.Printf(">>> %v", &parsed)
Result:
>>> <struct { Amqp1 struct { Test string "json:\"test\""; Float float64 "json:\"float\""; Connections []main.Connection "json:\"connections\"" } } Value>
I would recommend something very different. I generally avoid reflection where possible. You can accomplish what you are trying to do by simply providing structs for each type of config you expect and then do an initial "pre-unmarshalling" to determine what type of config you should actually use by name (which, in this case is a key of your JSON object):
package main
import (
"encoding/json"
"fmt"
"os"
)
//Amqp1 config struct
type Amqp1 struct {
Config struct {
Test string `json:"test"`
Float float64 `json:"float"`
Connections []Connection `json:"connections"`
} `json:"Amqp1"`
}
//Connection struct
type Connection struct {
Type string `json:"type"`
URL string `json:"url"`
}
//JSONConfigContent json
const JSONConfigContent = `{
"Amqp1": {
"test": "woobalooba",
"float": 5.5,
"connections": [
{"type": "test1", "url": "booyaka"},
{"type": "test2", "url": "foobar"}
]
}
}`
func main() {
configMap := make(map[string]interface{})
if err := json.Unmarshal([]byte(JSONConfigContent), &configMap); err != nil {
fmt.Printf("unable to parse data from provided configuration file: %s\n", err)
os.Exit(1)
}
//get config name
var configName string
for cfg := range configMap {
configName = cfg
break
}
//unmarshal appropriately
switch configName {
case "Amqp1":
var amqp1 Amqp1
if err := json.Unmarshal([]byte(JSONConfigContent), &amqp1); err != nil {
fmt.Printf("unable to parse data from provided configuration file: %s\n", err)
os.Exit(1)
}
fmt.Printf("%s >>\n", configName)
fmt.Printf("Test: %s\n", amqp1.Config.Test)
fmt.Printf("Float: %v\n", amqp1.Config.Float)
fmt.Printf("Connections: %#v\n", amqp1.Config.Connections)
default:
fmt.Printf("unknown config encountered: %s\n", configName)
os.Exit(1)
}
}
The initial "pre-unmarshalling" happens on the 2nd line of main(), to a plain map where the key is a string and the value is interface{} because you don't care. You're just doing that to get the actual type of config that it is, which is the key of the first nested object.
You can run this on playground as well.
I'm using the Decoder from package encoding/json to decode a JSON configuration file into a struct. The field names have a different case (lowercase first character in struct because of visibility concerns) in the file and struct so I'm using struct field tags as described in the documentation. The Problem is that Decoder seems to ignore these tags and the struct fields are empty. Any ideas what's wrong with my code?
config.json
{
"DataSourceName": "simple-blog.db"
}
Config struct
type Config struct {
dataSourceName string `json:"DataSourceName"`
}
Loading config
func loadConfig(fileName string) {
file, err := os.Open(fileName)
if err != nil {
log.Fatalf("Opening config file failed: %s", err)
}
defer file.Close()
decoder := json.NewDecoder(file)
config = &Config{} // Variable config is defined outside
err = decoder.Decode(config)
if err != nil {
log.Fatalf("Decoding config file failed: %s", err)
}
log.Print("Configuration successfully loaded")
}
Usage
loadConfig("config.json")
log.Printf("DataSourceName: %s", config.dataSourceName)
Output
2017/10/15 21:04:11 DB Name:
You need to export your dataSourceName field as encoding/json package requires them to be so
I'm trying to configure my Go program by creating a JSON file and parsing it into a struct:
var settings struct {
serverMode bool
sourceDir string
targetDir string
}
func main() {
// then config file settings
configFile, err := os.Open("config.json")
if err != nil {
printError("opening config file", err.Error())
}
jsonParser := json.NewDecoder(configFile)
if err = jsonParser.Decode(&settings); err != nil {
printError("parsing config file", err.Error())
}
fmt.Printf("%v %s %s", settings.serverMode, settings.sourceDir, settings.targetDir)
return
}
The config.json file:
{
"serverMode": true,
"sourceDir": ".",
"targetDir": "."
}
The Program compiles and runs without any errors, but the print statement outputs:
false
(false and two empty strings)
I've also tried with json.Unmarshal(..) but had the same result.
How do I parse the JSON in a way that fills the struct values?
You're not exporting your struct elements. They all begin with a lower case letter.
var settings struct {
ServerMode bool `json:"serverMode"`
SourceDir string `json:"sourceDir"`
TargetDir string `json:"targetDir"`
}
Make the first letter of your stuct elements upper case to export them. The JSON encoder/decoder wont use struct elements which are not exported.