Directly reading files from AWS S3 without downloading json files in Golang - json

Question

I am trying to develop a web application that parses JSON file AWS S3, Finally, save my local database. I don't save JSON files on my local machine. I want to read the JSON files directly from S3. Now I'm struggling to read JSON files, I cant read that parse JSON file. My workflow is 1.First read JSON file from amazon S3 withdout saving JSON file reading local machine 2.save on my DB
func (s *Server) awsDataDownload(res http.ResponseWriter, req *http.Request) {
AWSRegion := "eu-west-2"
AccessKeyID := "XXXXXXXXXXXX"
SecretAccessKey := "XXXXXXXXXXXXXXXXXX"
AWSBucketName := "sample-prod"
_, err := envcfg.New()
if err != nil {
s.logger.WithError(err).Infof("Failed to initalize AWS configuration")
http.Error(res, "Failed to initalize AWS configuration", http.StatusInternalServerError)
return
}
userS3, err := s3.NewS3(s3.Config{
Region: AWSRegion,
AccessKeyID: AccessKeyID,
SecretAccessKey: SecretAccessKey,
Buckets: map[string]string{
"userS3": AWSBucketName,
},
})
if err != nil {
s.logger.WithError(err).Infof("Failed to connect with AWS")
http.Error(res, "Failed to connect with AWS", http.StatusInternalServerError)
return
}
files, err := userS3.GetAllFiles(req.Context(), "userS3", "")
if err != nil {
s.logger.WithError(err).Infof("cannot load files from AWS")
http.Error(res, "Failed to download with AWS", http.StatusInternalServerError)
return
}
for _, f := range files {
awsUploadFile := AWSUploadedFile{
FileName: f.FileName,
URL: f.URL,
Created: time.Time{},
Size: f.Size,
}
// json file match
awsURLmatchJson := regexp.MustCompile(`.+?(.json)`).FindStringSubmatch(awsUploadFile.URL)[0]
//companiesNameMatch := regexp.MustCompile(`.+?(.companies)`).FindStringSubmatch(awsUploadFile.URL)
// date match
awsURLmatchDateJson := regexp.MustCompile(`\d{4}-\d{2}-\d{2}`).FindStringSubmatch(awsUploadFile.URL)[0]
//s.logger.Infof("%+v", awsURLmatchJson)
//s.logger.Infof("%+v", awsURLmatchDateJson)
if awsURLmatchDateJson == "2021-09-30" {
s.fctDataDownload(res,req,awsURLmatchJson,awsUploadFile.URL, awsUploadFile.FileName)
}
}
//s.logger.Info(":::::::::::::::::::::::AWS::::::::::::::::::::::::::::")
//s.logger.Info(files)
//s.logger.Infof("%+v", files)
}
Now, I got all files name, URL from this. Now I want to read JSON from getting this JSON file. this is my code
func (s *Server) fctDataDownload(res http.ResponseWriter, req *http.Request, awsURLmatchJson, URL, FileName string) {
s.fctOrgProfile(res, req, awsURLmatchJson, URL, FileName)
// s.fctOrgPeople(res, req)
// s.fctOrgFundings(res, req)
}
func (s *Server) fctOrgProfile(res http.ResponseWriter, req *http.Request, awsURLmatchJson, URL, FileName string) {
s.logger.Infof("awsURLmatchJson %+v", awsURLmatchJson)
//orgProfile := "/home/asad/go/src/codemen.org/json-dump-data/companies/companies.json"
companiesNameMatch := regexp.MustCompile(`.+?(.people)`).FindStringSubmatch(awsURLmatchJson)
s.logger.Infof("%+v", companiesNameMatch)
if len(companiesNameMatch) < 1 {
return
}
s.logger.Errorf(" companiesNameMatch %+v", companiesNameMatch)
s.logger.Errorf(" FileName %+v", FileName)
//orgProfile := companiesNameMatch
jsonFile, err := os.Open(FileName)
if err != nil {
fmt.Printf("failed to open json file: %s, error: %v", FileName, err)
s.logger.Errorf("%+v", companiesNameMatch)
return
}
//Rest code
But I'm getting this error
ERRO[0005] FileName part-00000-c6c78231-f8ca-498c-b949-472b53074a57-c000.json
failed to open json file: part-00000-c6c78231-f8ca-498c-b949-472b53074a57-c000.json, error: open part-00000-c6c78231-f8ca-498c-b949-472b53074a57-c000.json: no such file or directoryERRO[0005] [https://fct-apix-data-dump-prod.s3.eu-west-2.amazonaws.com/date%3D2021-09-30/people /people]

Answer 1

JSON file is stored in S3, you can't read it with os.Open
jsonFile, err := os.Open(FileName)
replace it with s3.GetObject