Trying to accomplish the following output with my data:
Convert to JSON string and write to file: output.json (this part is working)
Gzip Compress the JSON string and write that to a json.gz file: output.json.gz (NOT WORKING)
The code runs fine and writes to both files. But the gzipped file gives this error when I try to unzip it: Data error in 'output.json'. File is broken
Here's the code:
package main
import (
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"io/ioutil"
)
type Generic struct {
Name string
Cool bool
Rank int
}
func main() {
generic := Generic{"Golang", true, 100}
fileJson, _ := json.Marshal(generic)
err := ioutil.WriteFile("output.json", fileJson, 0644)
if err != nil {
fmt.Printf("WriteFileJson ERROR: %+v", err)
}
var fileGZ bytes.Buffer
zipper := gzip.NewWriter(&fileGZ)
defer zipper.Close()
_, err = zipper.Write([]byte(string(fileJson)))
if err != nil {
fmt.Printf("zipper.Write ERROR: %+v", err)
}
err = ioutil.WriteFile("output.json.gz", []byte(fileGZ.String()), 0644)
if err != nil {
fmt.Printf("WriteFileGZ ERROR: %+v", err)
}
}
What did I miss?
You need to call zipper.Close() immediately after finishing writing
http://play.golang.org/p/xNeMg3aXxO
_, err = zipper.Write(fileJson)
if err != nil {
log.Fatalf("zipper.Write ERROR: %+v", err)
}
err := zipper.Close() // call it explicitly and check error
Calling defer zipper.Close() would trigger the call at the end of the main function. Until you call .Close() the data is being written to an intermediate buffer and not flushed to the actual file.
Related
How can i read data from a json file and send it as a post request to a uri endpoint.
I am currently learning the Go language and working on my first learning project.
This is my sample data
// parent.json
{"name":"Jade Copnell","age":16,"gender":"Agender","occupation":"Account Representative II","numbers":"178-862-5967","children":{"name":"Kayne Belsham","age":38,"gender":"Genderqueer","occupation":"Clinical Specialist","interest":"Re-engineered discrete methodology","number":"145-355-4123"},"friends":{"name":"Stephi Aries","age":74,"gender":"Genderqueer","occupation":"Senior Sales Associate","numbers":"873-726-1453","interests":"Self-enabling systematic function","methow":"24/7"}}
This is what I have written, when i run the below script, I tend to get a data similar to the below as output and I also get empty data sent to the database.
"{\"name\":\"Jade Copnell\",\"age\":16,\"gender\":\"Agender\",\"occupation\":\"Account Representative II\",\"numbers\":\"178-862-5967\",\"children\":{\"name\":\"Kayne Belsham\",\"age\":38,\"gender\":\"Genderqueer\",\"occupation\":\"Clinical Specialist\",\"interest\":\"Re-engineered discrete methodology\",\"number\":\"145-355-4123\"},\"friends\":{\"name\":\"Stephi Aries\",\"age\":74,\"gender\":\"Genderqueer\",\"occupation\":\"Senior Sales Associate\",\"numbers\":\"873-726-1453\",\"interests\":\"Self-enabling systematic function\",\"methow\":\"24/7\"}}"
func main() {
// Open the file.
f, _ := os.Open("./go_data/parent.json")
// Create a new Scanner for the file.
scanner := bufio.NewScanner(f)
// Loop over all lines in the file and print them.
for scanner.Scan() {
responseBody := scanner.Text()
postBody, _ := json.Marshal(responseBody)
//fmt.Println(postBody)
time.Sleep(2 * time.Second)
webBody := bytes.NewBuffer(postBody)
// fmt.Println(webBody)
resp, err := http.Post("http://127.0.0.1:5000/v1/parent", "application/json", webBody)
if err != nil {
log.Fatalf("An Error Occured %v", err)
}
time.Sleep(2 * time.Second)
defer resp.Body.Close()
}
}
What if you do this instead. The third argument to http.Post is an io.Reader interface - that your file "f" implements.
package main
import (
"bufio"
"log"
"net/http"
"os"
"time"
)
func main() {
// Open the file.
f, _ := os.Open("./go_data/parent.json")
resp, err := http.Post("http://127.0.0.1:5000/v1/parent", "application/json", f)
if err != nil {
log.Fatalf("An Error Occured %v", err)
}
time.Sleep(2 * time.Second)
defer resp.Body.Close()
}
I'm new to Go and am trying to take advantage of the concurrency in Go to build a basic scraper to pull extract title, meta description, and meta keywords from URLs.
I am able to print out the results to terminal with the concurrency but can't figure out how to write output to CSV. I've tried many a variations that I could think of with limited knowledge of Go and many end up breaking the concurrency - so losing my mind a bit.
My code and URL input file is below - Thanks in advance for any tips!
// file name: metascraper.go
package main
import (
// import standard libraries
"encoding/csv"
"fmt"
"io"
"log"
"os"
"time"
// import third party libraries
"github.com/PuerkitoBio/goquery"
)
func csvParsing() {
file, err := os.Open("data/sample.csv")
checkError("Cannot open file ", err)
if err != nil {
// err is printable
// elements passed are separated by space automatically
fmt.Println("Error:", err)
return
}
// automatically call Close() at the end of current method
defer file.Close()
//
reader := csv.NewReader(file)
// options are available at:
// http://golang.org/src/pkg/encoding/csv/reader.go?s=3213:3671#L94
reader.Comma = ';'
lineCount := 0
fileWrite, err := os.Create("data/result.csv")
checkError("Cannot create file", err)
defer fileWrite.Close()
writer := csv.NewWriter(fileWrite)
defer writer.Flush()
for {
// read just one record
record, err := reader.Read()
// end-of-file is fitted into err
if err == io.EOF {
break
} else if err != nil {
fmt.Println("Error:", err)
return
}
go func(url string) {
// fmt.Println(msg)
doc, err := goquery.NewDocument(url)
if err != nil {
checkError("No URL", err)
}
metaDescription := make(chan string, 1)
pageTitle := make(chan string, 1)
go func() {
// time.Sleep(time.Second * 2)
// use CSS selector found with the browser inspector
// for each, use index and item
pageTitle <- doc.Find("title").Contents().Text()
doc.Find("meta").Each(func(index int, item *goquery.Selection) {
if item.AttrOr("name", "") == "description" {
metaDescription <- item.AttrOr("content", "")
}
})
}()
select {
case res := <-metaDescription:
resTitle := <-pageTitle
fmt.Println(res)
fmt.Println(resTitle)
// Have been trying to output to CSV here but it's not working
// writer.Write([]string{url, resTitle, res})
// err := writer.WriteString(`res`)
// checkError("Cannot write to file", err)
case <-time.After(time.Second * 2):
fmt.Println("timeout 2")
}
}(record[0])
fmt.Println()
lineCount++
}
}
func main() {
csvParsing()
//Code is to make sure there is a pause before program finishes so we can see output
var input string
fmt.Scanln(&input)
}
func checkError(message string, err error) {
if err != nil {
log.Fatal(message, err)
}
}
The data/sample.csv input file with URLs:
http://jonathanmh.com
http://keshavmalani.com
http://google.com
http://bing.com
http://facebook.com
In the code you supplied, you had commented the following code:
// Have been trying to output to CSV here but it's not working
err = writer.Write([]string{url, resTitle, res})
checkError("Cannot write to file", err)
This code is correct, except you have one issue.
Earlier in the function, you have the following code:
fileWrite, err := os.Create("data/result.csv")
checkError("Cannot create file", err)
defer fileWrite.Close()
This code causes the fileWriter to close once your csvParsing() func exits.
Because you've closed fileWriter with the defer, you are unable to write to it in your concurrent function.
Solution:
You'll need to use defer fileWrite.Close() inside your concurrent func or something similar so you do not close the fileWriter before you have written to it.
Something tells me I'm not understanding json correctly. I'm trying to grab some data off http://api.bitcoincharts.com/v1/trades.csv?symbol=rockUSD, but my Unmarshal seems to not be able to read the json data. I'm a fresh beginner to golang (and json as well), and I'm wondering how I am able to skip that wrong character error I'm making.
My error:
invalid character ',' after top-level value
panic: invalid character ',' after top-level value
My code:
package main
import ("fmt"
"net/http"
"io/ioutil"
"encoding/json"
)
type Prices struct {
Data string
}
func main() {
url := "http://api.bitcoincharts.com/v1/trades.csv?symbol=rockUSD"
httpresp, err := http.Get(url)
if err != nil{
fmt.Println(err)
panic(err)
}
defer httpresp.Body.Close()
htmldata, err := ioutil.ReadAll(httpresp.Body)
if err != nil{
fmt.Println(err)
panic (err)
}
var jsonData []Prices
err = json.Unmarshal([]byte(htmldata), &jsonData)
if err != nil {
fmt.Println(err)
panic (err)
}
fmt.Println(jsonData)
}
That is NOT json data at all, you'd have to write a custom parser.
Example:
.........
data := readData(httpresp.Body)
........
func readData(r io.Reader) (out [][3]float64) {
br := bufio.NewScanner(r)
for br.Scan() {
parts := strings.Split(br.Text(), ",")
if len(parts) != 3 {
continue
}
var fparts [3]float64
for i, p := range parts {
// bad idea to ignore errors, but it's left as exercise for the reader.
fparts[i], _ = strconv.ParseFloat(p, 64)
}
out = append(out, fparts)
}
return
}
playground
I'm a total noob at go and I'm trying to understand what I'm missing here. I'm expecting to use dec.Decode to loop over the json values and end up with a map of the response. What I'm getting is the entire json string as the key to the first element of the map. What am I missing?
Example response:
2015/03/02 10:03:16 map[error:invalid_request error_description:that is not a recognized WePay API call error_code:1001] map[string]interface {}
package main
import (
"encoding/json"
"io"
"log"
"net/http"
"reflect"
)
func main() {
var v map[string]interface{}
resp, err := http.Get("https://wepayapi.com/v2/")
if err != nil {
log.Println("Error: " + err.Error())
}
defer resp.Body.Close()
// resp.Body is an io.ReadCloser... NewDecoder expects an io.Reader
dec := json.NewDecoder(resp.Body)
// Decode reads the next JSON-encoded value from its input and stores it in the value pointed to by v.
for err := dec.Decode(&v); err != nil && err != io.EOF; {
log.Println("ERROR: " + err.Error())
return
}
log.Println(v, reflect.TypeOf(v))
}
Decoder will decode the whole JSON value at once (in this case the error object), you don;y have to call it in a loop:
if err := dec.Decode(&v); err != nil {
log.Println("ERROR: " + err.Error())
return
}
As a response you get a map equavalent of this JSON:
{"error":"invalid_request","error_description":"that is not a recognized WePay API call","error_code":1001}
Result:
map[string]interface{} {
"error":"invalid_request",
"error_description":"that is not a recognized WePay API call",
"error_code":1001,
}
I'm starting to learn Golang and I would like to know how to get a json response by calling an url, if you could give me an example it would be great in order to guide myself.
Here's a simple example to get you started. Instead of a map[string]interface{} you should consider making a struct to hold the result of your request.
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
)
func main() {
resp, err := http.Get("http://api.geonames.org/citiesJSON?north=44.1&south=-9.9&east=-22.4&west=55.2&lang=de&username=demo")
if err != nil {
log.Fatal(err)
}
var generic map[string]interface{}
err = json.NewDecoder(resp.Body).Decode(&generic)
if err != nil {
log.Fatal(err)
}
fmt.Println(generic)
}
I'd write a little helper function to do it:
// getJSON fetches the contents of the given URL
// and decodes it as JSON into the given result,
// which should be a pointer to the expected data.
func getJSON(url string, result interface{}) error {
resp, err := http.Get(url)
if err != nil {
return fmt.Errorf("cannot fetch URL %q: %v", url, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected http GET status: %s", resp.Status)
}
// We could check the resulting content type
// here if desired.
err := json.NewDecoder(resp.Body).Decode(result)
if err != nil {
return fmt.Errorf("cannot decode JSON: %v", err)
}
return nil
}
A full working example can be found here: http://play.golang.org/p/b1WJb7MbQV
Note that it is important to check the status code as well as the Get error, and the response body must be closed explicitly (see the documentation here: http://golang.org/pkg/net/http/#Get)