I'm reading a number of JSON files from S3, and want to return them all as one large JSON array. I have a struct matching my JSON data, and a for loop iterating over all objects in my s3 bucket. Each time I read, I unmarshal to my struct array. I want to append to my struct array so that I can get all the JSON data rather than just one file's data. Is there anyway to do this in Golang?
Yes, you should create a temporary array to Unmarshal the contents of each JSON, then append the items to your final result array in order to return the whole collection as one item.
See here an example of doing that.
In your case input would come from each of the S3 files you mention. Also, you would probably put that unmarshal logic in its own function to be able to call it for each input JSON.
package main
import (
"encoding/json"
"fmt"
"log"
)
type Record struct {
Author string `json:"author"`
Title string `json:"title"`
}
func main() {
var allRecords []Record
input := []byte(`[{
"author": "Nirvana",
"title": "Smells like teen spirit"
}, {
"author": "The Beatles",
"title": "Help"
}]`)
var tmpRecords []Record
err := json.Unmarshal(input, &tmpRecords)
if (err != nil) {
log.Fatal(err)
}
allRecords = append(allRecords, tmpRecords...)
fmt.Println("RECORDS:", allRecords)
}
https://play.golang.org/p/ZZGhy4UNhP
Related
This question already has an answer here:
Go json array of arrays
(1 answer)
Closed 9 days ago.
I am receiving this json from an external server:
[["010117", "070117", "080117"], ["080117", "140117", "150117"], ["150117", "210117", "220117"]]
and i need to parse it
package main
import (
"encoding/json"
"fmt"
"io"
"os"
"runtime"
)
type Range struct {
From string
To string
Do string
}
type AllRanges struct {
Ranges []Range
}
func main() {
var ranges AllRanges
j, err := os.ReadFile(file)
if err != nil {
panic("Can't read json file")
}
if json.Unmarshal(j, &v) != nil {
panic("Error reading the json")
}
}
When I execute, a panic it is thrown indicating an error reading the json
Thanks in advance !
This isn't the code that is failing. The code you have posted won't compile as it attempts to unmarshal into an undeclared variable, v.
Assuming that v is supposed to be ranges, the problem is very simple....
ranges is of type AllRanges which is a struct having a named member Ranges which is an array of structs, also having named members.
Therefore, when attempting to unmarshal json into this struct, the unmarshaller will expect to find:
{
"Ranges": [
{
"From": "..",
"To": ..,
"Do": ".."
},
{ etc }
]
}
To unmarshal your data, consisting of an anonymous array of arrays of string, you need instead to declare ranges as an array of array of strings:
var ranges [][]string
...
if json.Unmarshal(j, &ranges) != nil {
panic("Error reading the json")
}
Once you have unmarshalled into this array of arrays you will then need to write code to transform it into the desired structured values.
This playground demonstrates successfully unmarshalling your sample data into a [][]string. Transformation is left as an exercise.
I'm trying to convert a JSON that contains only 1 field which apparently an array to a complex struct in Golang but unfortunately I'm not getting the data back, instead, I got:
{Result:[]}
Anyone knows why? (code below)
package main
import (
"encoding/json"
"fmt"
)
type Account struct {
AccountId string
}
type Response struct {
Result []Account
}
func main() {
input := []byte(`{
"result": [
{"account_id" : "1"},
{"account_id" : "2"},
{"account_id" : "3"},
]
}
`)
var resp Response
json.Unmarshal(input, &resp)
fmt.Printf("%+v\n", resp)
}
use a explicit tag in your stucture type.
type Account struct {
AccountId string `json:"account_id, omitempty"`
}
If you are a novice, keep in mind the JSON size, if is large then use a stream library (jstream or easyjson etc),
other advice is check nullables or omit when they are empty anyway you can use nullable library like https://github.com/guregu/null
Cheers!
All of the ways I'm seeing involve building structs and unmarshalling the data into the struct. But what if I'm getting JSON responses with hundreds of fields? I don't want to have to create 100 field structs just to be able to get to the data I want. Coming from a Java background there are easy ways to simply get the http response as a string and then pass the JSON string into a JSON object that allows for easy traversal. It's very painless. Is there anything like this in Go?
Java example in pseudo code:
String json = httpResponse.getBody();
JsonObject object = new JsonObject(json);
object.get("desiredKey");
Golang: fetch JSON from an HTTP response without using structs as helpers
This is a typical scenario we come across. This is achieved by json.Unmarshal.
Here is a simple json
{"textfield":"I'm a text.","num":1234,"list":[1,2,3]}
which is serialized to send across the network and unmarshaled at Golang end.
package main
import (
"fmt"
"encoding/json"
)
func main() {
// replace this by fetching actual response body
responseBody := `{"textfield":"I'm a text.","num":1234,"list":[1,2,3]}`
var data map[string]interface{}
err := json.Unmarshal([]byte(responseBody), &data)
if err != nil {
panic(err)
}
fmt.Println(data["list"])
fmt.Println(data["textfield"])
}
Hope this was helpful.
The json.Unmarshal method will unmarshal to a struct that does not contain all the fields in the original JSON object. In other words, you can cherry-pick your fields. Here is an example where FirstName and LastName are cherry-picked and MiddleName is ignored from the json string:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
}
func main() {
jsonString := []byte("{\"first_name\": \"John\", \"last_name\": \"Doe\", \"middle_name\": \"Anderson\"}")
var person Person
if err := json.Unmarshal(jsonString, &person); err != nil {
panic(err)
}
fmt.Println(person)
}
The other answers here are misleading, as they don't show you what happens if you try to go deeper in the Map. This example works fine enough:
package main
import (
"encoding/json"
"fmt"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
body := map[string]interface{}{}
json.NewDecoder(r.Body).Decode(&body)
/*
[map[
id:com.github.android
platform:play
url:https://play.google.com/store/apps/details?id=com.github.android
]]
*/
fmt.Println(body["related_applications"])
}
but if you try to go one level deeper, it fails:
/*
invalid operation: body["related_applications"][0] (type interface {} does not
support indexing)
*/
fmt.Println(body["related_applications"][0])
Instead, you would need to assert type at each level of depth:
/*
map[
id:com.github.android
platform:play
url:https://play.google.com/store/apps/details?id=com.github.android
]
*/
fmt.Println(body["related_applications"].([]interface{})[0])
You can as well unmarshal it into a map[string]interface{}
body, err := ioutil.ReadAll(resp.Body)
map := &map[string]interface{}{}
json.Unmarshal(body, map)
desiredValue := map["desiredKey"]
The received json must have an object as the most outer element. The map can also contain lists or nested maps, depending on the json.
I am trying to parse a file which contains JSON data:
[
{"a" : "1"},
{"b" : "2"},
{"c" : "3"}
]
Since this is a JSON array with dynamic keys, I thought I could use:
type data map[string]string
However, I cannot parse the file using a map:
c, _ := ioutil.ReadFile("c")
dec := json.NewDecoder(bytes.NewReader(c))
var d data
dec.Decode(&d)
json: cannot unmarshal array into Go value of type main.data
What would be the most simple way to parse a file containing a JSON data is an array (only string to string types) into a Go struct?
EDIT: To further elaborate on the accepted answer -- it's true that my JSON is an array of maps. To make my code work, the file should contain:
{
"a":"1",
"b":"2",
"c":"3"
}
Then it can be read into a map[string]string
Try this: http://play.golang.org/p/8nkpAbRzAD
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
)
type mytype []map[string]string
func main() {
var data mytype
file, err := ioutil.ReadFile("test.json")
if err != nil {
log.Fatal(err)
}
err = json.Unmarshal(file, &data)
if err != nil {
log.Fatal(err)
}
fmt.Println(data)
}
It's because your json is actually an array of maps, but you're trying to unmarshall into just a map. Try using the following:
type YourJson struct {
YourSample []struct {
data map[string]string
}
}
you can try the bitly's simplejson package
https://github.com/bitly/go-simplejson
it's much easier.
I was marshaling and unmarshaling JSONs using golang and when I want to do it with number fields golang transforms it in floating point numbers instead of use long numbers, for example.
I have the following JSON:
{
"id": 12423434,
"Name": "Fernando"
}
After marshal it to a map and unmarshal again to a json string I get:
{
"id":1.2423434e+07,
"Name":"Fernando"
}
As you can see the "id" field is in floating point notation.
The code that I am using is the following:
package main
import (
"encoding/json"
"fmt"
"os"
)
func main() {
//Create the Json string
var b = []byte(`
{
"id": 12423434,
"Name": "Fernando"
}
`)
//Marshal the json to a map
var f interface{}
json.Unmarshal(b, &f)
m := f.(map[string]interface{})
//print the map
fmt.Println(m)
//unmarshal the map to json
result,_:= json.Marshal(m)
//print the json
os.Stdout.Write(result)
}
It prints:
map[id:1.2423434e+07 Name:Fernando]
{"Name":"Fernando","id":1.2423434e+07}
It appears to be that the first marshal to the map generates the FP. How can I fix it to a long?
This is the link to the program in the goland playground:
http://play.golang.org/p/RRJ6uU4Uw-
There are times when you cannot define a struct in advance but still require numbers to pass through the marshal-unmarshal process unchanged.
In that case you can use the UseNumber method on json.Decoder, which causes all numbers to unmarshal as json.Number (which is just the original string representation of the number). This can also useful for storing very big integers in JSON.
For example:
package main
import (
"strings"
"encoding/json"
"fmt"
"log"
)
var data = `{
"id": 12423434,
"Name": "Fernando"
}`
func main() {
d := json.NewDecoder(strings.NewReader(data))
d.UseNumber()
var x interface{}
if err := d.Decode(&x); err != nil {
log.Fatal(err)
}
fmt.Printf("decoded to %#v\n", x)
result, err := json.Marshal(x)
if err != nil {
log.Fatal(err)
}
fmt.Printf("encoded to %s\n", result)
}
Result:
decoded to map[string]interface {}{"id":"12423434", "Name":"Fernando"}
encoded to {"Name":"Fernando","id":12423434}
The JSON standard doesn't have longs or floats, it only has numbers. The json package will assume float64 when you haven't defined anything else (meaning, only provided Unmarshal with an interface{}).
What you should do is to create a proper struct (as Volker mentioned):
package main
import (
"encoding/json"
"fmt"
"os"
)
type Person struct {
Id int64 `json:"id"`
Name string `json:"name"`
}
func main() {
//Create the Json string
var b = []byte(`{"id": 12423434, "Name": "Fernando"}`)
//Marshal the json to a proper struct
var f Person
json.Unmarshal(b, &f)
//print the person
fmt.Println(f)
//unmarshal the struct to json
result, _ := json.Marshal(f)
//print the json
os.Stdout.Write(result)
}
Result:
{12423434 Fernando}
{"id":12423434,"name":"Fernando"}
Playground: http://play.golang.org/p/2R76DYVgMK
Edit:
In case you have a dynamic json structure and wish to use the benefits of a struct, you can solve it using json.RawMessage. A variable of type json.RawMessage will store the raw JSON string so that you later on, when you know what kind of object it contains, can unmarshal it into the proper struct. No matter what solution you use, you will in any case need some if or switch statement where you determine what type of structure it is.
It is also useful when parts of the JSON data will only be copied to the another JSON object such as with the id-value of a JSON RPC request.
Example of container struct using json.RawMessage and the corresponding JSON data:
type Container struct {
Type string `json:"type"`
Data json.RawMessage `json:"data"`
}
var b = []byte(`{"type": "person", "data":{"id": 12423434, "Name": "Fernando"}}`)
A modified version of your example on Playground: http://play.golang.org/p/85s130Sthu
Edit2:
If the structure of your JSON value is based on the name of a name/value pair, you can do the same with a:
type Container map[string]json.RawMessage