Golang json Unmarshal "unexpected end of JSON input" - json

I am working on some code to parse the JSON data from an HTTP response. The code I have looks something like this:
type ResultStruct struct {
result []map[string]string
}
var jsonData ResultStruct
err = json.Unmarshal(respBytes, &jsonData)
The json in the respBytes variable looks like this:
{
"result": [
{
"id": "ID 1"
},
{
"id": "ID 2"
}
]
}
However, err is not nil. When I print it out it says unexpected end of JSON input. What is causing this? The JSON seems to valid. Does this error have something to do with my custom struct?
Thanks in advance!

The unexpected end of JSON input is the result of a syntax error in the JSON input (likely a missing ", }, or ]). The error does not depend on the type of the value that you are decoding to.
I ran the code with the example JSON input on the playground. It runs without error.
The code does not decode anything because the result field is not exported. If you export the result field:
type ResultStruct struct {
Result []map[string]string
}
then the input is decoded as shown in this playground example.
I suspect that you are not reading the entire response body in your application. I suggest decoding the JSON input using:
err := json.NewDecoder(resp.Body).Decode(&jsonData)
The decoder reads directly from the response body.

You can also get this error if you're using json.RawMessage in an unexported field. For example, the following code produces the same error:
package main
import (
"encoding/json"
"fmt"
)
type MyJson struct {
Foo bool `json:"foo"`
bar json.RawMessage `json:"bar"`
}
type Bar struct {
X int `json:"x"`
}
var respBytes = []byte(`
{
"foo": true,
"bar": { "x": 10 }
}`)
func main() {
var myJson MyJson
err := json.Unmarshal(respBytes, &myJson)
if err != nil {
fmt.Println(err)
return
}
myBar := new(Bar)
err = json.Unmarshal(myJson.bar, myBar)
fmt.Println(err)
}
If you export "MyJson.bar" field (e.g. -> "MyJson.Bar", then the code works.

it is not the case here; but if you are getting this error loading json from a file it Will occur if the byte slice for the buffer is not initialized the the byte size of the file. [when you're new like me that happens! ] Since this is the first search result I got it still took some digging to figure out. In this use case the error is a bit misleading.
type GenesisResultStruct []GenesisField
fileinfo, _ := genesis.Stat()
bs := make([]byte, fileinfo.Size())
//bs := []byte {} // wrong!!
_, error := genesis.Read(bs)
if error != nil {
fmt.Println("genesis read error: ", error)
os.Exit(1)
}
var jsonData GenesisResultStruct
eGen = json.Unmarshal(bs, &jsonData)
if eGen != nil {
fmt.Println("genesis unmarshal error: ", eGen)
os.Exit(1)
}

Faced the same issue today.
You can also get this error if the respBytes is nil or there are no brackets [] if you are unmarshalling it to a slice.
In that case, you need to explicitly set respBytes.
As you are unmarshalling it to a slice, brackets [] are expected in the byte-slice
if src == nil {
src = []byte("[]")
}

Related

Golang - Can't get array of objects after fetching JSON

After I make a request to a server - I get JSON like this:
{
"actions": [
{
"class": "...",
"parameters": [
{ ... }
{ ... }
]
}
]
...
}
The type of the variable I've put this data in, is map[string]interface{}.
I want to access the parameters array in the first object of the actions array. I can successfully get the class property - data.(map[string]interface{})["class"].
However, if I try the same for the parameters property - I get nil ...
I tried data.(map[string][]interface{})["parameters"] - but I get error panic: interface conversion: interface {} is map[string]interface {}, not map[string][]interface {}.
Any idea what I'm missing here?
EDIT:
The Golang code for this is this:
func main() {
var jsonResult map[string]interface{}
errorFromJsonFetching := utils.GetJSON(theUrl, &jsonResult)
if errorFromJsonFetching != nil {
fmt.Printf("Error from checking deploy build: %#v\n", errorFromJsonFetching)
}
// get the top level "actions" prop ..
actionZero := jsonResult["actions"].([]interface{})[0]
fmt.Printf("Class prop: /%v %T\n", actionZero.(map[string]interface{})["class"], actionZero)
fmt.Printf("Parameters prop: /%v %T\n", actionZero.(map[string]interface{})["parameters"], actionZero)
}
The GetJSON function is from another file in the project:
func GetJSON (url string, result interface{}) error {
fmt.Printf("Getting JSON from %v\n", url)
resp, err := http.Get(url)
if err != nil {
return fmt.Errorf("Cannot fetch URL %q: %v\n", url, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("Bad resposne status code: %s\n", resp.Status)
}
// attempt to put the JSON in the `result` ..
err = json.NewDecoder(resp.Body).Decode(&result)
if err != nil {
return fmt.Errorf("Could not decode the JSON from response, err: %v\n", err)
}
return nil;
}
EDIT2:
Thanks to the ideas of #larsks in the comments - I created a minimal code sample with JSON in a string, directly in main.go file - and all worked fine.
Then I went ahead to the browser again and tried to fetch the data again - from directly hitting the URL or with $.getJSON from a page - and both returned one and the same JSON data.
However, in my Go code, when I dump the JSON data - I see this for the first member of actions:
map[_class:hudson.model.ParametersDefinitionProperty parameterDefinitions:[map[...
So when I try to get the parameters array by the key parameterDefinitions - then I get the array of object ... :O
Sooooo ... I don't know what's happening ... either Go itself modifies the JSON data when it gets it from the backend, or the backend itself returns different things, depending on how the data is being fetched.
(The backend is the Jenkins API by the way ... so I don't know why I get parameterDefinitions instead of parameters in Go ... :( ...)
Thanks for updating your question; having the correct data makes it easier to answer.
It's best if the code you include in your question is something we can just grab and run -- that means it compiles and runs, and when given the sample data in your question, it produces the behavior you're asking about.
I've modified your code so that I can run it locally, primarily by replacing the GetJSON method with something that reads from a file named data.json instead of hitting an API:
package main
import (
"encoding/json"
"fmt"
"os"
)
func GetJSON(result interface{}) error {
datafile, err := os.Open("data.json")
if err != nil {
panic(err)
}
return json.NewDecoder(datafile).Decode(&result)
}
func main() {
var jsonResult map[string]interface{}
errorFromJsonFetching := GetJSON(&jsonResult)
if errorFromJsonFetching != nil {
fmt.Printf("Error from checking deploy build: %#v\n", errorFromJsonFetching)
}
// get the top level "actions" prop ..
actionZero := jsonResult["actions"].([]interface{})[0]
fmt.Printf("Class prop: /%v %T\n", actionZero.(map[string]interface{})["class"], actionZero)
fmt.Printf("Parameters prop: /%v %T\n", actionZero.(map[string]interface{})["parameters"], actionZero)
}
If I feed it this sample data, which I believe matches the structure of what you show in your question:
{
"actions": [
{
"class": "Name of class",
"parameters": [
{
"name": "alice",
"color": "blue"
},
{
"name": "bob",
"count": 10,
},
{
"name": "mallory",
"valid": false,
}
]
}
]
}
It produces the following output:
Class prop: /Name of class map[string]interface {}
Parameters prop: /[map[color:blue name:alice] map[count:10 name:bob] map[name:mallory valid:false]] map[string]interface {}
This doesn't produce a nil value for the parameters key. In order to more fully answer your question, can you update it so that it has sample data and code that reproduces the behavior you're asking baout?

Handling different types when unmarshalling a json [duplicate]

This question already has an answer here:
unmarshal nested json without knowing structure
(1 answer)
Closed 3 years ago.
I'm consuming an endpoint (which I don't own and I cannot fix) and this endpoint returns JSON.
The problem is this JSON can come in different formats:
Format 1:
{
"message": "Message"
}
or
{
"message": ["ERROR_CODE"]
}
Depending on what happened.
I'd like to have one struct to hold this response so later I can check if the message is a string or array, and properly follow a flow.
Is it possible to do it in Go? The first approach I thought was to have two structs and try to decode to the one with string, and if an error happens, try to decode to the one with array.
Is there a more elegant approach?
Unmarshal it into a value of interface{} type, and use a type assertion or type switch to inspect the type of value it ends up. Note that by default JSON arrays are unmarshaled into a value of type []interface{}, so you have to check for that to detect the error response.
For example:
type Response struct {
Message interface{} `json:"message"`
}
func main() {
inputs := []string{
`{"message":"Message"}`,
`{"message":["ERROR_CODE"]}`,
}
for _, input := range inputs {
var r Response
if err := json.Unmarshal([]byte(input), &r); err != nil {
panic(err)
}
switch x := r.Message.(type) {
case string:
fmt.Println("Success, message:", x)
case []interface{}:
fmt.Println("Error, code:", x)
default:
fmt.Println("Something else:", x)
}
}
}
Output (try it on the Go Playground):
Success, message: Message
Error, code: [ERROR_CODE]
You can use a custom type that implements json.Unmarshaller and tries to decode each possible input format in turn:
type Message struct {
Text string
Codes []int
}
func (m *Message) UnmarshalJSON(input []byte) error {
var text string
err := json.Unmarshal(input, &text)
if err == nil {
m.Text = text
m.Codes = nil
return nil
}
var codes []int
err := json.Unmarshal(input, &codes)
if err == nil {
m.Text = nil
m.Codes = codes
return nil
}
return err
}
I prefer this approach over unmarshalling into interface{} and type-asserting later because all the type checks are encapsulated in the unmarshalling step.
For a real-world example, check out my type veryFlexibleUint64: https://github.com/sapcc/limes/blob/fb212143c5f5b3e9272994872fcc7b758ae47646/pkg/plugins/client_ironic.go
I would suggest creating a different type for Message and making that implement json.Unmarshaller
here is how the code would be
type message struct {
Text string
Codes []string //or int , assuming array of string as it was not mentioned in the question
}
func (m *message) UnmarshalJSON(input []byte) error {
if len(input) == 0 {
return nil
}
switch input[0] {
case '"':
m.Text = strings.Trim(string(input), `"`)
return nil
case '[':
return json.Unmarshal(input, &m.Codes)
default:
return fmt.Errorf(`invalid character %q looking for " or [`, input[0])
}
}
type Error struct {
Message message `json:"message"`
}
You may find full code with test here

Does json.Unmarshal require your result structure to match exactly the JSON passed in?

I have a JSON string I want to unmarshal:
{
"id":1720,
"alertId":1,
"alertName":"{stats} Test Lambda Alert",
"dashboardId":5,
"panelId":2,
"userId":0,
"newState":"alerting",
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0,
"tags":[],
"login":"",
"email":"",
"avatarUrl":"",
"data":{
"evalMatches":[
{
"metric":"{prod}{stats} Lambda Alert Test",
"tags":null,
"value":16.525333333333332
}
]
}
}
I get the raw stream via a request: bodyBytes, err := ioutil.ReadAll(resp.Body)
I was hoping I could just specify a struct that pulls the values I care about, e.g.,
type Result struct {
ID string `json:"id"`
Time int64 `json:"time"`
}
However, when I try this, I get errors.
type Result struct {
ID string `json:"id"`
Time string `json:"time"`
}
var result Result
err2 := json.Unmarshal(bodyBytes, &result)
if err2 != nil {
log.Fatal(fmt.Sprintf(`Error Unmarshalling: %s`, err2))
}
fmt.Println(result.ID)
Error Unmarshalling: json: cannot unmarshal array into Go value of type main.Result
I suspect this error may be due to what's actually returned from ioutil.ReadAll(), since it has the above JSON string wrapped in [ ] if I do a fmt.Println(string(bodyBytes)), but if I try to json.Unmarshal(bodyBytes[0], &result), I just get compile errors, so I'm not sure.
If I want to unmarshal a JSON string, do I have to specify the full structure in my type Result struct? Is there a way around this? I don't want to be bound to the JSON object I receive (if the API changes upstream, it requires us to modify our code to recognize that, etc.).
You can unmarshal into structs that represent only some fields of your JSON document, but the field types have to match, as the error clearly states:
cannot unmarshal number into Go struct field Result.id of type string
You cannot unmarshal a number into a string. If you define the ID field as any numeric type it'll work just fine:
package main
import (
"encoding/json"
"fmt"
)
var j = []byte(`
{
"id":1720,
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0
}
`)
type Result struct {
ID int `json:"id"` // or any other integer type, or float{32,64}, or json.Number
Time int64 `json:"time"`
}
func main() {
var r Result
err := json.Unmarshal(j, &r)
fmt.Println(r, err)
}
Try it on the playground: https://play.golang.org/p/lqsQwLW2dHZ
Update
You have just edited your question with the actual error you receive. You have to unmarshal JSON arrays into slices. So if the HTTP response in fact returns a JSON array, unmarshal into []Result:
var j = []byte(`
[
{
"id":1720,
"prevState":"ok",
"time":1523983581000,
"text":"",
"regionId":0
}
]
`)
var r []Result
err := json.Unmarshal(j, &r)
fmt.Println(r[0], err)
https://play.golang.org/p/EbOVA8CbcFO
To generate Go types that match your JSON document pretty well, use https://mholt.github.io/json-to-go/.

Marshalling to NonString type in Golang JSON POST Request

I'm having a little bit of trouble handling types in Golang. I'm making a POST router.
Consider the following struct:
type DataBlob struct {
Timestamp string
Metric_Id int `json:"id,string,omitempty"`
Value float32 `json:"id,string,omitempty"`
Stderr float32 `json:"id,string,omitempty"`
}
This is my POST router using json.Unmarshal() from a decoded stream:
func Post(w http.ResponseWriter, req * http.Request) {
body, err := ioutil.ReadAll(req.Body)
if err != nil {
panic()
}
var t DataBlob
err = json.Unmarshal(body, &t)
if err != nil {
panic()
}
fmt.Printf("%s\n", t.Timestamp)
fmt.Printf("%d\n", int(t.Metric_Id))
fmt.Printf("%f\n", t.Value)
fmt.Printf("%f\n", t.Stderr)
}
It seems that no matter what I make my values in my POST request:
{
"timestamp": "2011-05-16 15:36:38",
"metric_id": "28",
"value": "4.5",
"stderr": "8.5"
}
All the non-String values print as 0 or 0.000000, respectively. It also doesn't matter if I try to type convert inline after-the-fact, as I did with t.Metric_Id in the example.
If I edit my struct to just handle string types, the values print correctly.
I also wrote a version of the POST router using json.NewDecoder():
func Post(w http.ResponseWriter, req * http.Request) {
decoder := json.NewDecoder(req.Body)
var t DataBlob
err := decoder.Decode(&t)
if err != nil {
panic()
}
fmt.Printf("%s\n", t.Timestamp)
fmt.Printf("%d\n", t.Metric_Id)
fmt.Printf("%f\n", t.Value)
fmt.Printf("%f\n", t.Stderr)
}
This is building off of functionality described in this answer, although the solution doesn't appear to work.
I appreciate your help!
You need to change the names of your Datablob values. You've told the JSON decoder that they're all named "id". You should try something like the following. Also, take a look at the json.Marshal description and how it describes the tags for structs and how the json library handles them. https://golang.org/pkg/encoding/json/#Marshal
type DataBlob struct {
Timestamp string
Metric_Id int `json:"metric_id,string,omitempty"`
Value float32 `json:"value,string,omitempty"`
Stderr float32 `json:"stderr,string,omitempty"`
}

(un)marshalling json golang not working

I'm playing with Go and am stumped as to why json encode and decode don't work for me
I think i copied the examples almost verbatim, but the output says both marshal and unmarshal return no data. They also don't give an error.
can anyone hint to where i'm going wrong?
my sample code: Go playground
package main
import "fmt"
import "encoding/json"
type testStruct struct {
clip string `json:"clip"`
}
func main() {
//unmarshal test
var testJson = "{\"clip\":\"test\"}"
var t testStruct
var jsonData = []byte(testJson)
err := json.Unmarshal(jsonData, &t)
if err != nil {
fmt.Printf("There was an error decoding the json. err = %s", err)
return
}
fmt.Printf("contents of decoded json is: %#v\r\n", t)
//marshal test
t.clip = "test2"
data, err := json.Marshal(&t)
if err != nil {
fmt.Printf("There was an error encoding the json. err = %s", err)
return
}
fmt.Printf("encoded json = %s\r\n", string(data))
}
output:
contents of decoded json is: main.testStruct{clip:""}
encoded json = {}
in both outputs I would have expected to see the decoded or encoded json
For example,
package main
import "fmt"
import "encoding/json"
type testStruct struct {
Clip string `json:"clip"`
}
func main() {
//unmarshal test
var testJson = "{\"clip\":\"test\"}"
var t testStruct
var jsonData = []byte(testJson)
err := json.Unmarshal(jsonData, &t)
if err != nil {
fmt.Printf("There was an error decoding the json. err = %s", err)
return
}
fmt.Printf("contents of decoded json is: %#v\r\n", t)
//marshal test
t.Clip = "test2"
data, err := json.Marshal(&t)
if err != nil {
fmt.Printf("There was an error encoding the json. err = %s", err)
return
}
fmt.Printf("encoded json = %s\r\n", string(data))
}
Output:
contents of decoded json is: main.testStruct{Clip:"test"}
encoded json = {"clip":"test2"}
Playground:
http://play.golang.org/p/3XaVougMTE
Export the struct fields.
type testStruct struct {
Clip string `json:"clip"`
}
Exported identifiers
An identifier may be exported to permit access to it from another
package. An identifier is exported if both:
the first character of the identifier's name is a Unicode upper case letter (Unicode class "Lu"); and
the identifier is declared in the package block or it is a field name or method name.
All other identifiers are not exported.
Capitalize names of structure fields
type testStruct struct {
clip string `json:"clip"` // Wrong. Lowercase - other packages can't access it
}
Change to:
type testStruct struct {
Clip string `json:"clip"`
}
In my case, my struct fields were capitalized but I was still getting the same error.
Then I noticed that the casing of my fields was different. I had to use underscores in my request.
For eg:
My request body was:
{
"method": "register",
"userInfo": {
"fullname": "Karan",
"email": "email#email.com",
"password": "random"
}
}
But my golang struct was:
type AuthRequest struct {
Method string `json:"method,omitempty"`
UserInfo UserInfo `json:"user_info,omitempty"`
}
I solved this by modifying my request body to:
{
"method": "register",
"user_info": {
"fullname": "Karan",
"email": "email#email.com",
"password": "random"
}
}