Go: Reading JSON file vs. Reading JSON string - json

I have a JSON file called example.json that looks like:
{
"name": "example",
"type": "record"
}
I also have a variable representing the above as a "string":
const example = `{
"name": "example",
"type": "record",
}`
I am trying to understand why reading the contents of the JSON file into bytes is different from reading the contents of the example variable. My code is as follows:
bytesJSON, err := ioutil.ReadFile("example.json")
if err != nil {
fmt.Println(err)
}
bytesVar, err := json.Marshal(example)
if err != nil {
fmt.Println(err)
}
Both are of the type []uint8, but look very different. Any ideas on why? And how I can make sure that they are always the same?
EDIT: Even using bytesVar := []byte(example) results in the same issue.
EDIT:
bytesJSON looks like:
[123 10 32 32 32 32 34 110 97 109 101 34 58 32 34 101 120 97 109 112 108 101 34 44 10 32 32 32 32 34 116 121 112 101 34 58 32 34 114 101 99 111 114 100 34 10 125]
bytesVar looks like:
[34 112 117 98 115 117 98 95 101 120 97 109 112 108 101 95 116 111 112 105 99 34]
when printed to stdout.

Note: The "edit" output in the question is using different example input than in the question.
If we print them as strings it becomes clear.
fmt.Println(string(bytesJSON))
{
"name": "example",
"type": "record",
}
ioutil.ReadFile is just what's in the file.
fmt.Println(string(bytesVar))
"{\n \"name\": \"example\",\n \"type\": \"record\",\n }"
json.Marshal has encoded the string example as JSON. That is a JSON string containing a string.
The equivalent to ioutil.ReadFile("example.json") is simply example.
If we unmarshal bytesVar we get back the original string in example.
var unmarshal string;
json.Unmarshal(bytesVar,&unmarshal)
fmt.Println(unmarshal)
{
"name": "example",
"type": "record",
}

Related

Parsing data with e.parameters

I'm receiving the data from the JavaScript client in the format of:
type ContactDetails struct {
Email string
Subject string
Message string
Color1 []string
Color2 []string
}
The data had been sent from the PostForm as:
Post from website! r.PostFrom = map[color1:[my color 1 - 1 my color 1 - 2] color2:[my color 2 - 1 my color 2 - 2] email:[my email] message:[my message] subject:[my subject]]
So, I thought the best way to handle it the Google Apps script is using e.parameters not e.parameter, pls correct me if this is wrong approach.
In my GAS web app, only first field in both color1 and color2 had been posted in the related spreadsheet columns, though I expected to see array of 2 elements in each.
Sure, "Result of the code" is what I see after executed the codes in the question, "Expected result" is what I was expecting the code in the question will give as an output. The "Targeted final result" is the final result I'm working to approach, I think if I fixed the code to give the "Expected result", then it should not be difficult to tune it to generate the final output as needed in the format of "Targeted final result"
My GAS script is:
function doPost(e){
output = handleResponse(e)
}
function handleResponse(e) {
var result, message;
// Prevent concurrent access overwriting data
// we want a public lock, one that locks for all invocations
var lock = LockService.getPublicLock();
lock.waitLock(1000); // wait 1 seconds before conceding defeat.
// As we are passing JSON in the body, we need to unpairse it
var jsonString = e.postData.getDataAsString();
e.parameters = JSON.parse(jsonString);
try {
// next set where we write the data - you could write to multiple/alternate destinations
var doc = SpreadsheetApp.openById(FILE_Id);
var sheet = doc.getSheetByName(DATA_SHEET);
// we'll assume header is in row 1 but you can override with header_row in GET/POST data
var headers = sheet.getRange(1, 1, 1, sheet.getLastColumn()).getValues()[0];
var lastRow = sheet.getLastRow()
var nextRow = lastRow + 1; // get next row
var row = [];
if(lastRow < 10){
RefID = "PRF.00" + lastRow
} else {
if(lastRow < 100){
RefID = "PRF.0" + lastRow
} else {
RefID = "PRF." + lastRow
}
}
// loop through the header columns
for (i in headers){
if (headers[i] == "Ref"){ // special case if you include a 'Timestamp' column
row.push(RefID);
} else { // else use header name to get data
if (headers[i] == "Timestamp"){ // special case if you include a 'Timestamp' column
row.push(new Date());
} else { // else use header name to get data
row.push(e.parameters[headers[i]]);
}
}
}
// more efficient to set values as [][] array than individually
sheet.getRange(nextRow, 1, 1, row.length).setValues([row]);
// return json success results
result = true;
message = link;
} catch(e){
// if error return this
result = false;
message = e;
} finally { //release lock
lock.releaseLock();
var output = JSON.stringify({"result":result, "message": message});
}
return output;
}
I'm publishing my GAS a:
If info about my client side is required, here the details:
I'm using GO lang as below;
// go build -ldflags "-H=windowsgui"
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"strings"
"text/template"
"github.com/zserge/lorca"
)
// ContactDetails ...
type ContactDetails struct {
Email string
Subject string
Message string
Color1 []string
Color2 []string
}
// ReturnedResult ...
type ReturnedResult struct {
Result bool `json:"result"`
Message string `json:"message"`
}
func index(w http.ResponseWriter, r *http.Request) {
tmpl := template.Must(template.ParseFiles("forms.html"))
if r.Method != http.MethodPost {
tmpl.Execute(w, nil)
return
}
r.ParseForm()
details := ContactDetails{
Email: r.FormValue("email"),
Subject: r.FormValue("subject"),
Message: r.FormValue("message"),
Color1: r.Form["color1"], // as "color1" is array,
Color2: r.Form["color2"], // as "color2" is array,
}
fmt.Printf("Post from website! r.PostFrom = %v\n", r.PostForm)
sheetID := "AKfycbxfMucXOzX15tfU4errRSAa9IzuTRbHzvUdRxzzeYnNA8Ynz8LJuBuaMA/exec"
url := "https://script.google.com/macros/s/" + sheetID + "/exec"
bytesRepresentation, err := json.Marshal(details)
if err != nil {
log.Fatalln(err)
}
resp, err := http.Post(url, "application/json", bytes.NewBuffer(bytesRepresentation))
if err != nil {
log.Fatalln(err)
}
// read all response body
data, _ := ioutil.ReadAll(resp.Body)
// close response body
resp.Body.Close()
webReturn := ReturnedResult{}
if err := json.Unmarshal([]byte(data), &webReturn); err != nil {
panic(err)
}
fmt.Println(webReturn.Message)
//tmpl.Execute(w, struct{ Success bool }{webReturn.Result})
tmpl.Execute(w, webReturn)
}
func main() {
// Start Host goroutine
go func() {
http.HandleFunc("/", index)
http.ListenAndServe(":8090", nil)
}()
// Start UI
ui, err := lorca.New("http://localhost:8090/index", "", 480, 320)
if err != nil {
log.Fatal(err)
}
defer ui.Close()
<-ui.Done()
}
With the html in the below template forms.html
<title>Form Submittal</title>
<h1>Contact</h1>
<form method="POST">
<label>Email:</label><br />
<input type="text" name="email"><br />
<label>Subject:</label><br />
<input type="text" name="subject"><br />
<label>Message:</label><br />
<textarea name="message"></textarea><br />
<table>
<tr>
<td><input type="text" name="color1" /></td>
<td><input type="text" name="color2" /></td>
</tr>
<tr>
<td><input type="text" name="color1" /></td>
<td><input type="text" name="color2" /></td>
</tr>
<table>
<input type="submit">
</form>
{{if .Result}}
<div id='foo'>
<a href={{.Message}}>Download PDF file</a>
</div>
<h1></h1>
<script>
// setTimeout(function () {document.querySelector('#foo').style.display='none'}, 5000);
</script>
{{end}}
The console output data is:
Post from website! r.PostFrom = map[color1:[my color 1 - 1 my color 1 - 2] color2:[my color 2 - 1 my color 2 - 2] email:[my email] message:[my message] subject:[my subject]]
Details = {my email my subject my message [my color 1 - 1 my color 1 - 2] [my color 2 - 1 my color 2 - 2]}
bytesRepresentation = [123 34 69 109 97 105 108 34 58 34 109 121 32 101 109 97 105 108 34 44 34 83 117 98 106 101 99 116 34 58 34 109 121 32 115 117 98 106 101 99 116 34 44 34 77 101 115 115 97 103 101 34 58 34 109 121 32 109 101 115 115 97 103 101 34 44 34 67 111 108 111 114 49 34 58 91 34 109 121 32 99 111 108 111 114 32 49 32 45 32 49 34 44 34 109 121 32 99 111 108 111 114 32 49 32 45 32 50 34 93 44 34 67 111 108 111 114 50 34 58 91 34 109 121 32 99 111 108 111 114 32 50 32 45 32 49 34 44 34 109 121 32 99 111 108 111 114 32 50 32 45 32 50 34 93 125]
In order to achieve your goal of "Expected result", how about the following modification? In this case, please modify handleResponse() of your Google Apps Script as follows.
From:
row.push(e.parameters[headers[i]]);
To:
var temp = e.parameters[headers[i]];
row.push(Array.isArray(temp) ? temp.join(",") : temp);
// Or if you want to ensure empty objects are excluded from the .join(), you can use:
// row.push(Array.isArray(temp) ? (temp.filter(value => Object.keys(value).length !== 0)).join(",") : temp);
Note:
When you modified the script of Web Apps, please redeploy the Web Apps as new version. By this, the latest script is reflected to the Web Apps. Please be careful this.

Why do I convert map to json, map includes list values which is nothing after converting to json

func Test_JsonTtransfer(t *testing.T) {
uid := "306"
phoneList := list.New()
phoneList.PushBack("18513622928")
fmt.Println("phoneList=======", phoneList.Len())
jsonPhoneList, err := json.Marshal(phoneList)
if err != nil {
fmt.Println("error:", err)
}
fmt.Println("jsonPhoneList=======", string(jsonPhoneList))
idCardList := list.New()
idCardList.PushBack("230405197608040640")
request := make(map[string]interface{})
request["uid"] = uid
request["phones"] = phoneList
request["id_cards"] = idCardList
json, err := json.Marshal(request)
if err != nil {
fmt.Println("error:", err)
}
fmt.Println("json=======", json)
fmt.Println("json=======", string(json))
}
Output:
D:/Sys/server/Go\bin\go.exe test -v golang-test/com/http/test -run ^Test_JsonTtransfer$
phoneList======= 1
jsonPhoneList======= {}
json======= [123 34 105 100 95 99 97 114 100 115 34 58 123 125 44 34 112 104 111 110 101 115 34 58 123 125 44 34 117 105 100 34 58 34 51 48 54 34 125]
json======= {"id_cards":{},"phones":{},"uid":"306"}
ok golang-test/com/http/test 0.482s
Phones should be list values, but nothing. Help me.
Because the List type has no exported fields and the type does not implement the Marshalerinterface, List values always marshal to the text {}.
A simple fix is to use a slice instead of a list:
var phoneList []string
phoneList = append(phoneList, "18513622928")
fmt.Println("phoneList=======", len(phoneList)
playground example

convert plenty of json objects into dataframe R

I have plenty of json objects under the same format in one json file
like below. And I want to convert them into R dataframe and then to extract all the value of lantency.But when I enter the command
json_data <- fromJSON(file=json_flie)
only the first json object is stored in the dataframe, so what should I do???
Thanks!
{"task":[{"type":"ping","id":1,"value":" 159 159 152 153 149 147 150 151 148 149","IsFinished":true},{"type":"latency","id":2,"value":147,"IsFinished":true},{"type":"throughput","id":3,"value":"","IsFinished":false},{"type":"DNS","id":4,"value":12,"IsFinished":true}],"measurementTimes":10,"url":""}{"task":[{"type":"ping","id":1,"value":" 166 165 179 181 159 162 166 159 161 162","IsFinished":true},{"type":"latency","id":2,"value":159,"IsFinished":true},{"type":"throughput","id":3,"value":"","IsFinished":false},{"type":"DNS","id":4,"value":7,"IsFinished":true}],"measurementTimes":10,"url":""}{"task":[{"type":"ping","id":1,"value":" 172 172 159 160 159 159 159 158 160 162","IsFinished":true},{"type":"latency","id":2,"value":158,"IsFinished":true},{"type":"throughput","id":3,"value":"","IsFinished":false},{"type":"DNS","id":4,"value":14,"IsFinished":true}],"measurementTimes":10,"url":""}{"task":[{"type":"ping","id":1,"value":" 182 192 171 184 160 159 156 157 180 171","IsFinished":true},{"type":"latency","id":2,"value":156,"IsFinished":true},{"type":"throughput","id":3,"value":"","IsFinished":false},{"type":"DNS","id":4,"value":26,"IsFinished":true}],"measurementTimes":10,"url":""}{"task":[{"type":"ping","id":1,"value":" 158 186 168 189 190 233 168 160 188 157","IsFinished":true},{"type":"latency","id":2,"value":157,"IsFinished":true},{"type":"throughput","id":3,"value":"","IsFinished":false},{"type":"DNS","id":4,"value":1,"IsFinished":true}],"measurementTimes":10,"url":""}
Your input JSON is malformed, and has multiple elements "task" at the root level. This is akin to defining an XML document with more than one root, which is of course not allowed. If you create an outer element which contains an array of "task" elements, then you will be able to successfully load the file into R using fromJSON. Here is what the file should look like:
{
"root" : [
{
"task":
[
{"type":"ping","id":1,"value":" 159 159 152 153 149 147 150 151 148 149","IsFinished":true},
{"type":"latency","id":2,"value":147,"IsFinished":true},
{"type":"throughput","id":3,"value":"","IsFinished":false},
{"type":"DNS","id":4,"value":12,"IsFinished":true}
],
"measurementTimes":10,
"url":""
},
{
"task":
[
{"type":"ping","id":1,"value":" 166 165 179 181 159 162 166 159 161 162","IsFinished":true},
{"type":"latency","id":2,"value":159,"IsFinished":true},
{"type":"throughput","id":3,"value":"","IsFinished":false},\
{"type":"DNS","id":4,"value":7,"IsFinished":true}
],
"measurementTimes":10,
"url":""
},
... and so on for other entries
]
}
Here is what I saw in the R console:
> summary(json_data)
Length Class Mode
root 5 -none- list
And entering the variable name json_data gave me a dump on the entire JSON structure.

Go unmarshalling JSON from compessed HTTP: invalid character looking for beginning of value

I've just written my first Go application which downloads and unmarshales simple JSON object over http. Http content is compressed:
'content-encoding': 'deflate'
I used several well-known examples (like this). Unfortunately the application fails to parse the desired JSON with quite rare and strange error. I wasn't able to find out what's the problem. Any help will be appreciated.
JSON input
(Python was used for debugging)
In [8]: r = requests.get("http://172.17.0.31:20000/top")
In [9]: r.text
Out[9]: u'{"timestamp":{"tv_sec":1428447555,"tv_usec":600186},"string_timestamp":"2015-04-07 22:59:15.600186","monitor_status":"enabled"}'
In [18]: r.headers
Out[18]: {'content-length': '111', 'content-type': 'application/json', 'connection': 'close', 'content-encoding': 'deflate'}
Source code (UPDATED according to the answers)
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
)
type Top struct {
Timestamp Timestamp `json:"timestamp"`
String_timestamp string `json:"string_timestamp"`
Monitor_status string `json:"monitor_status"`
}
type Timestamp struct {
Tv_sec int `json:"tv_sec"`
Tv_usec int `json:"tv_usec"`
}
func get_content() {
url := "http://172.17.0.31:20000/top"
res, err := http.Get(url)
if err != nil {
panic(err.Error())
}
fmt.Println(res)
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err.Error())
}
fmt.Println(body)
var jsondata Top
err = json.Unmarshal(body, &jsondata)
if err != nil {
panic(err.Error())
}
fmt.Println(jsondata)
}
func main() {
get_content()
}
Error
[vitaly#thermaltake elliptics-manager]$ go run main.go
&{200 OK 200 HTTP/1.1 1 1 map[Content-Type:[application/json] Content-Length:[111] Content-Encoding:[deflate]] 0xc20803e340 111 [] true map[] 0xc208028820 <nil>}
[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]
panic: invalid character 'x' looking for beginning of value
UPD: Thanks everyone. Now it's obvious that the reason of this issue was a deflate compression of HTTP response. However, it's still not clear how to perform a decompression in Golang (see here).
The Go JSON marshaller can only marshal unicode strings. It seems that your JSON is not encoded in unicode, but with some other encoding (deflate?).
If you take your bytes stream:
[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]
And try to get a unicode string out of it:
body := []byte{120, 156, 77, 203, 65, 14, 130, 48, 16, 70, 225, 171, 152, 127, 93, 76, 59, 51, 162, 244, 50, 13, 96, 99, 154, 216, 98, 232, 192, 134, 112, 119, 81, 55, 110, 95, 190, 183, 65, 83, 142, 85, 251, 252, 130, 223, 160, 107, 168, 113, 132, 119, 66, 55, 145, 182, 117, 108, 62, 109, 249, 70, 98, 234, 108, 183, 27, 84, 157, 83, 121, 132, 191, 19, 100, 221, 165, 177, 210, 216, 235, 137, 200, 11, 123, 230, 243, 207, 195, 32, 79, 37, 233, 52, 135, 3, 235, 82, 15, 29, 75, 63, 60, 227, 29, 251, 27, 195, 90, 38, 189}
fmt.Println(string(body))
You would see a weird (compressed?) string in the console, not JSON.
I guess that the python http client automatically decompresses the deflated bytes while the Go http client does not (I know it does so for gzip but not sure if for deflate). You would have to read out the deflated bytes and convert them into a unicode string before you will be able to use the JSON marshaller to parse them.
I don't know about 'x', but struct fields must be public (start with a capital letter) to be considered by the json Unmarshaller. Of course then the names don't match the json keys, and you have to add json annotations like so:
type Top struct {
Timestamp Timestamp `json:"timestamp"`
String_timestamp string `json:"string_timestamp"`
Monitor_status string `json:"monitor_status"`
}
I believe this is due to you double encoding. ioutil.ReadAll(res.Body) returns a []byte so when you do []byte(body) you're casting what is already a byte array, my guess is that first bytes UTF value is x. Just update this; json.Unmarshal([]byte(body), &jsondata) to json.Unmarshal(body, &jsondata) and I bet it will unmarshal just fine.
Also, unrelated to your error but as pointed out in the other answer if you do not export the fields in your struct (in go this means start the field name with a capital letter) then the unmarshaler will not be able to make user of them. To make that work you need to update you type to;
type Top struct {
Timestamp Timestamp `json:"timestamp"`
String_timestamp string `json:"string_timestamp"`
Monitor_status string `json:"monitor_status"`
}
The json annotations are required because the unmarshaler is very strict and requires exact matches (case sensitive) for field names.
Try this please
func get_content() {
url := "http://172.17.0.31:20000/top"
res, err := http.Get(url)
if err != nil {
panic(err.Error())
}
defer res.Body.Close()
fmt.Println("res body:", res.Body)
body, err := ioutil.ReadAll(resp=.Body)
fmt.Println("body:", body)
re, err := zlib.NewReader(bytes.NewReader(body))
fmt.Println("zlib:", re)
enflated, err := ioutil.ReadAll(re)
fmt.Println("enflated:", string(enflated))
var jsondata Top
err = json.Unmarshal(body, &jsondata)
if err != nil {
panic(err.Error())
}
fmt.Println(jsondata)
}
and ensure http://172.17.0.31:20000/top return json type.

Why is json.Marshal seemingly producing an array of ints?

I'm trying to wrap my head around the Go language and I've hit my first stumbling block with this simple example:
package main
import (
"encoding/json"
"fmt"
)
type MyStructure struct {
Integer int `json:"integer"`
Label string `json:"label"`
}
func main() {
ms := &MyStructure{9001, "over9000"}
msjson, _ := json.Marshal(ms)
fmt.Println(msjson) // expect: {"integer": 9001, "label": "over9000"}
}
My output is as follows: [123 34 105 110 116 101 103 101 114 34 58 57 48 48 49 44 34 108 97 98 101 108 34 58 34 111 118 101 114 57 48 48 48 34 125]
I'm clearly missing something obvious; could someone please point me in the right direction?
It produces a byte slice (ref : http://golang.org/pkg/encoding/json/#Marshal), use string(msjson) to get the string.
Also never ignore errors, it bites you back whenever you least expect it.
fmt.Println(string(msjson))
// or
fmt.Printf("%s\n", msjson) //shamelessly taken from #dustin's comment