I have to unmarshal Json(Request, Response Body) in echo framework midleware for ELK, like this code.
var reqJSONBody, resJSONBody map[string]interface{}
if len(*reqBody) > 0 {
if err := unmarshalJSON(reqBody, &reqJSONBody); err != nil {
gl.ServiceLogger.Error("error parsing the request body: ", requestURI, err)
}
encryptPrivacyField(&reqJSONBody)
}
if len(*resBody) > 0 && resContentType != "" && strings.Contains(resContentType, "application/json") {
if err := unmarshalJSON(resBody, &resJSONBody); err != nil {
gl.ServiceLogger.Error("error parsing the response body: ", requestURI, err)
}
encryptPrivacyField(&resJSONBody)
}
And it is work,
But, Some URI response to []map[string]interface{} type.
So I got this error.
json: cannot unmarshal array into Go value of type map[string]interface {}
What is the best way to solve the problem?
I solved it by changing:
var reqJSONBody, resJSONBody interface{}
if len(*reqBody) > 0 {
if err := json.Unmarshal(*reqBody, &reqJSONBody); err != nil {
gl.ServiceLogger.Error("error parsing the request body: ", requestURI, err)
}
encryptPrivacyField(&reqJSONBody)
}
So I have to change encryptPrivacyField method like this:
func encryptPrivacyField(data *interface{}) {
switch reflect.TypeOf(*data).Kind() {
case reflect.Map:
for _, field := range getPrivacyFieldList() {
if item, ok := (*data).(map[string]interface{})[field]; ok && item != nil {
(*data).(map[string]interface{})[field] = db.NewEncString(fmt.Sprintf("%v", (*data).(map[string]interface{})[field]))
}
}
for _, field := range getHashFieldList() {
if item, ok := (*data).(map[string]interface{})[field]; ok && item != nil {
(*data).(map[string]interface{})[field] = db.NewHashString(fmt.Sprintf("%v", (*data).(map[string]interface{})[field]))
}
}
case reflect.Slice:
for index, _ := range (*data).([]interface{}) {
encryptPrivacyField(&(*data).([]interface{})[index])
}
}
}
Thanks to #mkopriva for the advice.
Related
I have a code like this
scanner := bufio.NewScanner(reader)
scanner.Split(splitJSON)
for scanner.Scan() {
bb := scanner.Bytes()
}
I would like to get from Scanner only valid JSON objects one at a time. In some case in Scanner may be bytes that represent struct like this
{
"some_object": "name",
"some_fileds": {}
}
{
"some_object":
}
I need only the first part of this
{
"some_object": "name",
"some_fileds": {}
}
For the other, I should wait for the end of JSON object.
I have a function like this, but it's horrible and doesn't work.
func splitJSON(
bb []byte, atEOF bool,
) (advance int, token []byte, err error) {
print(string(bb))
if len(bb) < 10 {
return 0, nil, nil
}
var nested, from, to int
var end bool
for i, b := range bb {
if string(b) == "{" {
if end {
to = i
break
}
if nested == 0 {
from = i
}
nested++
}
if string(b) == "}" {
nested--
if nested == 0 {
to = i
end = true
}
}
}
if atEOF {
return len(bb), bb, nil
}
return len(bb[from:to]), bb[from:to], nil
}
UPD
It was decided by this splitFunc
func splitJSON(data []byte, atEOF bool) (advance int, token []byte, err error) {
if atEOF && len(data) == 0 {
return 0, nil, nil
}
reader := bytes.NewReader(data)
dec := json.NewDecoder(reader)
var raw json.RawMessage
if err := dec.Decode(&raw); err != nil {
return 0, nil, nil
}
return len(raw) + 1, raw, nil
}
Use json.Decoder for this. Each Decoder.Decode() call will decode the next JSON-encoded value from the input, JSON objects in your case.
If you don't want to decode the JSON objects just need the JSON data (byte slice), use a json.RawMessage to unmarshal into.
For example:
func main() {
reader := strings.NewReader(src)
dec := json.NewDecoder(reader)
for {
var raw json.RawMessage
if err := dec.Decode(&raw); err != nil {
if err == io.EOF {
break
}
fmt.Printf("Error:", err)
return
}
fmt.Println("Next:", string(raw))
}
}
const src = `{
"some_object": "name",
"some_fileds": {}
}
{
"some_object": "foo"
}`
This will output (try it on the Go Playground):
Next: {
"some_object": "name",
"some_fileds": {}
}
Next: {
"some_object": "foo"
}
I am trying to build a TCP server that loads dataset from a CSV file and provide an interface to query the dataset. TCP server will expose port 4040. CSV file contains the following columns related to corona virus cases:
Cumulative Test Positive
Cumulative Tests Performed
Date
Discharged
Expired
Admitted
Region
Users should be able to connect to the server using NetCat nc localhost 4040 command on Linux/Unix based systems.
Once connected to TCP, the user should be able to communicate with the application by sending queries in JSON format.
{
"query": {
"region": "Sindh"
}
}
{
"query": {
"date": "2020-03-20"
}
}
My server.go
package main
import (
"fmt"
"net"
"os"
"flag"
"log"
"encoding/csv"
"encoding/json"
"bufio"
"io"
"strings"
)
type CovidPatient struct {
Positive string `json:"Covid_Positive"`
Performed string `json:"Coivd_Performed"`
Date string `json:"Covid_Date"`
Discharged string `json:"Covid_Discharged"`
Expired string `json:"Covid_Expired"`
Region string `json:"Covid_Region"`
Admitted string `json:"Covid_Admitted"`
}
type DataRequest struct {
Get string `json:"get"`
}
type DataError struct {
Error string `json:"Covid_error"`
}
func Load(path string) []CovidPatient {
table := make([]CovidPatient, 0)
var patient CovidPatient
file, err := os.Open(path)
if err != nil {
panic(err.Error())
}
defer file.Close()
reader := csv.NewReader(file)
csvData, err := reader.ReadAll()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, row := range csvData{
patient.Positive = row[0]
patient.Performed = row[1]
patient.Date = row[2]
patient.Discharged = row[3]
patient.Expired = row[4]
patient.Region = row[5]
patient.Admitted = row[6]
table = append(table, patient)
}
return table
}
func Find(table []CovidPatient, filter string) []CovidPatient {
if filter == "" || filter == "*" {
return table
}
result := make([]CovidPatient, 0)
filter = strings.ToUpper(filter)
for _, cp := range table {
if cp.Date == filter ||
cp.Region == filter ||
strings.Contains(strings.ToUpper(cp.Positive), filter) ||
strings.Contains(strings.ToUpper(cp.Performed), filter) ||
strings.Contains(strings.ToUpper(cp.Date), filter) ||
strings.Contains(strings.ToUpper(cp.Discharged), filter) ||
strings.Contains(strings.ToUpper(cp.Expired), filter) ||
strings.Contains(strings.ToUpper(cp.Region), filter) ||
strings.Contains(strings.ToUpper(cp.Admitted), filter){
result = append(result, cp)
}
}
return result
}
var (
patientsDetail = Load("./covid_final_data.csv")
)
func main(){
var addr string
var network string
flag.StringVar(&addr, "e", ":4040", "service endpoint [ip addr or socket path]")
flag.StringVar(&network, "n", "tcp", "network protocol [tcp,unix]")
flag.Parse()
switch network {
case "tcp", "tcp4", "tcp6", "unix":
default:
fmt.Println("unsupported network protocol")
os.Exit(1)
}
ln, err := net.Listen(network, addr)
if err != nil {
log.Println(err)
os.Exit(1)
}
defer ln.Close()
log.Println("Covid19 Condition in Pakistan")
log.Printf("Service started: (%s) %s\n", network, addr)
for {
conn, err := ln.Accept()
if err != nil {
log.Println(err)
conn.Close()
continue
}
log.Println("Connected to ", conn.RemoteAddr())
go handleConnection(conn)
}
}
func handleConnection(conn net.Conn) {
defer func() {
if err := conn.Close(); err != nil {
log.Println("error closing connection:", err)
}
}()
reader := bufio.NewReaderSize(conn, 4)
for {
buf, err := reader.ReadSlice('}')
if err != nil {
if err != io.EOF {
log.Println("connection read error:", err)
return
}
}
reader.Reset(conn)
var req DataRequest
if err := json.Unmarshal(buf, &req); err != nil {
log.Println("failed to unmarshal request:", err)
cerr, jerr := json.Marshal(DataError{Error: err.Error()})
if jerr != nil {
log.Println("failed to marshal DataError:", jerr)
continue
}
if _, werr := conn.Write(cerr); werr != nil {
log.Println("failed to write to DataError:", werr)
return
}
continue
}
result := Find(patientsDetail, req.Get)
rsp, err := json.Marshal(&result)
if err != nil {
log.Println("failed to marshal data:", err)
if _, err := fmt.Fprintf(conn, `{"data_error":"internal error"}`); err != nil {
log.Printf("failed to write to client: %v", err)
return
}
continue
}
if _, err := conn.Write(rsp); err != nil {
log.Println("failed to write response:", err)
return
}
}
}
This correctly loads the csv and convert it into JSON. But, when I try to run query using NetCat command it return empty JSON element. Kindly guide me where is error.
Guess you want this:
╭─root#DESKTOP-OCDRD7Q ~
╰─# nc localhost 4040
{"get": "Sindh"}
[{"Covid_Positive":"1","Coivd_Performed":"1","Covid_Date":"1","Covid_Discharged":"1","Covid_Expired":"1","Covid_Region":"Sindh","Covid_Admitted":"1"}]
What you should do is just to modify your json request.
package main
import (
"bufio"
"encoding/csv"
"encoding/json"
"flag"
"fmt"
"io"
"log"
"net"
"os"
)
type CovidPatient struct {
Positive string `json:"Covid_Positive"`
Performed string `json:"Coivd_Performed"`
Date string `json:"Covid_Date"`
Discharged string `json:"Covid_Discharged"`
Expired string `json:"Covid_Expired"`
Region string `json:"Covid_Region"`
Admitted string `json:"Covid_Admitted"`
}
type DataRequest struct {
Get CovidPatient `json:"get"`
}
type DataError struct {
Error string `json:"Covid_error"`
}
func Load(path string) []CovidPatient {
table := make([]CovidPatient, 0)
var patient CovidPatient
file, err := os.Open(path)
if err != nil {
panic(err.Error())
}
defer file.Close()
reader := csv.NewReader(file)
csvData, err := reader.ReadAll()
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, row := range csvData {
patient.Positive = row[0]
patient.Performed = row[1]
patient.Date = row[2]
patient.Discharged = row[3]
patient.Expired = row[4]
patient.Region = row[5]
patient.Admitted = row[6]
table = append(table, patient)
}
return table
}
func Find(table []CovidPatient, filter CovidPatient) []CovidPatient {
result := make([]CovidPatient, 0)
log.Println(filter, table)
for _, cp := range table {
if filter.Positive == "" {
} else if filter.Positive != cp.Positive {
continue
}
if filter.Performed == "" {
} else if filter.Performed != cp.Performed {
continue
}
if filter.Date == "" {
} else if filter.Date != cp.Date {
continue
}
if filter.Discharged == "" {
} else if filter.Discharged != cp.Discharged {
continue
}
if filter.Expired == "" {
} else if filter.Expired != cp.Expired {
continue
}
if filter.Region == "" {
} else if filter.Region != cp.Region {
continue
}
if filter.Admitted == "" {
} else if filter.Admitted != cp.Admitted {
continue
}
result = append(result, cp)
}
return result
}
var (
patientsDetail = Load("./covid_final_data.csv")
)
func main() {
log.SetFlags(log.Lshortfile | log.Ltime)
var addr string
var network string
flag.StringVar(&addr, "e", ":4040", "service endpoint [ip addr or socket path]")
flag.StringVar(&network, "n", "tcp", "network protocol [tcp,unix]")
flag.Parse()
switch network {
case "tcp", "tcp4", "tcp6", "unix":
default:
fmt.Println("unsupported network protocol")
os.Exit(1)
}
ln, err := net.Listen(network, addr)
if err != nil {
log.Println(err)
os.Exit(1)
}
defer ln.Close()
log.Println("Covid19 Condition in Pakistan")
log.Printf("Service started: (%s) %s\n", network, addr)
for {
conn, err := ln.Accept()
if err != nil {
log.Println(err)
conn.Close()
continue
}
log.Println("Connected to ", conn.RemoteAddr())
go handleConnection(conn)
}
}
func handleConnection(conn net.Conn) {
defer func() {
if err := conn.Close(); err != nil {
log.Println("error closing connection:", err)
}
}()
reader := bufio.NewReaderSize(conn, 100)
for {
buf, err := reader.ReadBytes('|')
if err != nil {
if err != io.EOF {
log.Println("connection read error:", err)
return
}
}
reader.Reset(conn)
var req DataRequest
if err := json.Unmarshal(buf[:len(buf)-1], &req); err != nil {
log.Println("failed to unmarshal request:", string(buf), err)
cerr, jerr := json.Marshal(DataError{Error: err.Error()})
if jerr != nil {
log.Println("failed to marshal DataError:", jerr)
continue
}
if _, werr := conn.Write(cerr); werr != nil {
log.Println("failed to write to DataError:", werr)
return
}
continue
}
result := Find(patientsDetail, req.Get)
rsp, err := json.Marshal(&result)
if err != nil {
log.Println("failed to marshal data:", err)
if _, err := fmt.Fprintf(conn, `{"data_error":"internal error"}`); err != nil {
log.Printf("failed to write to client: %v", err)
return
}
continue
}
if _, err := conn.Write(rsp); err != nil {
log.Println("failed to write response:", err)
return
}
}
}
The query is:
╭─root#DESKTOP-OCDRD7Q ~
╰─# nc localhost 4040 127 ↵
{
"get": {
"Covid_Region": "Sindh",
"Covid_Date": "2020-03-20"
}
}|
[{"Covid_Positive":"1","Coivd_Performed":"1","Covid_Date":"2020-03-20","Covid_Discharged":"1","Covid_Expired":"1","Covid_Region":"Sindh","Covid_Admitted":"1"}]
Inside function handleConnection, the first thing is "read until you find the first }", imagine the user is sending the request:
{ "get": { "Covid_Region": "Sindh", "Covid_Date": "2020-03-20" } }
then that step read:
{ "get": { "Covid_Region": "Sindh", "Covid_Date": "2020-03-20" }
Notice the trailing } is missing, then the json.Unmarshal is trying to unmarshal the query without the last } (which is an invalid json).
This problem can take advantage of JSON streaming decoding, in other words, use json.NewDecoder(r io.Reader) instead of json.Unmarshal. Let me copy and modify the first part of that function:
func handleConnection(conn net.Conn) {
defer func() {
if err := conn.Close(); err != nil {
log.Println("error closing connection:", err)
}
}()
jsonDecoder := json.NewDecoder(conn) // A json decoder read a stream to find a
// valid JSON and stop just the byte
// after the JSON ends. Process can be
// repeated.
for {
var req DataRequest
err := jsonDecoder.Decode(&req)
if err == io.EOF {
log.Println("finish")
return
}
if err != nil {
log.Println("unmarshal:", err)
return
}
result := Find(patientsDetail, req.Get) // Here query the system
// ...
Probably now it works, but you can also take advantage of json streaming to send the response back with a jsonEncoder := json.NewEncoder(conn) before de for loop and sending the request like this:
err := jsonEncoder.Encode(&result)
if err != nil {
log.Println("failed to marshal data:", err)
// ...
continue
}
I am having trouble Unmarshalling some Json I don't have control over.
There is one field that 99% of the time is a string but occasionally is an array.
type MyListItem struct {
Date string `json:"date"`
DisplayName string `json:"display_name"`
}
type MyListings struct {
CLItems []MyListItem `json:"myitems"`
}
var mylist MyListings
err = json.Unmarshal(jsn, &mylist)
if err != nil {
fmt.Print("JSON:\n%s\n error:%v\n", string(jsn),err)
return
}
Json is as follows:
{
"date": "30 Apr",
"display_name": "Mr Smith"
},
{
"date": "30 Apr",
"display_name": ["Mr Smith", "Mr Jones"],
}
error: json: cannot unmarshal array into Go struct field MyListItem.display_name of type string
Use json.RawMessage to capture the varying field.
Use the json "-" name to hide the DisplayName field from decoder. The application will fill this field after the top-level JSON is decoded.
type MyListItem struct {
Date string `json:"date"`
RawDisplayName json.RawMessage `json:"display_name"`
DisplayName []string `json:"-"`
}
Unmarshal the top-level JSON:
var li MyListItem
if err := json.Unmarshal(data, &li); err != nil {
// handle error
}
Unmarshal the display name depending on the type of the raw data:
if len(li.RawDisplayName) > 0 {
switch li.RawDisplayName[0] {
case '"':
if err := json.Unmarshal(li.RawDisplayName, &li.DisplayName); err != nil {
// handle error
}
case '[':
var s []string
if err := json.Unmarshal(li.RawDisplayName, &s); err != nil {
// handle error
}
// Join arrays with "&" per OP's comment on the question.
li.DisplayName = strings.Join(s, "&")
}
}
playground example
Incorporate the above into a for loop to handle MyListings:
var listings MyListings
if err := json.Unmarshal([]byte(data), &listings); err != nil {
// handle error
}
for i := range listings.CLItems {
li := &listings.CLItems[i]
if len(li.RawDisplayName) > 0 {
switch li.RawDisplayName[0] {
case '"':
if err := json.Unmarshal(li.RawDisplayName, &li.DisplayName); err != nil {
// handle error
}
case '[':
var s []string
if err := json.Unmarshal(li.RawDisplayName, &s); err != nil {
// handle error
}
li.DisplayName = strings.Join(s, "&")
}
}
}
playground example
If there's more than one place in the data model where a value can be a string or []string, it can be helpful to encapsulate the logic in a type. Parse the JSON data in an implementation of the json.Unmarshaler interface.
type multiString string
func (ms *multiString) UnmarshalJSON(data []byte) error {
if len(data) > 0 {
switch data[0] {
case '"':
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*ms = multiString(s)
case '[':
var s []string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*ms = multiString(strings.Join(s, "&"))
}
}
return nil
}
Use it like this:
type MyListItem struct {
Date string `json:"date"`
DisplayName multiString `json:"display_name"`
}
type MyListings struct {
CLItems []MyListItem `json:"myitems"`
}
var listings MyListings
if err := json.Unmarshal([]byte(data), &listings); err != nil {
log.Fatal(err)
}
Playground Example
Here's the code to get the value as a slice of strings instead of as a single string with values joined by &.
type multiString []string
func (ms *multiString) UnmarshalJSON(data []byte) error {
if len(data) > 0 {
switch data[0] {
case '"':
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*ms = multiString{s}
case '[':
if err := json.Unmarshal(data, (*[]string)(ms)); err != nil {
return err
}
}
}
return nil
}
Playground example.
As an alternative, this builds off of the answer from #ThunderCat but instead of using json.RawMessage, uses interface{} and a type switch:
package main
import (
"encoding/json"
"fmt"
"log"
)
type MyListItem struct {
Date string `json:"date"`
DisplayName string `json:"-"`
RawDisplayName interface{} `json:"display_name"`
}
func (li *MyListItem) UnmarshalJSON(data []byte) error {
type localItem MyListItem
var loc localItem
if err := json.Unmarshal(data, &loc); err != nil {
return err
}
*li = MyListItem(loc)
switch li.RawDisplayName.(type) {
case string:
li.DisplayName = li.RawDisplayName.(string)
case []interface{}:
vals := li.RawDisplayName.([]interface{})
if len(vals) > 0 {
li.DisplayName, _ = vals[0].(string)
for _, v := range vals[1:] {
li.DisplayName += "&" + v.(string)
}
}
}
return nil
}
func test(data string) {
var li MyListItem
if err := json.Unmarshal([]byte(data), &li); err != nil {
log.Fatal(err)
}
fmt.Println(li.DisplayName)
}
func main() {
test(`
{
"date": "30 Apr",
"display_name": "Mr Smith"
}`)
test(`
{
"date": "30 Apr",
"display_name": ["Mr Smith", "Mr Jones"]
}`)
}
playground
In the following http handler, I try to distinguish whether the request body is empty
type Request struct {
A bool `json:"lala"`
B bool `json:"kaka"`
C int32 `json:"cc"`
D int32 `json:"dd"`
}
var (
opts Request
hasOpts bool = true
)
err = json.NewDecoder(r.Body).Decode(&opts)
switch {
case err == io.EOF:
hasOpts = false
case err != nil:
return errors.New("Could not get advanced options: " + err.Error())
}
However, even with r.Body equals '{}', hasOpts is still true. Is this to be expected? In that case, how should I detect empty request body?
Read the body first, to check its content, then unmarshal it:
body, err := ioutil.ReadAll(r.Body)
if err != nil {
return err
}
if len(body) > 0 {
err = json.Unmarshal(body, &opts)
if err != nil {
return fmt.Errorf("Could not get advanced options: %s", err)
}
}
I am trying to send a POST request using this function -
{
func (Client *Client) doModify(method string, url string, createObj interface{}, respObject interface{}) error {
bodyContent, err := json.Marshal(createObj)
if err != nil {
return err
}
client := Client.newHttpClient()
req, err := http.NewRequest(method, url, bytes.NewBuffer(bodyContent))
if err != nil {
return err
}
Client.setupRequest(req)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Content-Length", string(len(bodyContent)))
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 300 {
return errors.New(fmt.Sprintf("Bad response from [%s], go [%d]", url, resp.StatusCode))
}
byteContent, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
return json.Unmarshal(byteContent, respObject)
}
}
I am calling my function like this -
{
func TestContainerCreate(t *testing.T) {
client := newClient(t)
container, err := client.Container.Create(&Container{
Name: "name",
ImageUuid: "xyz",
})
if err != nil {
t.Fatal(err)
}
defer client.Container.Delete(container)
}
}
The Create function calls internally calls the doCreate function which calls the doModify function pasted on the top .
{
func (self *ContainerClient) Create(container *Container) (*Container, error) {
resp := &Container{}
err := self.Client.doCreate(container_TYPE, container, resp)
return resp, err
}
}
{
func (Client *Client) doCreate(schemaType string, createObj interface{}, respObject interface{}) error {
if createObj == nil {
createObj = map[string]string{}
}
schema, ok := Client.Types[schemaType]
if !ok {
return errors.New("Unknown schema type [" + schemaType + "]")
}
return Client.doModify("POST", collectionUrl, createObj, respObject)
}
}
This gives me a 422 bad response.On doing further research, When doing a CURL, with "name" and "imageUuid" first letter as small case, gives a 201 created status but when passing "Name" and "ImageUuid" first letter as capital gives 422 bad response. Could there be issue with the json struct defined for container, or case of these entities being defined or something else?
{
curl -X POST -v -s http://localhost:8080/v1/containers -H 'Content-Type: application/json' -d '{"name" : "demo", "imageUuid" : "docker:nginx"}' | python -m 'json.tool'
}
Container struct definition looks like this -
{
type Container struct {
Resource
ImageId string `json:"ImageId,omitempty"`
ImageUuid string `json:"ImageUuid,omitempty"`
MemoryMb int `json:"MemoryMb,omitempty"`
Name string `json:"Name,omitempty"`
}
type ContainerCollection struct {
Collection
Data []Container `json:"data,omitempty"`
}
}
string(len(bodyContent)) isn't doing what you think it is. You're converting a single int to a utf-8 string. You want to use the strconv package to get the numerical representation.
Also note that you can't omitempty an int, since 0 is a valid value.