Parsing array of JSON objects into individual documents - json
I am collecting logs generated by an agent. It generates one large JSON output which I need to decompose into smaller JSON documents and use sarama to write to kafka. Due to MAX size constraint of kafka message, I have problems in demposing into several individual JSON documents. Any suggestions would be greatly appreciated. The log messages do not have any fixed fields or data types except date/time field indicating of logactivity
Sample #1
[{"date":1596206786.847531,"rand_value":11885153394315023285},{"date":1596206787.847446,"rand_value":6208802038498064748},{"date":1596206788.847526,"rand_value":932964293334035461},{"date":1596206789.847568,"rand_value":13217490172547025909}]
Sample 2
[{"date":1596206786.847743,"cpu_p":0,"user_p":0,"system_p":0,"cpu0.p_cpu":0,"cpu0.p_user":0,"cpu0.p_system":0,"cpu1.p_cpu":0,"cpu1.p_user":0,"cpu1.p_system":0,"cpu2.p_cpu":0,"cpu2.p_user":0,"cpu2.p_system":0,"cpu3.p_cpu":0,"cpu3.p_user":0,"cpu3.p_system":0,"cpu4.p_cpu":0,"cpu4.p_user":0,"cpu4.p_system":0,"cpu5.p_cpu":0,"cpu5.p_user":0,"cpu5.p_system":0,"cpu6.p_cpu":0,"cpu6.p_user":0,"cpu6.p_system":0,"cpu7.p_cpu":0,"cpu7.p_user":0,"cpu7.p_system":0},{"date":1596206787.847689,"cpu_p":1.25,"user_p":0.75,"system_p":0.5,"cpu0.p_cpu":2,"cpu0.p_user":1,"cpu0.p_system":1,"cpu1.p_cpu":1,"cpu1.p_user":0,"cpu1.p_system":1,"cpu2.p_cpu":2,"cpu2.p_user":1,"cpu2.p_system":1,"cpu3.p_cpu":3,"cpu3.p_user":2,"cpu3.p_system":1,"cpu4.p_cpu":1,"cpu4.p_user":0,"cpu4.p_system":1,"cpu5.p_cpu":1,"cpu5.p_user":1,"cpu5.p_system":0,"cpu6.p_cpu":2,"cpu6.p_user":2,"cpu6.p_system":0,"cpu7.p_cpu":0,"cpu7.p_user":0,"cpu7.p_system":0},{"date":1596206788.847754,"cpu_p":0.75,"user_p":0.5,"system_p":0.25,"cpu0.p_cpu":0,"cpu0.p_user":0,"cpu0.p_system":0,"cpu1.p_cpu":1,"cpu1.p_user":0,"cpu1.p_system":1,"cpu2.p_cpu":2,"cpu2.p_user":1,"cpu2.p_system":1,"cpu3.p_cpu":0,"cpu3.p_user":0,"cpu3.p_system":0,"cpu4.p_cpu":0,"cpu4.p_user":0,"cpu4.p_system":0,"cpu5.p_cpu":1,"cpu5.p_user":1,"cpu5.p_system":0,"cpu6.p_cpu":1,"cpu6.p_user":0,"cpu6.p_system":1,"cpu7.p_cpu":1,"cpu7.p_user":0,"cpu7.p_system":1},{"date":1596206789.847805,"cpu_p":0.8750000000000001,"user_p":0.5,"system_p":0.375,"cpu0.p_cpu":1,"cpu0.p_user":0,"cpu0.p_system":1,"cpu1.p_cpu":1,"cpu1.p_user":1,"cpu1.p_system":0,"cpu2.p_cpu":2,"cpu2.p_user":1,"cpu2.p_system":1,"cpu3.p_cpu":0,"cpu3.p_user":0,"cpu3.p_system":0,"cpu4.p_cpu":1,"cpu4.p_user":1,"cpu4.p_system":0,"cpu5.p_cpu":0,"cpu5.p_user":0,"cpu5.p_system":0,"cpu6.p_cpu":2,"cpu6.p_user":2,"cpu6.p_system":0,"cpu7.p_cpu":0,"cpu7.p_user":0,"cpu7.p_system":0}]
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"os"
)
func main() {
ibytes, err := ioutil.ReadFile("hello.json")
if err != nil {
fmt.Println(err)
os.Exit(-1)
}
var msgs []map[string]interface{}
err = json.Unmarshal(ibytes, &msgs)
if err != nil {
fmt.Println("Serialization Error", err)
os.Exit(-1)
}
for _,msg:=range msgs {
fmt.Println("%s",msg)
}
}
I am able to iterate over the individual messages but not in a friendly format to write into kafka.
I managed to find solution by myself using below code
var PlaceHolder []interface{}
err=json.Unmarshal(dbytes,&PlaceHolder)
if err!=nil {
return errors.New(fmt.Sprintf("Error during JSON Unmarshalling (%s) ",err))
}
for _,doc:=range PlaceHolder {
event,_:=json.Marshal(doc)
if err!=nil{
log.Println("Skipping: Error during JSON Marshaling (%s) ",err)
continue
}
KafkaMessage:= &sarama.ProducerMessage{
Topic: this.Topic,
Value: sarama.StringEncoder(event),
}
msgs=append(msgs,KafkaMessage)
}
Related
Is there any way to parse json file dynamically GO
I have a json file, but I don't know what the content and format of this json file is. It can change instantly. So I cannot create a struct and parse it according to this struct. Is there a way to dynamically parse this json file and access the values in this json? I couldn't find anything, if you have any help I'm open to it.
Yes, you can parse a JSON file dynamically in Go by using the built-in encoding/json package and an interface type. Here's an example:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
func main() {
// Read the JSON file into memory
file, err := ioutil.ReadFile("file.json")
if err != nil {
fmt.Println(err)
return
}
// Unmarshal the JSON into a dynamic interface
var data interface{}
err = json.Unmarshal(file, &data)
if err != nil {
fmt.Println(err)
return
}
// Access the values using type assertions
m := data.(map[string]interface{})
for k, v := range m {
fmt.Println(k, v)
}
}
In this example, the json.Unmarshal function is used to parse the JSON file into an interface{} type, which is a dynamic type in Go that can hold any value. The values in the JSON can then be accessed using type assertions to convert the dynamic type to a more specific type, such as map[string]interface{}.
Is this what you mean?
using http.PostForm to POST nested json data values?
I've seen the following example of how to use net/http PostForm to send a basic string map as a POST request:
How to send a POST request in Go?
But the data that I need to post is slightly more complicated as it has nested json / nested string maps. An example of the data I need to post:
{"MyAttributes" : {"AttributeOne" : "one", "AttributeTwo":"two"}}
Can net/url Values represent that kind of nested data and/or how do I pass this to net/http PostForm?
It's possible
package main
import (
"encoding/json"
"log"
"net/http"
"net/url"
)
type Attributes struct {
AttributeOne string
AttributeTwo string
}
func main() {
attributes := Attributes{"one", "two"}
data, err := json.Marshal(attributes)
if err != nil {
log.Fatal("bad", err)
}
values := url.Values{}
values.Set("MyAttributes", string(data))
resp, error := http.PostForm("localhost:2021", values)
// use resp and error later
}
How do I get out a specific value out of json in golang
Not quite sure how to access the value I'm interested in. What I have is response from my couchDB that looks like this:
response from couchDB in json format
What I am interested in is to get out the "name" and "phone" value using golang.
If I run this code, I at least get out the value of id or key:
package main
import(
"net/http"
"encoding/json"
"io/ioutil"
"fmt"
)
type rows struct{
Rows []info `json:"rows"`
}
type info struct{
Name string `json:"id"`
}
func main() {
resp, err := http.Get("http://localhost:5984/mydb/_all_docs?include_docs=true")
bytes, _ := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
var d rows
json.Unmarshal(bytes, &d)
fmt.Println(d)
}
Output from script
So I tried doing it this way to get "name" for starter:
package main
import(
"net/http"
"encoding/json"
"io/ioutil"
"fmt"
)
type rows struct{
Rows []doc `json:"rows"`
}
type doc struct {
Values []info `json:"doc"`
}
type info struct{
Name string `json:"name"`
}
func main() {
resp, err := http.Get("http://localhost:5984/mydb/_all_docs?include_docs=true")
bytes, _ := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
var d rows
json.Unmarshal(bytes, &d)
fmt.Println(d)
}
But get this response:
Output from script
I'm pretty new to golang but really want to learn how to build simple API with it, if there are any other ways to do this in golang please let me know
Here's how I would approach this problem 1) get the example json output and visit the website https://mholt.github.io/json-to-go/ Use this to automatically generate a struct that matches your data You may need to adjust what it makes to give you a "row" struct 2) Using the structs you have just devised, load the data with Unmarshall 3) dump the data with the %v Printf format to look at it and then work out how to get at the name and phone elements You would have some example code here but your example data is in a picture so is time consuming to reproduce: sorry :/
Is there a way to extract JSON from an http response without having to build structs?
All of the ways I'm seeing involve building structs and unmarshalling the data into the struct. But what if I'm getting JSON responses with hundreds of fields? I don't want to have to create 100 field structs just to be able to get to the data I want. Coming from a Java background there are easy ways to simply get the http response as a string and then pass the JSON string into a JSON object that allows for easy traversal. It's very painless. Is there anything like this in Go?
Java example in pseudo code:
String json = httpResponse.getBody();
JsonObject object = new JsonObject(json);
object.get("desiredKey");
Golang: fetch JSON from an HTTP response without using structs as helpers
This is a typical scenario we come across. This is achieved by json.Unmarshal.
Here is a simple json
{"textfield":"I'm a text.","num":1234,"list":[1,2,3]}
which is serialized to send across the network and unmarshaled at Golang end.
package main
import (
"fmt"
"encoding/json"
)
func main() {
// replace this by fetching actual response body
responseBody := `{"textfield":"I'm a text.","num":1234,"list":[1,2,3]}`
var data map[string]interface{}
err := json.Unmarshal([]byte(responseBody), &data)
if err != nil {
panic(err)
}
fmt.Println(data["list"])
fmt.Println(data["textfield"])
}
Hope this was helpful.
The json.Unmarshal method will unmarshal to a struct that does not contain all the fields in the original JSON object. In other words, you can cherry-pick your fields. Here is an example where FirstName and LastName are cherry-picked and MiddleName is ignored from the json string:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
}
func main() {
jsonString := []byte("{\"first_name\": \"John\", \"last_name\": \"Doe\", \"middle_name\": \"Anderson\"}")
var person Person
if err := json.Unmarshal(jsonString, &person); err != nil {
panic(err)
}
fmt.Println(person)
}
The other answers here are misleading, as they don't show you what happens if you try to go deeper in the Map. This example works fine enough:
package main
import (
"encoding/json"
"fmt"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
body := map[string]interface{}{}
json.NewDecoder(r.Body).Decode(&body)
/*
[map[
id:com.github.android
platform:play
url:https://play.google.com/store/apps/details?id=com.github.android
]]
*/
fmt.Println(body["related_applications"])
}
but if you try to go one level deeper, it fails:
/*
invalid operation: body["related_applications"][0] (type interface {} does not
support indexing)
*/
fmt.Println(body["related_applications"][0])
Instead, you would need to assert type at each level of depth:
/*
map[
id:com.github.android
platform:play
url:https://play.google.com/store/apps/details?id=com.github.android
]
*/
fmt.Println(body["related_applications"].([]interface{})[0])
You can as well unmarshal it into a map[string]interface{}
body, err := ioutil.ReadAll(resp.Body)
map := &map[string]interface{}{}
json.Unmarshal(body, map)
desiredValue := map["desiredKey"]
The received json must have an object as the most outer element. The map can also contain lists or nested maps, depending on the json.
How to fetch values from JSON in golang
i have following piece of code which calls the yahoo finance api to get the stock values for given stock symbol.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"os"
)
//Response structure
type Response struct {
Query struct {
Count int `json:"count"`
Created string `json:"created"`
Lang string `json:"lang"`
Results struct {
Quote []struct {
LastTradePriceOnly string `json:"LastTradePriceOnly"`
} `json:"quote"`
} `json:"results"`
} `json:"query"`
}
func main() {
var s Response
response, err := http.Get("http://query.yahooapis.com/v1/public/yql?q=select%20LastTradePriceOnly%20from%20yahoo.finance.quote%20where%20symbol%20in%20(%22AAPL%22,%22FB%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys")
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
} else {
defer response.Body.Close()
contents, err := ioutil.ReadAll(response.Body)
json.Unmarshal([]byte(contents), &s)
fmt.Println(s.Query.Results.Quote)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
}
fmt.Println(s.Query.Results.Quote) is giving me a multi value array since Quote is a array of structure. For eg: [{52.05},{114.25}]
How should i split it in a single value in golang ?
For eg: 52.05
114.25
Help is highly appreciated.
Thanks
I am new to golang and not aware of many data structures. But i figured out how to get the single value out of array of structure. fmt.Println(s.Query.Results.Quote[0].LastTradePriceOnly) this worked for me..I only have to iterate this in a loop to fetch all values. Thanks.