elixir poison not able to decode JSON - json

I am trying below code
data = Poison.decode!(payload)
|> ProperCase.to_snake_case
Logger.info("data is #{data}")
Where the payload is from Messging Queue
{
"name":"Joe",
"id": "13",
"version": 0
}
With this I am getting an error
[error] Could not process PerfectFlight: %Protocol.UndefinedError{description: "", protocol: String.Chars, value: %{"id" => "13", "name" => "Joe", "version" => 0}}
However, If I change my input json to
"{
\"name\":\"Joe\",
\"id\": \"13\",
\"version\": 0
}"
The Poison.decode() works pefectly fine. Now the issue is that I don't want to change my input JSON because of many reasons. What am I missing?
Edit : The code was failing not at the decode but on the next line Logger.info("data is #{data}"). Since the output of decode function is not a String I should rather use IO.inspect as below. Accepting Adams answer for his confidence in decode function.
data = Poison.decode!(payload)
|> ProperCase.to_snake_case
IO.inspect(data)

There is nothing wrong with Poison:
iex(1)> Poison.decode!(~S'{"name":"Joe", "id": "13", "version": 0}')
%{"id" => "13", "name" => "Joe", "version" => 0}
Your example also works for me:
iex(1)> payload = ~S'{"name":"Joe", "id": "13", "version": 0}'
"{\"name\":\"Joe\", \"id\": \"13\", \"version\": 0}"
iex(2)> Poison.decode!(payload) |> ProperCase.to_snake_case
%{"id" => "13", "name" => "Joe", "version" => 0}
The error you are getting is probably because something somewhere is trying to convert a map to a string:
iex(1)> IO.puts %{"id" => "13", "name" => "Joe", "version" => 0}
** (Protocol.UndefinedError) protocol String.Chars not implemented for %{"id" => "13", "name" => "Joe", "version" => 0}
It looks like you're getting an error while trying to debug your problem. You could use IO.inspect instead of IO.puts for debugging, and see if you can get a more helpful error message.

Related

API POST request in Julia

I am trying to convert some Python code to Julia. Here is the Python code:
url = "http://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101G/BefUtvKon1749"
json = {
"query": [
{
"code": "Kon",
"selection": {
"filter": "item",
"values": [
"1",
"2"
]
}
},
{
"code": "ContentsCode",
"selection": {
"filter": "item",
"values": [
"000000LV"
]
}
}
],
"response": {
"format": "px"
}
}
r = requests.post(url=url, json=json)
Below is the Julia code, that is not working, with this error message:
syntax: { } vector syntax is discontinued around path:8
top-level scope at population_data.jl:8
using DataFrames, DataFramesMeta, HTTP, JSON3
url = "http://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101G/BefUtvKon1749"
json = {
"query": [
{
"code": "Kon",
"selection": {
"filter": "item",
"values": [
"1",
"2",
"1+2"
]
}
},
{
"code": "ContentsCode",
"selection": {
"filter": "item",
"values": [
"000000LV"
]
}
}
],
"response": {
"format": "px"
}
}
r = HTTP.post(url, json)
My attempts to solve this are the following:
Convert the json variable to a string using """ around it.
Converting the JSON string to Julia data types, using JSON3.read()
Passing the converted JSON string to the POST request. This gives the following error:
IOError(Base.IOError("read: connection reset by peer (ECONNRESET)", -54) during request(http://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101G/BefUtvKon1749)
None of it works, and I am not even sure that it is about the JSON format. It could be that I am passing the wrong parameters to the POST request. What should I do?
One way of solving this consists in building the parameters as native julia data structures, and use JSON to convert and use them as the body of your PUT request:
Dictionaries in julia are built using a syntax like Dict(key => value). Arrays are built using a standard syntax: [a, b, c]. The julia native data structure equivalent to your parameters would look like this:
params = Dict(
"query" => [
Dict("code" => "Kon",
"selection" => Dict(
"filter" => "item",
"values" => [
"1",
"2",
"1+2"
]),
),
Dict("code"=> "ContentsCode",
"selection" => Dict(
"filter" => "item",
"values" => [
"000000LV"
]),
),
],
"response" => Dict(
"format" => "px"
))
Then, you can use JSON.json() to build the JSON representation of it as a string and pass it to the HTTP request:
using HTTP
using JSON
url = "http://api.scb.se/OV0104/v1/doris/sv/ssd/START/BE/BE0101/BE0101G/BefUtvKon1749"
# send the request
r = HTTP.request("POST", url,
["Content-Type" => "application/json"],
JSON.json(params))
# retrieve the response body as a string
b = String(r.body)

How to insert an empty object into JSON using Circe?

I'm getting a JSON object over the network, as a String. I'm then using Circe to parse it. I want to add a handful of fields to it, and then pass it on downstream.
Almost all of that works.
The problem is that my "adding" is really "overwriting". That's actually ok, as long as I add an empty object first. How can I add such an empty object?
So looking at the code below, I am overwriting "sometimes_empty:{}" and it works. But because sometimes_empty is not always empty, it results in some data loss. I'd like to add a field like: "custom:{}" and then ovewrite the value of custom with my existing code.
Two StackOverflow posts were helpful. One worked, but wasn't quite what I was looking for. The other I couldn't get to work.
1: Modifying a JSON array in Scala with circe
2: Adding field to a json using Circe
val js: String = """
{
"id": "19",
"type": "Party",
"field": {
"id": 1482,
"name": "Anne Party",
"url": "https"
},
"sometimes_empty": {
},
"bool": true,
"timestamp": "2018-12-18T11:39:18Z"
}
"""
val newJson = parse(js).toOption
.flatMap { doc =>
doc.hcursor
.downField("sometimes_empty")
.withFocus(_ =>
Json.fromFields(
Seq(
("myUrl", Json.fromString(myUrl)),
("valueZ", Json.fromString(valueZ)),
("valueQ", Json.fromString(valueQ)),
("balloons", Json.fromString(balloons))
)
)
)
.top
}
newJson match {
case Some(v) => return v.toString
case None => println("Failure!")
}
We need to do a couple of things. First, we need to zoom in on the specific property we want to update, if it doesn't exist, we'll create a new empty one. Then, we turn the zoomed in property in the form of a Json into JsonObject in order to be able to modify it using the +: method. Once we've done that, we need to take the updated property and re-introduce it in the original parsed JSON to get the complete result:
import io.circe.{Json, JsonObject, parser}
import io.circe.syntax._
object JsonTest {
def main(args: Array[String]): Unit = {
val js: String =
"""
|{
| "id": "19",
| "type": "Party",
| "field": {
| "id": 1482,
| "name": "Anne Party",
| "url": "https"
| },
| "bool": true,
| "timestamp": "2018-12-18T11:39:18Z"
|}
""".stripMargin
val maybeAppendedJson =
for {
json <- parser.parse(js).toOption
sometimesEmpty <- json.hcursor
.downField("sometimes_empty")
.focus
.orElse(Option(Json.fromJsonObject(JsonObject.empty)))
jsonObject <- json.asObject
emptyFieldJson <- sometimesEmpty.asObject
appendedField = emptyFieldJson.+:("added", Json.fromBoolean(true))
res = jsonObject.+:("sometimes_empty", appendedField.asJson)
} yield res
maybeAppendedJson.foreach(obj => println(obj.asJson.spaces2))
}
}
Yields:
{
"id" : "19",
"type" : "Party",
"field" : {
"id" : 1482,
"name" : "Anne Party",
"url" : "https"
},
"sometimes_empty" : {
"added" : true,
"someProperty" : true
},
"bool" : true,
"timestamp" : "2018-12-18T11:39:18Z"
}

Laravel Array & JSON Casting to Algolia

I am trying to send some data along to Algolia through the toSearchableArray. Any strings I have stored in my DB are sending along fine, but I hit a roadblock when trying to push nested JSON data along—the information is being sent as a string with characters escaped.
This is a sample of the nested object that I am storing in my table (MySQL with a JSON data type):
[
{
"id": 19,
"name": "Mathematics",
"short": "Math"
},
{
"id": 23,
"name": "Science",
"short": "Science"
},
{
"id": 14,
"name": "Health and Life Skills",
"short": "Health"
}
]
My model looks like this:
<?php
namespace App;
use Illuminate\Database\Eloquent\Model;
use Laravel\Scout\Searchable;
class Resource extends Model
{
use Searchable;
protected $primaryKey = 'objectID';
public function toSearchableArray()
{
$data = $this->toArray();
$data['grades'] = explode(';', $data['grades']);
$data['units'] = explode(';', $data['units']);
return $data;
}
}
I get an output that looks like this:
array:22 [
"objectID" => 1
"name" => "Resource #1"
"slug" => "resource-1"
"write_up" => """
This is an example write up.
"""
"author" => "johnny"
"type_name" => "Lesson Plan"
"language" => "English"
"grades" => array:3 [
0 => "Kindergarten"
1 => "Grade 1"
2 => "Grade 4"
]
"subjects" => "[{"id": 19, "name": "Mathematics", "short": "Math"}, {"id": 23, "name": "Science", "short": "Science"}, {"id": 14, "name": "Health and Life Skills", "short": "Health"}]"
"units" => array:2 [
0 => "Unit A"
1 => "Unit B"
]
"main_image" => "https://dummyimage.com/250x325/000000/fff.png&text=Just+a+Test"
"loves" => 88
"downloads" => 280
"created_at" => "2018-01-01 13:26:47"
"updated_at" => "2018-01-02 10:10:32"
]
As you can see, the 'subjects' attribute is being stored as a string. I know there is attribute casting in 5.5 (I am running 5.5), but I am not too clear on how I would implement the example they have for Array & JSON Casting in my work above. https://laravel.com/docs/5.5/eloquent-mutators#attribute-casting
Would anyone be willing to show me an example?
I'd rely on Attribute Casting for this, add a $casts property in your model and it will be done automatically.
<?php
namespace App;
use Illuminate\Database\Eloquent\Model;
use Laravel\Scout\Searchable;
class Resource extends Model
{
use Searchable;
protected $primaryKey = 'objectID';
protected $casts = [
'subjects' => 'array',
];
public function toSearchableArray()
{
// Same function as you posted
}
}
You can also do it manually in your toSearchableArray method with $data['subjects'] = json_decode($this->subjects, true);
I answered with more details on this other posts: https://discourse.algolia.com/t/laravel-array-json-casting-to-algolia/4125/2

ELM parse nested json

I have a json array with multiple comments which can be nested.
exemple:
[
{
"author": "john",
"comment" : ".....",
"reply": "",
},
{
"author": "Paul",
"comment" : ".....",
"reply": [
{
"author": "john",
"comment" : "nested comment",
"reply": [
{
"author": "Paul",
"comment": "second nested comment"
}
]
},
{
"author": "john",
"comment" : "another nested comment",
"reply": ""
}
]
},
{
"author": "Dave",
"comment" : ".....",
"reply": ""
},
]
So it's a list of comment, which every comment can have a reply with an infinite number of reply.
With Json.Decode.list I can decode the first level of comment, but how do I checked if there is some reply and then parse again ?
This is a simplify version of what I'm try to do. I'm actually trying to decode reddit comments. exemple
Elm won't let you create a recursive record type alias, so you'll have to use a union type for Customer. You may also want a convenience function for creating a user so you can use Json.map3.
Your example json has an oddity: Sometimes reply is an empty string and sometimes it's a list. You'll need a special decoder to turn that string into an empty list (assuming an empty list is synonymous with an empty list in this context).
Since you have a recursive type, you need to use lazy for decoding the child comments to avoid a runtime error.
import Html exposing (Html, text)
import Json.Decode as Json exposing (..)
main : Html msg
main =
text <| toString <| decodeString (list commentDecoder) s
type Comment
= Comment
{ author : String
, comment : String
, reply : List Comment
}
newComment : String -> String -> List Comment -> Comment
newComment author comment reply =
Comment
{ author = author
, comment = comment
, reply = reply
}
emptyStringToListDecoder : Decoder (List a)
emptyStringToListDecoder =
string
|> andThen
(\s ->
case s of
"" ->
succeed []
_ ->
fail "Expected an empty string"
)
commentDecoder : Decoder Comment
commentDecoder =
map3 newComment
(field "author" string)
(field "comment" string)
(field "reply" <|
oneOf
[ emptyStringToListDecoder
, list (lazy (\_ -> commentDecoder))
]
)
s =
"""
[{
"author": "john",
"comment": ".....",
"reply": ""
}, {
"author": "Dave",
"comment": ".....",
"reply": ""
}, {
"author": "Paul",
"comment": ".....",
"reply": [{
"author": "john",
"comment": "nested comment",
"reply": [{
"author": "Paul",
"comment": "second nested comment",
"reply": ""
}]
}, {
"author": "john",
"comment": "another nested comment",
"reply": ""
}]
}]
"""
(Your json is also a little off in other ways: There are a few extra commas after the last parts of list and one of the reply fields is missing)

Parse JSON message in Logstash

I am sending my jenkins logs to logstash with following config:
redis {
host => "localhost"
key => "logstash"
data_type => "list"
codec => json
}
This works as smooth as expected, now i see follwoing message in KIBANA:
{
"_index": "logstash-2015.12.18",
"_type": "logs",
"_id": "AVG1BN5LXZBIbp7HE4xN",
"_score": null,
"_source": {
"data": {
"id": "965",
"projectName": "NicePJ",
"displayName": "#965",
"fullDisplayName": "NicePJ",
"url": "job/NIcePJ/965/",
"buildHost": "Jenkins",
"buildLabel": "master",
"buildNum": 965,
"buildDuration": 1,
"rootProjectName": "NicePJ",
"rootProjectDisplayName": "#965",
"rootBuildNum": 965,
"buildVariables": {
"target_SUT": "0201",
"report_warnings": "false",
"product": "Ours",
"testsuite": "Exciting_stuff5",
"qft_version": "current",
"target_task": "t324",
"branch": "test",
"testcase": "",
"revision": "HEAD",
"node": "hsqs960",
"client": "Desktop",
"run_specific_test": "false",
"user": "xxxxx"
}
},
"message": [
"A This is a message XYZ"
],
"source": "jenkins",
"source_host": "http://serverXL:8080/",
"#timestamp": "2015-12-18T12:16:02.000Z",
"#version": 1
},
"fields": {
"#timestamp": [
1450440962000
]
},
"sort": [
1450440962000
]
}
Now i want to filter the message field for certain messages, but i cant get it work. How can i filter the message field and how can i access the buildHost field to use it in an if statement in the pipeline?
Following i tried after many examples:
if[data][buildHost]== "jenkins"
{
grok
{
match => { "message[0]" => "\[exec\]\s*\<%{GREEDYDATA:test}\s*\[%{GREEDYDATA:result}\]" }
}
}
But this is just not working at all, please help me out.
Conditional
The == compares simple string and case sensitive, so "jenkins" will not match as your data shows ("buildHost": "Jenkins",):
if[data][buildHost]== "jenkins"
But following does:
if[data][buildHost]== "Jenkins"
If you need match both, you can either use || or regex =~.
Grok
The grok is a filter to parse message with regex pattern. You can test your regex pattern with
online grok debugger
Kibana dev tools's grok debugger