When I try to use "Get Value From Json" on a Json with specific JsonPath , I always have KeyError
even web simple example doesn't work for me...
When I try that code :
Library JSONLibrary
*** Test Cases ***
Example
${tmp} = Convert String To JSON {"foo": {"bar": [1,2,3]}}
Log to console tmp: ${tmp}
${values_test}= Get Value From Json ${tmp} $.foo.bar
Log to console values_test: ${values_test}
I always have this kind of Errors and log :
tmp: {'foo': {'bar': [1, 2, 3]}}
...
Resolving variable '${tmp['$.foo.bar']}' failed: KeyError: '$.foo.bar'
Can somebody Help me please ?
it is really basic example by the way and community always says that it works like that in comments..
I have a json log that expands to this:
JSON:
|-host : hostname
|-httpRequest
|-httpVersion : HTTP/1.1
|-headers
|-0
|-name: X-Forwarded-For
|-value: 1.1.1.1
|-1
|-name: X-Forwarded-Prot
|-value: https
|-2
|-name: X-Forwarded-Port
|-value: 443
|-httpMethod: post
|-action: allow
etc..
I would like to reformat it like this:
JSON:
|-host : hostname
|-httpRequest
|-httpVersion : HTTP/1.1
|-headers
|-X-Forwarded-For : 1.1.1.1
|-X-Forwarded-Prot : https
|-X-Forwarded-Port : 443
|-httpMethod: post
|-action: allow
Split will just take the last [#] name/value as it overwrites the previous.
I am pretty sure this will need a ruby code block, but I haven't had luck following along with ruby code I have found online for similar scenarios.
I think the main issue here/difference with other article/answers is that it's not just a pure flatten. But need to rearrange the name:value a bit as well.
This seems to do the trick as well:
Newfield
ruby {
code => '
event.get("[#metadata][json][httpRequest][headers]").each do |header|
event.set("[newfield][#{header["name"]}]", header["value"])
end
'
}
Same field
ruby {
code => '
headerHash = {}
event.get("[#metadata][json][httpRequest][headers]").each do |header|
headerHash[header["name"]] = header["value"]
end
event.set("[#metadata][json][httpRequest][headers]", headerHash)
'
}
I am trying to parse a json payload sent via a POST request to a NGINX/Openresty location. To do so, I combined Openresty's content_by_lua_block with its cjson module like this:
# other locations above
location /test {
content_by_lua_block {
ngx.req.read_body()
local data_string = ngx.req.get_body_data()
local cjson = require "cjson.safe"
local json = cjson.decode(data_string)
local endpoint_name = json['endpoint']['name']
local payload = json['payload']
local source_address = json['source_address']
local submit_date = json['submit_date']
ngx.say('Parsed')
}
}
Parsing sample data containing all required fields works as expected. A correct JSON object could look like this:
{
"payload": "the payload here",
"submit_date": "2018-08-17 16:31:51",
},
"endpoint": {
"name": "name of the endpoint here"
},
"source_address": "source address here",
}
However, a user might POST a differently formatted JSON object to the location. Assume a simple JSON document like
{
"username": "JohnDoe",
"password": "password123"
}
not containing the desired fields/keys.
According to the cjson module docs, using cjson (without its safe mode) will raise an error if invalid data is encountered. To prevent any errors being raised, I decided to use its safe mode by importing cjson.safe. This should return nil for invalid data and provide the error message instead of raising the error:
The cjson module will throw an error during JSON conversion if any invalid data is encountered. [...]
The cjson.safe module behaves identically to the cjson module, except when errors are encountered during JSON conversion. On error, the cjson_safe.encode and cjson_safe.decode functions will return nil followed by the error message.
However, I do not encounter any different error handling behavior in my case and the following traceback is shown in Openresty's error.log file:
2021/04/30 20:33:16 [error] 6176#6176: *176 lua entry thread aborted: runtime error: content_by_lua(samplesite:50):16: attempt to index field 'endpoint' (a nil value)
Which in turn results in an Internal Server Error:
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
I think a workaround might be writing a dedicated function for parsing the JSON data and calling it with pcall() to catch any errors. However, this would make the safe mode kind of useless. What am I missing here?
Your “simple JSON document” is a valid JSON document. The error you are facing is not related to cjson, it's a standard Lua error:
resty -e 'local t = {foo = 1}; print(t["foo"]); print(t["foo"]["bar"])'
1
ERROR: (command line -e):1: attempt to index field 'foo' (a number value)
stack traceback:
...
“Safeness” of cjson.safe is about parsing of malformed documents:
cjson module raises an error:
resty -e 'print(require("cjson").decode("[1, 2, 3"))'
ERROR: (command line -e):1: Expected comma or array end but found T_END at character 9
stack traceback:
...
cjson.safe returns nil and an error message:
resty -e 'print(require("cjson.safe").decode("[1, 2, 3"))'
nilExpected comma or array end but found T_END at character 9
I'm just trying to use Postgrex without any kind of ecto setup, so just the example from the documentation readme.
Here is what my module looks like:
defmodule Receive do
def start(_type, _args) do
{:ok, pid} = Postgrex.start_link(
hostname: "localhost",
username: "john",
# password: "",
database: "property_actions",
extensions: [{Postgrex.Extensions.JSON}]
)
Postgrex.query!(
pid,
"INSERT INTO actions (search_terms) VALUES ($1)",
[
%{foo: 'bar'}
]
)
end
end
when I run the code I get
** (RuntimeError) type `json` can not be handled by the types module Postgrex.DefaultTypes, it must define a `:json` library in its options to support JSON types
Is there something I'm not setting up correctly? From what I've gathered in the documentation, I shouldn't even need to have that extensions line because json is handled by default.
On Postgrex <= 0.13, you need to define your own types:
Postgrex.Types.define(MyApp.PostgrexTypes, [], json: Poison)
and then when starting Postgrex:
Postgrex.start_link(types: MyApp.PostgrexTypes)
On Postgrex >= 0.14 (currently master), it was made easier:
config :postgrex, :json_library, Poison
I'm trying to configure the Shovel plugin for RabbitMQ with a list of declarations. I have configured my remote exchange to have an alternate-exchange when I created it.
My problem is that I can't get the config file for shovel to include this argument so RabbitMQ crashes upon startup.
This is what my config looks like:
[
{mnesia, [{dump_log_write_threshold, 100}]},
{rabbit, [{vm_memory_high_watermark, 0.4}]},
{rabbitmq_shovel,
[{shovels,
[{call_stats_shovel,
[{sources, [{broker, "amqp://guest:guest#localhost:5672/test"},
{declarations,
[{'queue.declare', [{queue, <<"incoming">>}, durable]},
{'exchange.declare',[{exchange, <<"my-exchange-topic">>},{type, <<"topic">>},durable]},
{'queue.bind',[{exchange, <<"my-exchange-topic">>},{queue, <<"incoming">>}]}
]}]},
{destinations, [{broker, "amqp://guest:guest#172.16.3.162:5672/blah"},
{declarations,
[
{'queue.declare',[{queue, <<"billing">>},durable]},
{'exchange.declare',[{exchange, <<"my-exchange-topic">>},{type, <<"topic">>},{alternate_exchange, <<"alt">>}, durable]},
{'queue.bind',[{exchange, <<"my-exchange-topic">>},{queue, <<"billing">>},{routing_key, <<"physical">>}]}
]}
]},
{queue, <<"incoming">>},
{ack_mode, no_ack},
{publish_properties, [{delivery_mode, 2}]},
{reconnect_delay, 5}
]}
]
}]
}
].
The problem is on the destination exchange called my-exchange-topic. If I take out the declarations section then the config file works.
This is the error:
=INFO REPORT==== 31-Jul-2012::12:15:25 ===
application: rabbitmq_shovel
exited: {{invalid_shovel_configuration,call_stats_shovel,
{invalid_parameter_value,destinations,
{unknown_fields,'exchange.declare',
[alternate_exchange]}}},
{rabbit_shovel,start,[normal,[]]}}
type: permanent
If I leave the alternate_exchange section out of the declaration I get this error in RabbitMQ web management:
{{shutdown,
{server_initiated_close,406,
<<"PRECONDITION_FAILED - inequivalent arg 'alternate-exchange'for exchange 'my-exchange-topic' in vhost 'blah':
received none but current is the value 'alt' of type 'longstr'">>}},
{gen_server,call,
[<0.473.0>,
{call,
{'exchange.declare',0,<<"my-exchange-topic">>,<<"topic">>,false,
true,false,false,false,[]},
none,<0.444.0>},
infinity]}}
For anyone looking at how to configure exchanges and queues that require additional arguments you do it like this:
{'exchange.declare',[{exchange, <<"my-exchange-topic">>},{type, <<"topic">>}, durable, {arguments, [{<<"alternate-exchange">>, longstr, <<"alternate-exchange">>}]} ]},
you can do a similar thing with queues:
{'queue.declare',[{queue, <<"my-queue">>},durable, {arguments, [{<<"x-dead-letter-exchange">>, longstr, <<"dead-letter-queue">>}]}]}
For clarification of the comment above, in case of an exchange2exchange shovel, the config would be:
{'exchange.declare',[{exchange, <<"my-exchange-topic">>},{type, <<"topic">>}, durable, {arguments, [{<<"alternate-exchange">>, longstr, <<"name-of-your-alternate-exchange">>}]} ]},