How to fetch an attribute value from a variable, having the content of a JSON response - json

I'm using the Robot Framework API automation. Here, storing the JSON response in a variable [POSTResp.content]. I.e., "POSTResp.content" has the whole response, as given below. Please help me to get an attribute's value (for ex, value of referenceId) from the stored content.
Example of JSON response:
{
"serviceResponseHeader": {
"responseContext": {
"responseCode": "MS19",
"responseDescription": "Success",
"serviceResponseTimeInGMT": "18 Sep 2018 16:12:43 GMT"
},
"requesterContext": {
"applicationCode": null,
"applicationSubCode": null,
"countryCode": null,
"requesterReferenceNumber": null,
"requestTimeInGMT": "30 Jun 2015 11:54:49 GMT",
"requesterUserIdentity": "23483",
"requesterGroupIdentity": "1620",
"requesterIpAddress": "",
"sessionIdentity": "2536kjhfdashfkhfsab",
"ssoSessionIdentity": "2536kjhfdashfkhfsab",
"requesterAbbreviatedGroupName": "NEWCOMP"
},
"serviceContext": {
"serviceVersionNumber": "1.0",
"serviceCode": "30"
}
},
"getProxyDetailResponseBody": {
"proxyDetails": {
"proxyType": "",
"proxyValue": "20140005K",
"referenceId": "PR18090000847597",
"transactionId": "18091801657466"
}
}
}
I've tried the below ways,
1) ${json} To JSON ${POSTResp.content} true
log to console \n the Proxy ID is ${json["proxyValue"]}
Result: Resolving variable '${json["proxyValue"]}' failed: TypeError: string indices must be integers, not str
2) ${json} Evaluate json.loads(${POSTResp.content}} json
log to console \n the Proxy ID is ${json["proxyValue"]}
Result: failed: SyntaxError: unexpected EOF while parsing (, line 1)

Issues with your two approaches:
1) the library keyword call passes a true argument (well, truth-like) to the pretty_print parameter:
${json} To JSON ${POSTResp.content} true
Looking at the library's source, in that case the keyword does not return a dict object - but a string, a beatified version of the source json. That coincides with the error your received.
Remove the "true" argument and it must return a dict.
2) In the Evaluate surround the variable with triple quotes (python's literal string):
${json} Evaluate json.loads('''${POSTResp.content}'''}
json
Without it, the framework just dumped the variable's value, which raised a python syntax error.
By the way, try not to make your variables with language keywords/library names - like ${json} up there.

Related

How to get object name when it contains DOTs, CURLY BRACES and HASHTAGs on JSONPath?

I have the following JSON structure, generated by Zabbix Discovery key, with the following data:
[{
"{#SERVICE.NAME}": ".WindowsService1",
"{#SERVICE.DISPLAYNAME}": ".WindowsService1 - Testing",
"{#SERVICE.DESCRIPTION}": "Application Test 1 - Master",
"{#SERVICE.STATE}": 0,
"{#SERVICE.STATENAME}": "running",
"{#SERVICE.PATH}": "E:\\App\\Test\\bin\\testingApp.exe",
"{#SERVICE.USER}": "LocalSystem",
"{#SERVICE.STARTUPTRIGGER}": 0,
"{#SERVICE.STARTUP}": 1,
"{#SERVICE.STARTUPNAME}": "automatic delayed"
},
{
"{#SERVICE.NAME}": ".WindowsService2",
"{#SERVICE.DISPLAYNAME}": ".WindowsService2 - Testing",
"{#SERVICE.DESCRIPTION}": "Application Test 2 - Slave",
"{#SERVICE.STATE}": 0,
"{#SERVICE.STATENAME}": "running",
"{#SERVICE.PATH}": "E:\\App\\Test\\bin\\testingApp.exe",
"{#SERVICE.USER}": "LocalSystem",
"{#SERVICE.STARTUPTRIGGER}": 0,
"{#SERVICE.STARTUP}": 1,
"{#SERVICE.STARTUPNAME}": "automatic delayed"
}]
So, what i want to do is: Use JSONPath to get ONLY the object that {#SERVICE.NAME} == WindowsService1...
The problem is, i am trying to create the JSONPath but it's giving me a couple of errors.
Here's what i tried, and what i discovered so far:
JSONPath:
$.[?(#.{#SERVICE.NAME} == '.WindowsService1')]
Error output:
jsonPath: Unexpected token '{': _$_v.{#SERVICE.NAME} ==
'.WindowsService1'
I also tried doing the following JSONPath, to match Regular Expression:
$.[?(#.{#SERVICE.NAME} =~ '^(.WindowsService1$)')]
It gave me the same error - So the problem is not after the == or =~ ...
What i discovered is, if i REMOVE the curly braces {}, the hashtag # and replace the dot . in "Service name" with _ (Underline), in JSONPath and in JSON data, it works, like this:
Data without # {} . :
[{
"SERVICE_NAME": ".WindowsService1",
[...]
JSONPath following new data structure:
$.[?(#.SERVICE_NAME == '.WindowsService1')]
But the real problem is, i need to maintain the original strucutre, with the curly braces, dots, and hashtags...
How can i escape those and stop seeing this error?
Thank you...
$.[?(#['{#SERVICE.NAME}'] == '.WindowsService1')]

Parse JSON with missing fields using cjson Lua module in Openresty

I am trying to parse a json payload sent via a POST request to a NGINX/Openresty location. To do so, I combined Openresty's content_by_lua_block with its cjson module like this:
# other locations above
location /test {
content_by_lua_block {
ngx.req.read_body()
local data_string = ngx.req.get_body_data()
local cjson = require "cjson.safe"
local json = cjson.decode(data_string)
local endpoint_name = json['endpoint']['name']
local payload = json['payload']
local source_address = json['source_address']
local submit_date = json['submit_date']
ngx.say('Parsed')
}
}
Parsing sample data containing all required fields works as expected. A correct JSON object could look like this:
{
"payload": "the payload here",
"submit_date": "2018-08-17 16:31:51",
},
"endpoint": {
"name": "name of the endpoint here"
},
"source_address": "source address here",
}
However, a user might POST a differently formatted JSON object to the location. Assume a simple JSON document like
{
"username": "JohnDoe",
"password": "password123"
}
not containing the desired fields/keys.
According to the cjson module docs, using cjson (without its safe mode) will raise an error if invalid data is encountered. To prevent any errors being raised, I decided to use its safe mode by importing cjson.safe. This should return nil for invalid data and provide the error message instead of raising the error:
The cjson module will throw an error during JSON conversion if any invalid data is encountered. [...]
The cjson.safe module behaves identically to the cjson module, except when errors are encountered during JSON conversion. On error, the cjson_safe.encode and cjson_safe.decode functions will return nil followed by the error message.
However, I do not encounter any different error handling behavior in my case and the following traceback is shown in Openresty's error.log file:
2021/04/30 20:33:16 [error] 6176#6176: *176 lua entry thread aborted: runtime error: content_by_lua(samplesite:50):16: attempt to index field 'endpoint' (a nil value)
Which in turn results in an Internal Server Error:
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>
I think a workaround might be writing a dedicated function for parsing the JSON data and calling it with pcall() to catch any errors. However, this would make the safe mode kind of useless. What am I missing here?
Your “simple JSON document” is a valid JSON document. The error you are facing is not related to cjson, it's a standard Lua error:
resty -e 'local t = {foo = 1}; print(t["foo"]); print(t["foo"]["bar"])'
1
ERROR: (command line -e):1: attempt to index field 'foo' (a number value)
stack traceback:
...
“Safeness” of cjson.safe is about parsing of malformed documents:
cjson module raises an error:
resty -e 'print(require("cjson").decode("[1, 2, 3"))'
ERROR: (command line -e):1: Expected comma or array end but found T_END at character 9
stack traceback:
...
cjson.safe returns nil and an error message:
resty -e 'print(require("cjson.safe").decode("[1, 2, 3"))'
nilExpected comma or array end but found T_END at character 9

AWS Batch Job container_properties is invalid: Error decoding JSON: invalid character 'v' looking for beginning of value

I'm using terraform to create aws batch job definition:
resource "aws_batch_job_definition" "test" {
name = "jobtest"
type = "container"
container_properties =<<CONTAINER_PROPERTIES
{
"image": var.image,
"memory": 512,
"vcpus": 1,
"jobRoleArn": "${aws_iam_role.job_role.arn}"
}
CONTAINER_PROPERTIES
}
When I run terraform I get this error:
AWS Batch Job container_properties is invalid: Error decoding JSON: invalid character 'v' looking for beginning of value
on admin/prd/batch.tf line 1, in resource "aws_batch_job_definition" "test":
1: resource "aws_batch_job_definition" "test" {
I don't know what's wrong here. I couldn't find any answers in the other StackOverflow questions.

Emit Python embedded object as native JSON in YAML document

I'm importing webservice tests from Excel and serialising them as YAML.
But taking advantage of YAML being a superset of JSON I'd like the request part of the test to be valid JSON, i.e. to have delimeters, quotes and commas.
This will allow us to cut and paste requests between the automated test suite and manual test tools (e.g. Postman.)
So here's how I'd like a test to look (simplified):
- properties:
METHOD: GET
TYPE: ADDRESS
Request URL: /addresses
testCaseId: TC2
request:
{
"unitTypeCode": "",
"unitNumber": "15",
"levelTypeCode": "L",
"roadNumber1": "810",
"roadName": "HAY",
"roadTypeCode": "ST",
"localityName": "PERTH",
"postcode": "6000",
"stateTerritoryCode": "WA"
}
In Python, my request object has a dict attribute called fields which is the part of the object to be serialised as JSON. This is what I tried:
import yaml
def request_presenter(dumper, request):
json_string = json.dumps(request.fields, indent=8)
return dumper.represent_str(json_string)
yaml.add_representer(Request, request_presenter)
test = Test(...including embedded request object)
serialised_test = yaml.dump(test)
I'm getting:
- properties:
METHOD: GET
TYPE: ADDRESS
Request URL: /addresses
testCaseId: TC2
request: "{
\"unitTypeCode\": \"\",\n
\"unitNumber\": \"15\",\n
\"levelTypeCode": \"L\",\n
\"roadNumber1\": \"810\",\n
\"roadName\": \"HAY\",\n
\"roadTypeCode\": \"ST\",\n
\"localityName\": \"PERTH\",\n
\"postcode\": \"6000\",\n
\"stateTerritoryCode\": \"WA\"\n
}"
...only worse because it's all on one line and has white space all over the place.
I tried using the | style for literal multi-line strings which helps with the line breaks and escaped quotes (it's more involved but this answer was helpful.) However, escaped or multiline, the result is still a string that will need to be parsed separately.
How can I stop PyYaml analysing the JSON block as a string and make it just accept a block of text as part of the emitted YAML? I'm guessing it's something to do with overriding the emitter but I could use some help. If possible I'd like to avoid post-processing the serialised test to achieve this.
Ok, so this was the solution I came up with. Generate the YAML with a placemarker ahead of time. The placemarker marks the place where the JSON should be inserted, and also defines the root-level indentation of the JSON block.
import os
import itertools
import json
def insert_json_in_yaml(pre_insert_yaml, key, obj_to_serialise):
marker = '%s: null' % key
marker_line = line_of_first_occurrence(pre_insert_yaml, marker)
marker_indent = string_indent(marker_line)
serialised = json.dumps(obj_to_serialise, indent=marker_indent + 4)
key_with_json = '%s: %s' % (key, serialised)
serialised_with_json = pre_insert_yaml.replace(marker, key_with_json)
return serialised_with_json
def line_of_first_occurrence(basestring, substring):
"""
return line number of first occurrence of substring
"""
lineno = lineno_of_first_occurrence(basestring, substring)
return basestring.split(os.linesep)[lineno]
def string_indent(s):
"""
return indentation of a string (no of spaces before a nonspace)
"""
spaces = ''.join(itertools.takewhile(lambda c: c == ' ', s))
return len(spaces)
def lineno_of_first_occurrence(basestring, substring):
"""
return line number of first occurrence of substring
"""
return basestring[:basestring.index(substring)].count(os.linesep)
embedded_object = {
"unitTypeCode": "",
"unitNumber": "15",
"levelTypeCode": "L",
"roadNumber1": "810",
"roadName": "HAY",
"roadTypeCode": "ST",
"localityName": "PERTH",
"postcode": "6000",
"stateTerritoryCode": "WA"
}
yaml_string = """
---
- properties:
METHOD: GET
TYPE: ADDRESS
Request URL: /addresses
testCaseId: TC2
request: null
after_request: another value
"""
>>> print(insert_json_in_yaml(yaml_string, 'request', embedded_object))
- properties:
METHOD: GET
TYPE: ADDRESS
Request URL: /addresses
testCaseId: TC2
request: {
"unitTypeCode": "",
"unitNumber": "15",
"levelTypeCode": "L",
"roadNumber1": "810",
"roadName": "HAY",
"roadTypeCode": "ST",
"localityName": "PERTH",
"postcode": "6000",
"stateTerritoryCode": "WA"
}
after_request: another value

How to evaluate JSON Path with fields that contain quotes inside a value?

I have a NiFi flow that takes JSON files and evaluates a JSON Path argument against them. It work perfectly except when dealing with records that contain Korean text. The Jayway JSONPath evaluator does not seem to recognize the escape "\" in the headline field before the double quote character. Here is an example (newlines added to help with formatting):
{"data": {"body": "[이데일리 김관용 기자] 우리 군이 2018 남북정상회담을 앞두고 남
북간 군사적 긴장\r\n완화와 평화로운 회담 분위기 조성을 위해 23일 0시를 기해 군사분계선
(MDL)\r\n일대에서의 대북확성기 방송을 중단했다.\r\n\r\n국방부는 이날 남북정상회담 계기
대북확성기 방송 중단 관련 내용을 발표하며\r\n“이번 조치가 남북간 상호 비방과 선전활동을
중단하고 ‘평화, 새로운 시작’을\r\n만들어 나가는 성과로 이어지기를 기대한다”고 밝혔
다.\r\n\r\n전방부대 우리 군 장병이 대북확성기 방송을 위한 장비를 점검하고 있다.\r\n[사
진=국방부공동취재단]\r\n\r\n\r\n\r\n▶ 당신의 생활 속 언제 어디서나 이데일리 \r\n▶
스마트 경제종합방송 ‘이데일리 TV’ | 모바일 투자정보 ‘투자플러스’\r\n▶ 실시간 뉴스와
속보 ‘모바일 뉴스 앱’ | 모바일 주식 매매 ‘MP트래블러Ⅱ’\r\n▶ 전문가를 위한 국내 최상의
금융정보단말기 ‘이데일리 마켓포인트 3.0’ | ‘이데일리 본드웹 2.0’\r\n▶ 증권전문가방송
‘이데일리 ON’ 1666-2200 | ‘ON스탁론’ 1599-2203\n<ⓒ종합 경제정보 미디어 이데일리 -
무단전재 & 재배포 금지> \r\n",
"mimeType": "text/plain",
"language": "ko",
"headline": "국방부 \"軍 대북확성기 방송, 23일 0시부터 중단\"",
"id": "EDYM00251_1804232beO/5WAUgdlYbHS853hYOGrIL+Tj7oUjwSYwT"}}
If this object is in my file, the JSON path evaluates blanks for all the path arguments. Is there a way to force the Jayway engine to recognize the "\"? It appears to function correctly in other languages.
I was able to resolve this after understanding the difference between definite and indefinite paths. The Jayway github README points out the following will make a path indefinite and return a list:
When evaluating a path you need to understand the concept of when a
path is definite. A path is indefinite if it contains:
.. - a deep scan operator
?(<expression>) - an expression
[<number>, <number> (, <number>)] - multiple array indexes Indefinite paths
always returns a list (as represented by current JsonProvider).
My JSON looked like the following:
{
"Version":"14",
"Items":[
{"data": {"body": "[이데일리 ... \r\n",
"mimeType": "text/plain",
"language": "ko",
"headline": "국방부 \"軍 ... 중단\"",
"id": "1"}
},
{"data": {"body": "[이데일리 ... \r\n",
"mimeType": "text/plain",
"language": "ko",
"headline": "국방부 \"軍 ... 중단\"",
"id": "2"}
...
}
]
}
This JSON path selector I was using ($.data.headline) did not grab the values as I expected. It instead returned null values.
Changing it to $.Items[*].data.headline or $..data.headline returns a list of each headline.