Json handling in ROBOT - json

I have a Json file in which there is a field which I need to edit and save the file for next usage.
But the field which I need to edit is as shown below,
The value I need to assign fr the field is generated Randomly in run time which i'll be capturing in a variable and pass it to this json specific key "dp" then save the json.
The saved json will be used for REST POST url.
{
"p": "10",
"v": 100,
"vt": [
{
"dp": "Field to be edited"(integer value) ,
]
}

The simplest solution would be to write a python keyword that can change the value for you. However, you can solve this with robot keywords by performing the following steps:
convert the JSON string to a dictionary
modify the dictionary
convert the dictionary back to a JSON string
Convert the JSON string to a dictionary
Python has a module (json) for working with JSON data. You can use the evaluate keyword to convert your JSON string to a python dictionary using the loads (load string) method of that module.
Assuming your JSON data is in a robot variable named ${json_string}, you can convert it to a python dictionary like this:
${json}= evaluate json.loads('''${json_string}''') json
With the above, ${json} now holds a reference to a dictionary that contains all of the json data.
Modify the dictionary
The Collections library that comes with robot has a keyword named set to dictionary which can be used to set the value of a dictionary element. In this case, you need to change the value of a dictionary nested inside the vt element of the JSON object. We can reference that nested dictionary using robot's extended variable syntax.
For example:
set to dictionary ${json["vt"]} dp=the new value
With that, ${json} now has the new value. However, it is still a python dictionary rather than JSON data, so there's one more step.
Convert the dictionary back to JSON
Converting the dictionary back to JSON is the reverse of the first step. Namely, use the dumps (dump string) method of the json module:
${json_string}= evaluate json.dumps(${json}) json
With that, ${json_string} will contain a valid JSON string with the modified data.
Complete example
The following is a complete working example. The JSON string will be printed before and after the substitution of the new value:
*** Settings ***
Library Collections
*** Test Cases ***
Example
${json_string}= catenate
... {
... "p": "10",
... "v": 100,
... "vt": {
... "dp": "Field to be edited"
... }
... }
log to console \nOriginal JSON:\n${json_string}
${json}= evaluate json.loads('''${json_string}''') json
set to dictionary ${json["vt"]} dp=the new value
${json_string}= evaluate json.dumps(${json}) json
log to console \nNew JSON string:\n${json_string}

For reading and writing data to and from file I am using OperatingSystem library
${json} Get Binary File ${json_path}nameOfJsonFile.json
It works for me on API testing, to read .json and POST, like here
*** Settings ***
Library Collections
Library ExtendedRequestsLibrary
Library OperatingSystem
*** Variables ***
${uri} https://blabla.com/service/
${json_path} C:/home/user/project/src/json/
*** Test Cases ***
Robot Test Case
Create Session alias ${uri}
&{headers} Create Dictionary Content-Type=application/json; charset=utf-8
${json} Get Binary File ${json_path}nameOfJsonFile.json
${resp} Post Request alias data=${json} headers=${headers}
Should Be Equal As Strings ${resp.status_code} 200

For integer values in JSON, the other answers did not work for me.
This worked:
${json}= Catenate { "p": "10", "v": 100, "vt": { "dp": "Field to be edited" } }
${value} Set Variable 2 #the value you want
${value} Convert To Integer ${value}
${json}= Evaluate json.loads('''${json}''') json
Set To Dictionary ${json["vt"]} dp=${value}
${json}= Evaluate json.dumps(${json}) json
Log ${json}
Convert To Integer was required, otherwise the value is still in string "${value}"

Related

Elasticsearch 8.0.0 mapper_parsing_exception of a string literal for field type "flattened"

I have a Problem to insert a document via Api to my ES 8.0.0.
In my IndexTemplate I defined a mapping of a property called [Data] of type "flattend".
For "normal" JSON-Objects it works fine. But when I try to insert a plain string literal (for example "test") or a number (for example 4) I get a "400 Bad Request". JSONLint says it's a valid JSON!!
{
....
"Data": "test",
....
}
Can i configure ES to accept such kind of JSON for type "flattened"??
As Elasticsearch document mentions:
The flattened type provides an alternative approach, where the entire
object is mapped as a single field. Given an object, the flattened
mapping will parse out its leaf values and index them into one field
as keywords.
So, the value provided for the "flattened" field type should be a JsonObject.
Hence, below works as where "full_name" is of type "flattened"
"full_name":{
"name":"nishikant"
}
But below does not
"full_name":"nishikant".
Same has been given in exception
"reason" : "Failed to parse object: expecting token of type [START_OBJECT] but found [VALUE_STRING]"

Why is initial JSON object parseable, but object within it not?

I'm storing a config file in version control (GitLab) which contains information to be read by my ruby app. This info is stored as an object containing objects containing objects.
(Update adding more detail and examples for clarity as requested...)
From within my app I can successfully GET the file (which returns the following JSON Object (some bits trimmed with ... for readability):
{"file_name"=>"approval_config.json", "file_path"=>"approval_config.json", "size"=>1331, "encoding"=>"base64", "content_sha256"=>"1c21cbb...fa453fe", "ref"=>"master", "blob_id"=>"de...915", "commit_id"=>"07e...4ff", "last_commit_id"=>"07e...942f", "content"=>"ogICAg...AgICB"}
I can JSON parse the above object and access the contents property on that object. The value of the contents property is a base64Encoded string containing the actual contents of my file in GitLab. I can successfully decode this and see the JSON string stored in GitLab:
"{"G000":{"1":{"max":"4000","name":"Matthew Lewis","id":"ord-matthewl","email":"matthew.lewis#companyx.com"},"2":{"max":"4000","name":"Brendan Jones","id":"ord-brendanj","email":"brendan.jones#companyx.com"},"3":{"max":"20000","name":"Henry Orson","id":"ord-henryo","email":"henry.orson#companyx.com"},"4":{"max":"10000000","name":"Chris Adams","id":"ord-chrisa","email":"chris.adams#companyx.com"}},"G15":{"1":{"max":"4000","name":"Mike Butak","id":"ord-mikebu","email":"mike.butak#companyx.com"},"2":{"max":"4000","name":"Joseph Lister","id":"ord-josephl","email":"joseph.lister#companyx.com"},"3":{"max":"20000","name":"Mike Geisler","id":"ord-mikeg","email":"mike.geisler#companyx.com"},"4":{"max":"10000000","name":"Samuel Ahn","id":"ord-samuela","email":"samuel.ahn#companyx.com"}}}"
THIS string (above), I cannot JSON parse. I get an "unexpected token at '{ (JSON::ParserError)" error.
While writing this update it occurs to me that this "un-parsable" string is simply what I put in the file to begin with. Perhaps the method I used to stringify the file's contents in the first place is the issue. I simply pasted a valid javascript object in my browser's console, JSON.stringify'd it, copied the result from the console, and pasted it in my file in GitLab. Perhaps I need to use Ruby's JSON.stringify method to stringify it?
Based on feedback from #ToddA.Jacobs, I tried the following in my ruby script:
require 'rest-client'
require 'json'
require 'base64'
data = RestClient.get 'https://gitlab.companyx.net/api/v4/projects/3895/repository/files/approval_config.json?ref=master', {'PRIVATE-TOKEN':'*********'}
# get the encoded data stored on the 'content' key:
content = JSON.parse(data)['content']
# decode it:
config = Base64.decode64(content)
# print some logs
$evm.log(:info, config)
$evm.log(:info, "config is a Hash? :" + config.is_a?(Hash).to_s) #prints false
$evm.log(:info, "config is a string? :" + config.is_a?(String).to_s) #prints true
hash = JSON.parse(config)
example = hash.dig "G000" "4" "id"
$evm.log(:info, "print exmaple on next line")
$evm.log(:info, example)
That last line prints:
The following error occurred during method evaluation: NoMethodError: undefined method 'gsub' for nil:NilClass (drbunix:///tmp/automation_engine20200903-3826-1nbuvl) /usr/local/ lib/ruby/gems/2.5.0/gems/manageiq-password-0.3.0/lib/manageiq/password.rb:89:in 'sanitize_string'
Remove Outer Quotes
Your input format is invalid: you're nesting unescaped double quotes, and somehow expecting that to work. Just leave off the outer quotes. For example:
require 'json'
json = <<~'EOF'
{"G000":{"1":{"max":"4000","name":"Matthew Lewis","id":"ord-matthewl","email":"matthew.lewis#companyx.com"},"2":{"max":"4000","name":"Brendan Jones","id":"ord-brendanj","email":"brendan.jones#companyx.com"},"3":{"max":"20000","name":"Henry Orson","id":"ord-henryo","email":"henry.orson#companyx.com"},"4":{"max":"10000000","name":"Chris Adams","id":"ord-chrisa","email":"chris.adams#companyx.com"}},"G15":{"1":{"max":"4000","name":"Mike Butak","id":"ord-mikebu","email":"mike.butak#companyx.com"},"2":{"max":"4000","name":"Joseph Lister","id":"ord-josephl","email":"joseph.lister#companyx.com"},"3":{"max":"20000","name":"Mike Geisler","id":"ord-mikeg","email":"mike.geisler#companyx.com"},"4":{"max":"10000000","name":"Samuel Ahn","id":"ord-samuela","email":"samuel.ahn#companyx.com"}}}
EOF
hash = JSON.parse(json)
hash.dig "G000", "4", "id"
#=> "ord-chrisa"
hash.dig "G15", "4", "id"
#=> "ord-samuela"
This question was answered by users on another post I opened: Why can Ruby not parse local JSON file?
Ultimately the issue was not Ruby failing to parse my JSON. Rather it was the logging function being unable to log the hash.

JSON deserialize with newtonsoft [duplicate]

I have seen the terms "deserialize" and "serialize" with JSON. What do they mean?
JSON is a format that encodes objects in a string. Serialization means to convert an object into that string, and deserialization is its inverse operation (convert string -> object).
When transmitting data or storing them in a file, the data are required to be byte strings, but complex objects are seldom in this format. Serialization can convert these complex objects into byte strings for such use. After the byte strings are transmitted, the receiver will have to recover the original object from the byte string. This is known as deserialization.
Say, you have an object:
{foo: [1, 4, 7, 10], bar: "baz"}
serializing into JSON will convert it into a string:
'{"foo":[1,4,7,10],"bar":"baz"}'
which can be stored or sent through wire to anywhere. The receiver can then deserialize this string to get back the original object. {foo: [1, 4, 7, 10], bar: "baz"}.
Serialize and Deserialize
In the context of data storage, serialization (or serialisation) is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer) or transmitted (for example, across a network connection link) and reconstructed later. [...]
The opposite operation, extracting a data structure from a series of bytes, is deserialization.
– wikipedia.org
JSON
JSON (JavaScript Object Notation) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other serializable values). It is a common data format with diverse uses in electronic data interchange, including that of web applications with servers.
JSON is a language-independent data format. It was derived from JavaScript, but many modern programming languages include code to generate and parse JSON-format data. JSON filenames use the extension .json.
– wikipedia.org
Explained using Python
In Python serialization does nothing else than just converting the given data structure into its valid JSON pendant (e.g., Python's True will be converted to JSON's true and the dictionary itself will be converted to a string) and vice versa for deserialization.
Python vs. JSON
You can easily spot the difference between Python and JSON representations in a side-by-side comparison. For example, by examining their Boolean values. Have a look at the following table for the basic types used in both contexts:
Python
JSON
True
true
False
false
None
null
int, float
number
str (with single ', double " and tripple """ quotes)
string (only double " quotes)
dict
object
list, tuple
array
Code Example
Python builtin module json is the standard way to do serialization and deserialization:
import json
data = {
'president': {
"name": """Mr. Presidente""",
"male": True,
'age': 60,
'wife': None,
'cars': ('BMW', "Audi")
}
}
# serialize
json_data = json.dumps(data, indent=2)
print(json_data)
# {
# "president": {
# "name": "Mr. Presidente",
# "male": true,
# "age": 60,
# "wife": null,
# "cars": [
# "BMW",
# "Audi"
# ]
# }
# }
# deserialize
restored_data = json.loads(json_data) # deserialize
Sources: realpython.com, geeksforgeeks.org
Explanation of Serialize and Deserialize using Python
In python, pickle module is used for serialization. So, the serialization process is called pickling in Python. This module is available in Python standard library.
Serialization using pickle
import pickle
#the object to serialize
example_dic={1:"6",2:"2",3:"f"}
#where the bytes after serializing end up at, wb stands for write byte
pickle_out=open("dict.pickle","wb")
#Time to dump
pickle.dump(example_dic,pickle_out)
#whatever you open, you must close
pickle_out.close()
The PICKLE file (can be opened by a text editor like notepad) contains this (serialized data):
€}q (KX 6qKX 2qKX fqu.
Deserialization using pickle
import pickle
pickle_in=open("dict.pickle","rb")
get_deserialized_data_back=pickle.load(pickle_in)
print(get_deserialized_data_back)
Output:
{1: '6', 2: '2', 3: 'f'}
Share what I learned about this topic.
What is Serialization
Serialization is the process of converting a data object into a byte stream.
What is byte stream
Byte stream is just a stream of binary data. Because only binary data can be stored or transported.
What is byte string vs byte stream
Sometime you see people use the word byte string as well. String encodings of bytes are called byte strings. Then it can explain what is JSON as below.
What’s the relationship between JSON and serialization
JSON is a string format representational of byte data. JSON is encoded in UTF-8. So while we see human readable strings, behind the scenes strings are encoded as bytes in UTF-8.

Parsing Json in Robot Framework

{
"data": [
{
"name": "John",
"mobile_phone": false,
"carrier": "none"
},
{
"name": "Jim",
"mobile_phone": true,
"carrier": "T-Mobile"
}
],
"result": 0
}
Hi, will it be possible to parse such JSON response in Robot Framework in a way where I will create kind of 'sub' list for each value ?
I would like to separate John from Jim and get for example information about carrier for Jim only (via another get request later in test).
Thanks !
Say the source text (the json) is stored in a variable ${source data}:
${source data}= Evaluate json.loads("""${source data}""") json
# the variable ${source data} is now a python dictionary - the same as the original json, but only - accessible as dictionary in robotframwork
${all data members}= Set Variable ${source data['data']}
${user_phone}= Create Dictionary
:FOR ${member} IN #{all data members} # iterate through the 'data', each ${member} is a dictionary in the source list
\ ${name}= Get From Dictionary ${member} name # will assign to the variable ${name} the value of the key 'name'; if there is no such key - the keyword will fail
\ Log The user ${name} has a mobile phone: ${member['mobile_phone']} # Will print "The user John has a mobile phone: False", "The user Jim has a mobile phone: True"
\ Set To Dictionary ${user_phone} ${name} ${member['mobile_phone']} # will fill-in a dictionary in the form "name": boolean_does_the_person_has_phone
This commented code sample shows how you can work with json/dictionary objects in robotframework.
The Evaluate keyword on line 1 runs arbitrary python code (its first argument, which calls the loads() method of the json module); its 2nd argument is any extra libraries that need to be imported - like json in our case.
The 4th line, Set Variable shows the Extended variable syntax - in this case, knowing that source data is a dictionary, getting the value of that key. At the end of this line execution, the variable all data members is the list that's inside the json's 'data' key.
Line 8 starts a loop, over that same list; the variable member will hold the value of each list's member on every iteration.
Line 9 uses a different (more orthodox) way to get the value of a dictionary's key - by using the keyword Get From Dictionary from the Collections library.
Line 10 logs a message, by employing a normal (name) and extended syntax (member['mobile_phone']) variables.
And on line 11, a dictionary entry is created, where the name is used as a key, and the boolean member['mobile_phone'] as a value (if there is already a key with the same name - it is overwritten). This keyword is again in the Collections library.

Erlang: Parse string to json

I have the following string:
"{\"headers\":[\"CNPJ\",\"PDF\",\"error\"],\"rows\":[[\"17192451000170\",\"FILE:application/pdf;170286;\",null],[\"234566767544\",\"FILE:application/pdf;456378;\",null],[\"233456767544\",\"FILE:application/pdf;456378;\",null]]}"
how do I parse it to a normal Json format?
meaning:
{"rows" :[
{"CNPJ":"17192451000170","PDF":"FILE:application/pdf;170286;","error":null},
{"CNPJ":"17192451000170","PDF":"FILE:application/pdf;170286;","error":null},
{"CNPJ":"17192451000170", "PDF":"FILE:application/pdf;170286;,"error":null"}
]}
or any other json format
This is already a valid JSON format.
If you just want to strip \ then you can simply:
(hbd#crayon2.yoonka.com)31> JsonOrg = <<"{\"headers\":[\"CNPJ\",\"PDF\",\"error\"],\"rows\":[[\"17192451000170\",\"FILE:application/pdf;170286;\",null],[\"234566767544\",\"FILE:application/pdf;456378;\",null],[\"233456767544\",\"FILE:application/pdf;456378;\",null]]}">>.
<<"{\"headers\":[\"CNPJ\",\"PDF\",\"error\"],\"rows\":[[\"17192451000170\",\"FILE:application/pdf;170286;\",null],[\"234566767544\",\"FI"...>>
(hbd#crayon2.yoonka.com)32> io:format("~s~n", [binary_to_list(JsonOrg)]).
{"headers":["CNPJ","PDF","error"],"rows":[["17192451000170","FILE:application/pdf;170286;",null],["234566767544","FILE:application/pdf;456378;",null],["233456767544","FILE:application/pdf;456378;",null]]}
ok
You can also parse back and forth between Json and Erlang. I tested that with the yajler decoder:
(hbd#crayon2.yoonka.com)43> {ok, Parsed} = yajler:decode(<<"{\"headers\":[\"CNPJ\",\"PDF\",\"error\"],\"rows\":[[\"17192451000170\",\"FILE:application/pdf;170286;\",null],[\"234566767544\",\"FILE:application/pdf;456378;\",null],[\"233456767544\",\"FILE:application/pdf;456378;\",null]]}">>).
{ok,[{<<"headers">>,[<<"CNPJ">>,<<"PDF">>,<<"error">>]},
{<<"rows">>,
[[<<"17192451000170">>,<<"FILE:application/pdf;170286;">>,
undefined],
[<<"234566767544">>,<<"FILE:application/pdf;456378;">>,
undefined],
[<<"233456767544">>,<<"FILE:application/pdf;456378;">>,
undefined]]}]}
(hbd#crayon2.yoonka.com)44> Json = binary:list_to_bin(yajler:encode(Parsed)).
<<"{\"headers\":[\"CNPJ\",\"PDF\",\"error\"],\"rows\":[[\"17192451000170\",\"FILE:application/pdf;170286;\",\"undefined\"],[\"2345667675"...>>
Yajler is an Erlang NIF so it is using a C library, in this case called yajl, to do the actual parsing, but I imagine a similar result you would get from other Erlang applications that can parse JSON.