Say I have this document stored.
exemplary data
{"name": "haris","projects": [{"title": "Splunk"},{"title": "QRadar"},{"title": "LogAnalysis"}]}
{"name": "khalid","projects": [{"title": "MS"},{"title": "Google"},{"title": "Apple"}]}
{"name": "Hamid","projects": [{"title": "Toyota"},{"title": "Honda"},{"title": "Kia"}]}
code for mapping
PUT index_name
{
"mappings": {
"iterations_type": {
"properties": {
"projects": {
"type": "nested",
"include_in_parent": true
}
}
}
}
}
My Question: I want to get 2nd element of "projects" array against some specific name e.g "name": "khalid".
What I tried so far:
I am new to ES and I searched forums for the answer, some suggest to specify mapping of "projects" key. I tried that too and the issue I face is
1. when I create empty index, specify the mapping first and then insert the data, I get an error like "reason": "object mapping ["projects"] can't be changed from nested to non-nested"
2. when I create empty index, insert data first and then specify the mapping, I get and error like ""reason": "index [<>] already exists"
ard for me to specify mapping for each field explicitly
Thanks in advance
Related
I want to take the data from here: https://raw.githubusercontent.com/usnistgov/oscal-content/master/examples/ssp/json/ssp-example.json
which I've pulled into a mySQL database called "ssp_models" into a JSON column called 'json_data', and I need add a new 'name' and 'type' entry into the 'parties' node with a new uuid in the same format as the example.
So in my mySQL database table, "ssp_models", I have this entry: Noting that I should be able to write the data by somehow referencing "66c2a1c8-5830-48bd-8fdd-55a1c3a52888" as the record to modify.
All the example I've seen online seem to force me to read out the entire JSON into a variable, make the addition, and then cram it back into the json_data column, which seems costly, especially with large JSON data-sets.
Isn't there a simple way I can say
"INSERT INTO ssp_models JSON_INSERT <somehow burrow down to 'system-security-plan'.metadata.parties (name, type) VALUES ('Raytheon', 'organization') WHERE uuid = '66c2a1c8-5830-48bd-8fdd-55a1c3a52888'
I was looking at this other stackoverflow example for inserting into JSON:
How to create and insert a JSON object using MySQL queries?
However, that's basically useful when you are starting from scratch, vs. needing to add JSON data to data that already exists.
You may want to read https://dev.mysql.com/doc/refman/8.0/en/json-function-reference.html and explore each of the functions, and try them out one by one, if you're going to continue working with JSON data in MySQL.
I was able to do what you describe this way:
update ssp_models set json_data = json_array_append(
json_data,
'$."system-security-plan".metadata.parties',
json_object('name', 'Bingo', 'type', 'farmer')
)
where uuid = '66c2a1c8-5830-48bd-8fdd-55a1c3a52888';
Then I checked the data:
mysql> select uuid, json_pretty(json_data) from ssp_models\G
*************************** 1. row ***************************
uuid: 66c2a1c8-5830-48bd-8fdd-55a1c3a52888
json_pretty(json_data): {
"system-security-plan": {
"uuid": "66c2a1c8-5830-48bd-8fdd-55a1c3a52888",
"metadata": {
"roles": [
{
"id": "legal-officer",
"title": "Legal Officer"
}
],
"title": "Enterprise Logging and Auditing System Security Plan",
"parties": [
{
"name": "Enterprise Asset Owners",
"type": "organization",
"uuid": "3b2a5599-cc37-403f-ae36-5708fa804b27"
},
{
"name": "Enterprise Asset Administrators",
"type": "organization",
"uuid": "833ac398-5c9a-4e6b-acba-2a9c11399da0"
},
{
"name": "Bingo",
"type": "farmer"
}
]
}
}
}
I started with data like yours, but for this test, I truncated everything after the parties array.
I am currently implementing some test automation that uses a json POST to a REST API to initialize the test data in the SUT. Most of the fields I don't have an issue editing using information I found in another thread: Json handling in ROBOT
However, one of the sets of information I am editing is a dictionary of meta data.
{
"title": "Test Auotmation Post 2018-03-06T16:12:02Z",
"content": "dummy text",
"excerpt": "Post made by automation for testing purposes.",
"name": "QA User",
"status": "publish",
"date": "2018-03-06T16:12:02Z",
"primary_section": "Entertainment",
"taxonomy": {
"section": [
"Entertainment"
]
},
"coauthors": [
{
"name": "QA User - CoAuthor",
"meta": {
"Title": "QA Engineer",
"Organization": "That One Place"
}
}
],
"post_meta": [
{
"key": "credit",
"value": "QA Engineer"
},
{
"key": "pub_date",
"value": "2018-03-06T16:12:02Z"
},
{
"key": "last_update",
"value": "2018-03-06T16:12:02Z"
},
{
"key": "source",
"value": "wordpress"
}
]
}
Is it possible to use the Set to Dictionary Keyword on a dictionary inside a dictionary? I would like to be able to edit the value of the pub_date and last_update inside of post_meta, specifically.
The most straightforward way would be to use the Evaluate keyword, and set the sub-dict value in it. Presuming you are working with a dictionary that's called ${value}:
Evaluate $value['post_meta'][1]['pub_date'] = 'your new value here'
I won't get into how to find the index of the post_meta list that has the 'key' with value 'pub_date', as that's not part of your question.
Is it possible to use the Set to Dictionary Keyword on a dictionary inside a dictionary?
Yes, it's possible.
However, because post_meta is a list rather than a dictionary, you will have to write some code to iterate over all of the values of post_meta until you find one with the key you want to update.
You could do this in python quite simply. You could also write a keyword in robot to do that for you. Here's an example:
*** Keywords ***
Set list element by key
[Arguments] ${data} ${target_key} ${new_value}
:FOR ${item} IN #{data}
\ run keyword if '''${item['key']}''' == '''${target_key}'''
\ ... set to dictionary ${item} value=${new_value}
[return] ${data}
Assuming you have a variable named ${data} contains the original JSON data as a string, you could call this keyword like the following:
${JSON}= evaluate json.loads('''${data}''') json
set list element by key ${JSON['post_meta']} pub_date yesterday
set list element by key ${JSON['post_meta']} last_update today
You will then have a python object in ${JSON} with the modified values.
I am trying to create a schema for a piece of JSON and have slimmed down an example of what I am trying to achieve.
I have the following JSON schema:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Set name",
"description": "The exmaple schema",
"type": "object",
"properties": {
"name": {
"type": "string"
}
},
"additionalProperties": false
}
The following JSON is classed as valid when compared to the schema:
{
"name": "W",
"name": "W"
}
I know that there should be a warning about the two fields having the same name, but is there a way to force the validation to fail if the above is submitted? I want it to only validate when there is only one occurrence of the field 'name'
This is outside of the responsibility of JSON Schema. JSON Schema is built on top of JSON. In JSON, the behavior of duplicate properties in an object is undefined. If you want to get warning about this you should run it through a separate validation step to ensure valid JSON before passing it to a JSON Schema validator.
There is a maxProperties constraint that can limit total number of properties in an object.
Though having data with duplicated properties is a tricky case as many json decoding implementions would ignore duplicate.
So your JSON schema validation lib would not even know duplicate existed.
In Athena, I am reading some nested JSON files into a table. The field that actually contains the nested JSON has an inconsistent number of fields within it across the different files in the raw data.
Sometimes the data looks something like this:
{
"id": "9f1e07b4",
"date": "05/20/2018 02:30:53.110 AM",
"data": {
"a": "asd",
"b": "adf",
"body": {
"sid": {
"uif": "yes",
"sidd": "no",
"state": "idle"
}
},
"category": "scene"
}
}
Other times the data looks something like this:
{
"id": "9f1e07b4",
"date": "05/20/2018 02:30:45.436 AM",
"data": {
"a": "event",
"b": "state",
"body": {
"persona": {
"one": {
"movement": "idle"
}
}
},
"category": "scene"
}
}
Other times the "body" field contains both the "sid" struct and the "persona" struct.
As you can see the fields given within "body" are not always consistent. I tried to add all of the possible fields and their structures within my CREATE EXTERNAL TABLE query. However, the "data" column that contains the "body" field still does not fill and remains blank when I "preview table" in Athena.
In the CREATE TABLE DDL, is there a way to indicate that I want to fill all of columns that aren't present in the nested JSON of each file with null values?
Furthermore, the 'names' given to the fields in the query do not have to correspond to the key values in the raw JSON. It seems Athena is simply reading the structure and nothing else. Is there a way to indicate which JSON key corresponds to which Athena field name directly? So that if some fields are missing from the "body" of one file, Athena can know which one is missing and fill it in as null?
Is there a way to express reference to another element in the same JSON document using JSON schema? The title might be a bit confusing, but i'm not looking for the "$ref" attribute, which references another type, but I'm curious, if there is a way, to reference another element in the document using a specified field. I know this is possible to enforce using xsd for xml documents, not sure about JSON.
I want to do something like this:
{
"people": [
{ "id": "1", "name": "A" },
{ "id": "2", "name": "B" },
{ "id": "3", "name": "C" }
],
"chosenOne": "1" // I want the schema to enforce a person ID here
}
I have been looking at the schema definition of v4: http://json-schema.org/draft-04/schema but didn't find anything, that looks like what I'm trying to do. Did I just miss it?
What you want is that you describe a reference ($ref) in the object your schema is describing.
kind of like this
{
"people": []
"chosenOne": { $ref: "#1"}
}
(or maybe a pointer if you want the value of the Id (https://json-spec.readthedocs.io/pointer.html)
I know of no direct way to do this but you might be able to use the pattern or oneof properties to force it being the right value. Kind of like this
"properties": {
"chosenOne"
"type": "string",
"oneOf": ["1","2","3"]
]
},
}
Similarly you could force the value of the property to be a reference pattern. That said since there is no reference value type (http://www.tutorialspoint.com/json/json_data_types.htm) only number or string you can't guarantee the meaning of the value. You can just guarantee that if follows some kind of reference pattern.
If you need more than what json schema's can give you you might want to look in odata for example. OData has some extra things so you can describe an entitySet and then define a navigation property to that set.
It does however force you to follow the odata structure so you aren't as free as you would be with a regular json schema.