mySQL JSON Document Store method for inserting data into node 3 levels deep - mysql

I want to take the data from here: https://raw.githubusercontent.com/usnistgov/oscal-content/master/examples/ssp/json/ssp-example.json
which I've pulled into a mySQL database called "ssp_models" into a JSON column called 'json_data', and I need add a new 'name' and 'type' entry into the 'parties' node with a new uuid in the same format as the example.
So in my mySQL database table, "ssp_models", I have this entry: Noting that I should be able to write the data by somehow referencing "66c2a1c8-5830-48bd-8fdd-55a1c3a52888" as the record to modify.
All the example I've seen online seem to force me to read out the entire JSON into a variable, make the addition, and then cram it back into the json_data column, which seems costly, especially with large JSON data-sets.
Isn't there a simple way I can say
"INSERT INTO ssp_models JSON_INSERT <somehow burrow down to 'system-security-plan'.metadata.parties (name, type) VALUES ('Raytheon', 'organization') WHERE uuid = '66c2a1c8-5830-48bd-8fdd-55a1c3a52888'
I was looking at this other stackoverflow example for inserting into JSON:
How to create and insert a JSON object using MySQL queries?
However, that's basically useful when you are starting from scratch, vs. needing to add JSON data to data that already exists.

You may want to read https://dev.mysql.com/doc/refman/8.0/en/json-function-reference.html and explore each of the functions, and try them out one by one, if you're going to continue working with JSON data in MySQL.
I was able to do what you describe this way:
update ssp_models set json_data = json_array_append(
json_data,
'$."system-security-plan".metadata.parties',
json_object('name', 'Bingo', 'type', 'farmer')
)
where uuid = '66c2a1c8-5830-48bd-8fdd-55a1c3a52888';
Then I checked the data:
mysql> select uuid, json_pretty(json_data) from ssp_models\G
*************************** 1. row ***************************
uuid: 66c2a1c8-5830-48bd-8fdd-55a1c3a52888
json_pretty(json_data): {
"system-security-plan": {
"uuid": "66c2a1c8-5830-48bd-8fdd-55a1c3a52888",
"metadata": {
"roles": [
{
"id": "legal-officer",
"title": "Legal Officer"
}
],
"title": "Enterprise Logging and Auditing System Security Plan",
"parties": [
{
"name": "Enterprise Asset Owners",
"type": "organization",
"uuid": "3b2a5599-cc37-403f-ae36-5708fa804b27"
},
{
"name": "Enterprise Asset Administrators",
"type": "organization",
"uuid": "833ac398-5c9a-4e6b-acba-2a9c11399da0"
},
{
"name": "Bingo",
"type": "farmer"
}
]
}
}
}
I started with data like yours, but for this test, I truncated everything after the parties array.

Related

How to insert json data to sqflite table without clicking anything in app(Something like at first init of the app)

Can you help me with inserting ready json data that is stored in assets file to sqflite table.
And for example my json file
[
{
"word": "json",
"value": "format that uses human-readable text to store and transmit data "
},
{
"word": "life",
"value": "a gift"
},
{
"word": "work",
"value": "money"
}
]
and also at inserting that data to table, sqflite should add an another string for each key-value containing something like "inFavorite": "false".
so I've seen a lot of videos doing that in app, but not one that insert ready data to sqflite.
I would be grateful if you could help me!
Is that possible?
Thank you for your help!
I tried a lot of things but there is no result

Pentaho Kettle: How to dynamically fetch JSON file columns

Background: I work for a company that basically sells passes. Every order that is placed by the customer will contain N number of passes.
Issue: I have these JSON event-transaction files coming into a S3 bucket on a daily basis from DocumentDB (MongoDB). This JSON file is associated to the relevant type of event (insert, modify or delete) for every document key (which is an order in my case). The example below illustrates a "Insert" type of event that came through to the S3 bucket:
{
"_id": {
"_data": "11111111111111"
},
"operationType": "insert",
"clusterTime": {
"$timestamp": {
"t": 11111111,
"i": 1
}
},
"ns": {
"db": "abc",
"coll": "abc"
},
"documentKey": {
"_id": {
"$uuid": "abcabcabcabcabcabc"
}
},
"fullDocument": {
"_id": {
"$uuid": "abcabcabcabcabcabc"
},
"orderNumber": "1234567",
"externalOrderId": "12345678",
"orderDateTime": "2020-09-11T08:06:26Z[UTC]",
"attraction": "abc",
"entryDate": {
"$date": 2020-09-13
},
"entryTime": {
"$date": 04000000
},
"requestId": "abc",
"ticketUrl": "abc",
"tickets": [
{
"passId": "1111111",
"externalTicketId": "1234567"
},
{
"passId": "222222222",
"externalTicketId": "122442492"
}
],
"_class": "abc"
}
}
As we see above, every JSON file might contain N number of passes and every pass is - in turn - is associated to an external ticket id, which is a different column (as seen above). I want to use Pentaho Kettle to read these JSON files and load the data into the DW. I am aware of the Json input step and Row Normalizer that could then transpose "PassID 1", "PassID 2", "PassID 3"..."PassID N" columns into 1 unique column "Pass" and I would have to have to apply a similar logic to the other column "External ticket id". The problem with that approach is that it is quite static, as in, I need to "tell" Pentaho how many Passes are coming in advance in the Json input step. However what if tomorrow I have an order with 10 different passes? How can I do this dynamically to ensure the job will not break?
If you want a tabular output like
TicketUrl Pass ExternalTicketID
---------- ------ ----------------
abc PassID1Value1 ExTicketIDvalue1
abc PassID1Value2 ExTicketIDvalue2
abc PassID1Value3 ExTicketIDvalue3
And make incoming value dynamic based on JSON input file values, then you can download this transformation Updated Link
I found everything work dynamic in JSON input.

How to get particular element by index of an array in elasticsearch

Say I have this document stored.
exemplary data
{"name": "haris","projects": [{"title": "Splunk"},{"title": "QRadar"},{"title": "LogAnalysis"}]}
{"name": "khalid","projects": [{"title": "MS"},{"title": "Google"},{"title": "Apple"}]}
{"name": "Hamid","projects": [{"title": "Toyota"},{"title": "Honda"},{"title": "Kia"}]}
code for mapping
PUT index_name
{
"mappings": {
"iterations_type": {
"properties": {
"projects": {
"type": "nested",
"include_in_parent": true
}
}
}
}
}
My Question: I want to get 2nd element of "projects" array against some specific name e.g "name": "khalid".
What I tried so far:
I am new to ES and I searched forums for the answer, some suggest to specify mapping of "projects" key. I tried that too and the issue I face is
1. when I create empty index, specify the mapping first and then insert the data, I get an error like "reason": "object mapping ["projects"] can't be changed from nested to non-nested"
2. when I create empty index, insert data first and then specify the mapping, I get and error like ""reason": "index [<>] already exists"
ard for me to specify mapping for each field explicitly
Thanks in advance

Express JS fetch proper JSON with Mysql

I am working on WebAPI on ExpressJS. I am fetching data from MySQL database by using node module mysql. I have a mater and a detail table let say
Countries(ID, Name, Code)
States (ID, Name, CountryId)
Now I want to query the data in JSON as following:
{
"country":{
"ID" : 1,
"Name" : "United States of America"
"Code" : "USA"
"Cities" :
[
{
"1",
"Alabama",
"AL"
},
{
"2",
"Alaska",
"AK"
},
{
"3",
"California",
"CA"
}
]
}
}
How can I query data in the above format?
Thanks and Regards.
How can I query data in the above format?
You can't because this is not a valid JSON. No matter what you do, you will never be able to represent your data in that format.
To represent data coming from a relational database as a JavaScript nested objects and arrays, it will be easiest to use an ORM like Sequelize, Bookshelf, ORM2 or Waterline. See:
http://docs.sequelizejs.com/
http://bookshelfjs.org/
https://github.com/dresende/node-orm2
http://sailsjs.com/documentation/reference/waterline-orm
ORM is an Object-Relational Mapper and lets you map the relational model to objects, which is exactly what you're trying to do here.

Ad-hoc queries to a massive JSON dataset

I have a massive dataset stored in Azure BLOB in JSON format. Some apps are constantly adding new data to it. BLOBs are organized in partitions like
/dataset={name}/date={YYYY-MM-DD}/one_or_more_json_files
Data pieces do not follow any particular schema. JSON field names are not in consistent letter case. Some JSON rows can be broken.
Could someone advise a good way to query this data without defining schema in advance. I would like to do something like
select * from my_huge_json_dataset where dataset='mydataset' and date>'2015-04-01'
without defining explicit schema for the table
My first consideration was HIVE but it turns out that SerDe needs schema to be defined to create a table. json_tuple could be an answer but it is case-sensitive and crashes if meets malformed json row.
I am also considering Apache Drill and Pig but have no experience with them and would like some guidance.
You could use Apache Drill, you only need to configure a new storage pointing to your dataset folder:
{
"type": "file",
"enabled": true,
"connection": "file:///",
"config": null,
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null
}
},
"formats": {
"json": {
"type": "json",
"extensions": [
"json"
]
}
}
So, if you defined that Storage Plugin as 'dfs' for example you could query from the root directory without defining any schema using ANSI SQL, just like:
SELECT * FROM dfs.dataset.date.`file.json`;
or even filter by your folder name in the same query using dir0.
I encourage you to visit their documentation site Apache Drill documentation in your case specially Querying JSON files