I am using JSON column type in MySQL database table. When I try to save JSON values in table column, the JSON array automatically re-order(shuffle)
I have following JSON:
{"TIMER_HEADER": [{"XX!TIMERHDR": "XXTIMERHDR", "VER": " 7", "REL": " 0", "COMPANYNAME": "XXX", "IMPORTEDBEFORE": "N", "FROMTIMER": "N", "COMPANYCREATETIME": "12423426"}, {"XX!HDR": "XXHDR", "PROD": "Qics for Wnows", "VER": "Version 6.0", "REL": "Release R", "IIFVER": "1", "DATE": "2018-01-20", "TIME": "1516520267", "ACCNTNT": "N", "ACCNTNTSPLITTIME": "0"}], "COLUMN_HEADER": ["!TIMEACT", "DATE", "JOB", "EMP", "ITEM", "PITEM", "DURATION", "PROJ", "NOTE", "BILLINGSTATUS"]}
After saving in JSON-column of MySql table, this becomes:
{"TIMER_HEADER": [{"REL": " 0", "VER": " 7", "FROMTIMER": "N", "COMPANYNAME": "XXX", "XX!TIMERHDR": "XXTIMERHDR", "IMPORTEDBEFORE": "N", "COMPANYCREATETIME": "12423426"}, {"REL": "Release R", "VER": "Version 6.0", "DATE": "2018-01-20", "PROD": "Qics for Wnows", "TIME": "1516520267", "IIFVER": "1", "XX!HDR": "XXHDR", "ACCNTNT": "N", "ACCNTNTSPLITTIME": "0"}], "COLUMN_HEADER": ["!TIMEACT", "DATE", "JOB", "EMP", "ITEM", "PITEM", "DURATION", "PROJ", "NOTE", "BILLINGSTATUS"]}
I need the same order as I have original, because there is an validation on 3rd party.
Please help. Thanks.
The items of arrays don't change order in MySQL JSON columns, but object key/value pairs might.
MySQL manual says:
To make lookups more efficient, it also sorts the keys of a JSON object. You should be aware that the result of this ordering is subject to change and not guaranteed to be consistent across releases.
There is no standard that says it has to be in a certain order, so probably the best thing to do is to speak to someone on 3rd party to adjust validation.
We are using blob for storage. JSON validation is handled by our application.
Using this blob column, keys are not sorted since it not optimized for JSON.
Please refer the blob docs for details. https://dev.mysql.com/doc/refman/8.0/en/blob.html
Related
I'm trying to check if a value contains a partial string in this sample API call result in JSON format. It should look through job title text to see if it contains the word "Virtual" or "Remote" anywhere in the string. If so, append 1 to new dictionary, and if not, append 0. I keep getting 1's for each value, but not sure why 0's don't show up if it can't find any match. Thanks! Here's the code:
import requests
remote_job_check = {'Remote':[]}
job_lists = {
"jobs": [
{
"country_code": "US",
"state": "AL",
"title": "Sr. Customer Solutions Manager - Virtual Opportunities",
},
{
"country_code": "US",
"state": "CA",
"title": "Data Engineer",
},
{
"country_code": "US",
"state": "CA",
"title": "Data Engineer - Remote",
},
{
"country_code": "US",
"state": "NV",
"title": "Remote Software Engineer",
},
{
"country_code": "US",
"state": "FL",
"title": "Product Manager",
}
]
}
job_data = requests.get(job_lists).json()['jobs']
for details in job_data:
if 'Virtual' or 'Remote' in details.get('title'):
remote_job_check['Remote'].append(1)
else:
remote_job_check['Remote'].append(0)
I was able to figure it out. Thanks for the suggestions though. I learned that my original code only searches for one of the strings. The 'or' operator wasn't even working in this case:
for details in job_data:
if 'Virtual' or 'Remote' in details.get('title'): # The 'or' operator not working
remote_job_check['Remote'].append(1)
else:
remote_job_check['Remote'].append(0)
Then I tried using just one string and it worked.
for details in job_data:
if 'Virtual' in details.get('title'): # This works
remote_job_check['Remote'].append(1)
else:
remote_job_check['Remote'].append(0)
So now I figured I need to create some sort of array to iterate through and found the 'any' function. Here's the working code:
remote_job_keywords = ['Virtual', 'Remote']
if any(keyword in details['title'] for keyword in remote_job_keywords):
job_data['Remote'].append(1)
else:
job_data['Remote'].append(0)
UPDATE: Thanks to #shriakhilc for the feedback on my answer. Like he said, the above works but it was easier to fix the or operator like so:
for details in job_data:
if 'Virtual' in details.get('title') or 'Remote' in details.get('title'):
remote_job_check['Remote'].append(1)
else:
remote_job_check['Remote'].append(0)
I'm struggling to make a regex work with splunk. It works with regex 101, but splunk doesn't seem to recognize it!
Regex: \"([\w]+)\":([^,}]+)
Log entry:
May 20 12:22:21 127.0.0.1 {"rootId": "AXIxikL8ao-yaSvA", "requestId": "f6a873jkjjkjk:-8000:5738",
"details": {"flag": false, "title": "task 1", "status": "Waiting", "group": "", "order": 0},
"operation": "Creation", "objectId": "AXIyCN5Oao-H5aYyaSvd", "startDate": 1589977341890,
"objectType": "case_task", "base": true, "object": {"_routing": "AXIxikL8ao-H5aYyaSvA", "flag":
false, "_type": "case_task", "title": "task 1", "createdAt": 1589977341516, "_parent": "AXIxikL8ao-
H5aYyaSvA", "createdBy": "user", "_id": "AXIyCN5Oao-H5aYyaSvd", "id": "AXIyCN5Oao-H5aYyaSvd",
"_version": 1, "order": 0, "status": "Waiting", "group": ""}}
Regex 101 link:
https://regex101.com/r/XBuz9Y/2/
I suspect splunk may have a different regex syntax, but i don't really know how to adapt it.
Any help?
Thanks!
You may use
... | rex max_match=0 "\"(?<key>\w+)\":(?<value>[^,}]+)"
Here, max_match=0 will enable multiple matching (by defauly, if you do not use max_match parameter, only the first match is returned) and the named capturing groups (here, see (?<key>...) and (?<value>...)) will ensure field creation.
See more about the Splunk rex command.
Grab the JSON fragment of your event using rex, and then use spath to do the extraction.
rex field=_raw "^[^{]+(?<json>.*)" | spath input=json
This should extract the JSON fields with the appropriate structure.
I am trying to query some chatbot data from Azure Blob Storage with Excel Power Query. Unfortunately, Excel cells not only contain the value but also the key in each cell.
This is some sample content of a blob in my blobstorage. The data always has the same structure.
{"id":"3398","realId":"3398","document":{"userID":"3398","name":"Testdurchgang","age":18,"gender":"männlich","education":"Diplom","complete":true,"advisoryConversationId":"6EFa4fsLaJhB4U1LlwNksU-f","roundCounter":5,"riskchoices":["A","A","A","A","B","B","B","B","B","B"],"riskAssessmentComplete":true,"riskDescription":"risikoneutral","order":["2","1","0"],"botRecommendation":"Breen GmbH","choice":"ACG GmbH","follow":false,"eTag":"*","resultConversationId":"Kxw9FZ2KwdWKqElSQQ0nG2-f","win1":"none","win2":"ACG GmbH","loss1":"Breen GmbH","loss2":"Plus GmbH","payout":"Du bekommst 6500 Geldeinheiten = 6,50€ ausgezahlt.","payoutNumber":"6,5"}}
This is how the data looks like in the Power Query tool:
As you can see, the first column even contains the bracket from the JSON and column 3 contains the "content" tag. In the best case, I want only the value in each cell. I can set the column description manually if necessary.
Is there a way to achieve this?
I know I could do it with Excel functions but this would not be a really nice solution.
Thank you in advance.
Make sure the file you have stored in Storage blob container is of .json format.
Was able to upload your sample content into blob as below
{
"id": "3398",
"realId": "3398",
"document": {
"userID": "3398",
"name": "Testdurchgang",
"age": 18,
"gender": "männlich",
"education": "Diplom",
"complete": true,
"advisoryConversationId": "6EFa4fsLaJhB4U1LlwNksU-f",
"roundCounter": 5,
"riskchoices": [
"A",
"A",
"A",
"A",
"B",
"B",
"B",
"B",
"B",
"B"
],
"riskAssessmentComplete": true,
"riskDescription": "risikoneutral",
"order": [
"2",
"1",
"0"
],
"botRecommendation": "Breen GmbH",
"choice": "ACG GmbH",
"follow": false,
"eTag": "*",
"resultConversationId": "Kxw9FZ2KwdWKqElSQQ0nG2-f",
"win1": "none",
"win2": "ACG GmbH",
"loss1": "Breen GmbH",
"loss2": "Plus GmbH",
"payout": "Du bekommst 6500 Geldeinheiten = 6,50€ ausgezahlt.",
"payoutNumber": "6,5"
}
}
In excel was able to import
Click on Binary and should see below
Click on Record and should see the below
Insert into the table should show something as below
Hope the above steps help !
A MySQL table has a JSON column containing a large amount of JSON data.
For example:
SELECT nodes From Table LIMIT 1;
results in:
'{"data": {"id": "Node A", "state": true, "details": [{"value": "Value","description": "Test"}, {"value": "Value2", "description": "Test2"}, {"value": "Value 7", "description": "Test 7"}, {"value": "Value 9", "description": "Test 9"}]}}'
How can I write queries that return rows in accordance with the following examples:
Where Node A state is True. In this case "Node A" is the value of key "id" and "state" contains True or False.
Where "value" is "Value2" or where "description" is "Test2." Note that these values are in a list that contains key value pairs.
I doubt if you can make a direct MySQL query to achieve above task. You will have to load all the string data from MySQL db then parse this string to get JSON object upon which you can perform your custom query operation to get your output.
But here in this case i will suggest you to use MongoDB which will be an ideal database storage solution and you can make direct queries.
I want to be able to access deeper elements stored in a json in the field json, stored in a postgresql database. For example, I would like to be able to access the elements that traverse the path states->events->time from the json provided below. Here is the postgreSQL query I'm using:
SELECT
data#>> '{userId}' as user,
data#>> '{region}' as region,
data#>>'{priorTimeSpentInApp}' as priotTimeSpentInApp,
data#>>'{userAttributes, "Total Friends"}' as totalFriends
from game_json
WHERE game_name LIKE 'myNewGame'
LIMIT 1000
and here is an example record from the json field
{
"region": "oh",
"deviceModel": "inHouseDevice",
"states": [
{
"events": [
{
"time": 1430247045.176,
"name": "Session Start",
"value": 0,
"parameters": {
"Balance": "40"
},
"info": ""
},
{
"time": 1430247293.501,
"name": "Mission1",
"value": 1,
"parameters": {
"Result": "Win ",
"Replay": "no",
"Attempt Number": "1"
},
"info": ""
}
]
}
],
"priorTimeSpentInApp": 28989.41467999999,
"country": "CA",
"city": "vancouver",
"isDeveloper": true,
"time": 1430247044.414,
"duration": 411.53,
"timezone": "America/Cleveland",
"priorSessions": 47,
"experiments": [],
"systemVersion": "3.8.1",
"appVersion": "14312",
"userId": "ef617d7ad4c6982e2cb7f6902801eb8a",
"isSession": true,
"firstRun": 1429572011.15,
"priorEvents": 69,
"userAttributes": {
"Total Friends": "0",
"Device Type": "Tablet",
"Social Connection": "None",
"Item Slots Owned": "12",
"Total Levels Played": "0",
"Retention Cohort": "Day 0",
"Player Progression": "0",
"Characters Owned": "1"
},
"deviceId": "ef617d7ad4c6982e2cb7f6902801eb8a"
}
That SQL query works, except that it doesn't give me any return values for totalFriends (e.g. data#>>'{userAttributes, "Total Friends"}' as totalFriends). I assume that part of the problem is that events falls within a square bracket (I don't know what that indicates in the json format) as opposed to a curly brace, but I'm also unable to extract values from the userAttributes key.
I would appreciate it if anyone could help me.
I'm sorry if this question has been asked elsewhere. I'm so new to postgresql and even json that I'm having trouble coming up with the proper terminology to find the answers to this (and related) questions.
You should definitely familiarize yourself with the basics of json
and json functions and operators in Postgres.
In the second source pay attention to the operators -> and ->>.
General rule: use -> to get a json object, ->> to get a json value as text.
Using these operators you can rewrite your query in the way which returns correct value of 'Total Friends':
select
data->>'userId' as user,
data->>'region' as region,
data->>'priorTimeSpentInApp' as priotTimeSpentInApp,
data->'userAttributes'->>'Total Friends' as totalFriends
from game_json
where game_name like 'myNewGame';
Json objects in square brackets are elements of a json array.
Json arrays may have many elements.
The elements are accessed by an index.
Json arrays are indexed from 0 (the first element of an array has an index 0).
Example:
select
data->'states'->0->'events'->1->>'name'
from game_json
where game_name like 'myNewGame';
-- returns "Mission1"
select
data->'states'->0->'events'->1->>'name'
from game_json
where game_name like 'myNewGame';
This did help me