I working on parsing a json string stored in a table CLOB in oracle 11g. This process is part of a long parsing routine that parses the data and stores the values in another table and I've just noticed that part of my data is not getting out. The json parses and validates with JSONLint. So I've simplified the parsing to try and find out where I'm going wrong.
So my json coming out my table looks like this.
{
"JSON_data": {
"plant_id": "3006",
"transmit_time": "2015-12-18 11:57:45",
"messages": [{
"work_msg": {
"msg_time": "2015-06-23 04:54:17",
"trigger_type": "interval",
"vert_correction": 358.3,
"ch_latitude": 37.916302,
"ch_longitude": -87.487365,
"ch_heading": 212.3,
"ch_cable_port": 1029.79,
"ch_cable_stbd": 348.63,
"ch_depth": -27.03,
"slurry_velocity": 25.71,
"slurry_density": 1.02,
"ch_rpm": 205.49,
"ch_psi": 540.89,
"prod_instantaneous": 0,
"prod_cumulative": 1216.100000,
"outfall_latitude": 37.915967,
"outfall_longitude": -87.484369,
"outfall_heading": 120.7,
"pump_entries": [{
"pump_name": "main",
"vacuum": 12.73,
"outlet_psi": 22.88
}],
"spud_entries": [{
"position": 6
}]
},
"pipe_length_event": {
"msg_time": "2015-06-23 04:54:17",
"length_floating": 970
}
}]
}
}
My parsing is correctly finding and doing its thing with the 'work_msg' data. It's the 'pipe_length_event' data that I'm not getting to. Below is my simplified pl/sql procedure.
DECLARE
vCONTENT CLOB;
v_parent_json json;
v_json_message_list json_list;
v_json_message_list_value json_value;
v_parent_json_value json_value;
BEGIN
SELECT CONTENT INTO vCONTENT FROM SJM_TEMP4;
v_parent_json := json(vCONTENT);
v_parent_json := json(v_parent_json.get(1));
v_json_message_list := json_list(v_parent_json.get('messages'));
DBMS_OUTPUT.PUT_LINE(v_json_message_list.count);
for message_loop_counter in 1 ..v_json_message_list.count loop
v_parent_json_value := json(v_json_message_list.get(message_loop_counter)).get(1);
DBMS_OUTPUT.PUT_LINE(v_parent_json_value.mapname);
END LOOP;
END;
My dbms_output first gives me a sub-list count of 1. Not 2 so my parsing is not even recognizing the "pipe_length_event" as a sub-list of "messages".
How do I get "pipe_length_event" data using this procedure? I'm almost certain this was working in the past so my first thought is that the json is formatted differently. Is the json ill-formatted?
Thanks in advance.
FOUND IT!!
The issue is in fact the JSON formatting. Below is the correct format. The "work_msg" list was not closed so "pipe_length_event" list was not recognized.
{
"JSON_data": {
"plant_id": "3006",
"transmit_time": "2015-12-18 11:57:45",
"messages": [{
"work_msg": {
"msg_time": "2015-06-23 04:54:17",
"trigger_type": "interval",
"vert_correction": 358.3,
"ch_latitude": 37.916302,
"ch_longitude": -87.487365,
"ch_heading": 212.3,
"ch_cable_port": 1029.79,
"ch_cable_stbd": 348.63,
"ch_depth": -27.03,
"slurry_velocity": 25.71,
"slurry_density": 1.02,
"ch_rpm": 205.49,
"ch_psi": 540.89,
"prod_instantaneous": 0,
"prod_cumulative": 1216.100000,
"outfall_latitude": 37.915967,
"outfall_longitude": -87.484369,
"outfall_heading": 120.7,
"pump_entries": [{
"pump_name": "main",
"vacuum": 12.73,
"outlet_psi": 22.88
}],
"spud_entries": [{
"position": 6
}]
}
}, {
"pipe_length_event": {
"msg_time": "2015-06-23 04:54:17",
"length_floating": 970
}
}]
}
}
Related
I have a sqlite database and in one of the fields I have stored complete json object . I have to make some json select requests . If you see my json
the ALL key has value which is an array . We need to extract some data like all comments where "pod" field is fb . How to extract properly when sqlite json has value as an array ?
select json_extract(data,'$."json"') from datatable ; gives me entire thing . Then I do
select json_extract(data,'$."json"[0]') but i dont want to do it manually . i want to iterate .
kindly suggest some source where i can study and work on it .
MY JSON
{
"ALL": [{
"comments": "your site is awesome",
"pod": "passcode",
"originalDirectory": "case1"
},
{
"comments": "your channel is good",
"data": ["youTube"],
"pod": "library"
},
{
"comments": "you like everything",
"data": ["facebook"],
"pod": "fb"
},
{
"data": ["twitter"],
"pod": "tw",
"ALL": [{
"data": [{
"codeLevel": "3"
}],
"pod": "mo",
"pod2": "p"
}]
}
]
}
create table datatable ( path string , data json1 );
insert into datatable values("1" , json('<abovejson in a single line>'));
Simple List
Where your JSON represents a "simple" list of comments, you want something like:
select key, value
from datatable, json_each( datatable.data, '$.ALL' )
where json_extract( value, '$.pod' ) = 'fb' ;
which, using your sample data, returns:
2|{"comments":"you like everything","data":["facebook"],"pod":"fb"}
The use of json_each() returns a row for every element of the input JSON (datatable.data), starting at the path $.ALL (where $ is the top-level, and ALL is the name of your array: the path can be omitted if the top-level of the JSON object is required). In your case, this returns one row for each comment entry.
The fields of this row are documented at 4.13. The json_each() and json_tree() table-valued functions in the SQLite documentation: the two we're interested in are key (very roughly, the "row number") and value (the JSON for the current element). The latter will contain elements called comment and pod, etc..
Because we are only interested in elements where pod is equal to fb, we add a where clause, using json_extract() to get at pod (where $.pod is relative to value returned by the json_each function).
Nested List
If your JSON contains nested elements (something I didn't notice at first), then you need to use the json_tree() function instead of json_each(). Whereas the latter will only iterate over the immediate children of the node specified, json_tree() will descend recursively through all children from the node specified.
To give us some data to work with, I have augmented your test data with an extra element:
create table datatable ( path string , data json1 );
insert into datatable values("1" , json('
{
"ALL": [{
"comments": "your site is awesome",
"pod": "passcode",
"originalDirectory": "case1"
},
{
"comments": "your channel is good",
"data": ["youTube"],
"pod": "library"
},
{
"comments": "you like everything",
"data": ["facebook"],
"pod": "fb"
},
{
"data": ["twitter"],
"pod": "tw",
"ALL": [{
"data": [{
"codeLevel": "3"
}],
"pod": "mo",
"pod2": "p"
},
{
"comments": "inserted by TripeHound",
"data": ["facebook"],
"pod": "fb"
}]
}
]
}
'));
If we were to simply switch to using json_each(), then we see that a simple query (with no where clause) will return all elements of the source JSON:
select key, value
from datatable, json_tree( datatable.data, '$.ALL' ) limit 10 ;
ALL|[{"comments":"your site is awesome","pod":"passcode","originalDirectory":"case1"},{"comments":"your channel is good","data":["youTube"],"pod":"library"},{"comments":"you like everything","data":["facebook"],"pod":"fb"},{"data":["twitter"],"pod":"tw","ALL":[{"data":[{"codeLevel":"3"}],"pod":"mo","pod2":"p"},{"comments":"inserted by TripeHound","data":["facebook"],"pod":"fb"}]}]
0|{"comments":"your site is awesome","pod":"passcode","originalDirectory":"case1"}
comments|your site is awesome
pod|passcode
originalDirectory|case1
1|{"comments":"your channel is good","data":["youTube"],"pod":"library"}
comments|your channel is good
data|["youTube"]
0|youTube
pod|library
Because JSON objects are mixed in with simple values, we can no longer simply add where json_extract( value, '$.pod' ) = 'fb' because this produces errors when value does not represent an object. The simplest way around this is to look at the type values returned by json_each()/json_tree(): these will be the string object if the row represents a JSON object (see above documentation for other values).
Adding this to the where clause (and relying on "short-circuit evaluation" to prevent json_extract() being called on non-object rows), we get:
select key, value
from datatable, json_tree( datatable.data, '$.ALL' )
where type = 'object'
and json_extract( value, '$.pod' ) = 'fb' ;
which returns:
2|{"comments":"you like everything","data":["facebook"],"pod":"fb"}
1|{"comments":"inserted by TripeHound","data":["facebook"],"pod":"fb"}
If desired, we could use json_extract() to break apart the returned objects:
.mode column
.headers on
.width 30 15 5
select json_extract( value, '$.comments' ) as Comments,
json_extract( value, '$.data' ) as Data,
json_extract( value, '$.pod' ) as POD
from datatable, json_tree( datatable.data, '$.ALL' )
where type = 'object'
and json_extract( value, '$.pod' ) = 'fb' ;
Comments Data POD
------------------------------ --------------- -----
you like everything ["facebook"] fb
inserted by TripeHound ["facebook"] fb
Note: If your structure contained other objects, of different formats, it may not be sufficient to simply select for type = 'object': you may have to devise a more subtle filtering process.
I have a file composed of a single array containing multiple records.
{
"Client": [
{
"ClientNo": 1,
"ClientName": "Alpha",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "12345"
},
{
"BusinessNo": 2,
"IndustryCode": "23456"
}
]
},
{
"ClientNo": 2,
"ClientName": "Bravo",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "34567"
},
{
"BusinessNo": 2,
"IndustryCode": "45678"
}
]
}
]
}
I load it with the following code:
create or replace stage stage.test
url='azure://xxx/xxx'
credentials=(azure_sas_token='xxx');
create table if not exists stage.client (json_data variant not null);
copy into stage.client_test
from #stage.test/client_test.json
file_format = (type = 'JSON' strip_outer_array = true);
Snowflake imports the entire file as one row.
I would like the the COPY INTO command to remove the outer array structure and load the records into separate table rows.
When I load larger files, I hit the size limit for variant and get the error Error parsing JSON: document is too large, max size 16777216 bytes.
If you can import the file into Snowflake, into a single row, then you can use LATERAL FLATTEN on the Clients field to generate one row per element in the array.
Here's a blog post on LATERAL and FLATTEN (or you could look them up in the snowflake docs):
https://support.snowflake.net/s/article/How-To-Lateral-Join-Tutorial
If the format of the file is, as specified, a single object with a single property that contains an array with 500 MB worth of elements in it, then perhaps importing it will still work -- if that works, then LATERAL FLATTEN is exactly what you want. But that form is not particularly great for data processing. You might want to use some text processing script to massage the data if that's needed.
RECOMMENDATION #1:
The problem with your JSON is that it doesn't have an outer array. It has a single outer object containing a property with an inner array.
If you can fix the JSON, that would be the best solution, and then STRIP_OUTER_ARRAY will work as you expected.
You could also try to recompose the JSON (an ugly business) after reading line for line with:
CREATE OR REPLACE TABLE X (CLIENT VARCHAR);
COPY INTO X FROM (SELECT $1 CLIENT FROM #My_Stage/Client.json);
User Response to Recommendation #1:
Thank you. So from what I gather, COPY with STRIP_OUTER_ARRAY can handle a file starting and ending with square brackets, and parse the file as if they were not there.
The real files don't have line breaks, so I can't read the file line by line. I will see if the source system can change the export.
RECOMMENDATION #2:
Also if you would like to see what the JSON parser does, you can experiment using this code, I have parsed JSON on the copy command using similar code. Working with your JSON data in small project can help you shape the Copy command to work as intended.
CREATE OR REPLACE TABLE SAMPLE_JSON
(ID INTEGER,
DATA VARIANT
);
INSERT INTO SAMPLE_JSON(ID,DATA)
SELECT
1,parse_json('{
"Client": [
{
"ClientNo": 1,
"ClientName": "Alpha",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "12345"
},
{
"BusinessNo": 2,
"IndustryCode": "23456"
}
]
},
{
"ClientNo": 2,
"ClientName": "Bravo",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "34567"
},
{
"BusinessNo": 2,
"IndustryCode": "45678"
}
]
}
]
}');
SELECT
C.value:ClientNo AS ClientNo
,C.value:ClientName::STRING AS ClientName
,ClientBusiness.value:BusinessNo::Integer AS BusinessNo
,ClientBusiness.value:IndustryCode::Integer AS IndustryCode
from SAMPLE_JSON f
,table(flatten( f.DATA,'Client' )) C
,table(flatten(c.value:ClientBusiness,'')) ClientBusiness;
User Response to Recommendation #2:
Thank you for the parse_json example!
Trouble is, the real files are sometimes 500 MB, so the parse_json function chokes.
Follow-up on Recommendation #2:
The JSON needs to be in the NDJSON http://ndjson.org/ format. Otherwise the JSON will be impossible to parse because of the potential for large files.
Hope the above helps other running into similar questions!
I would like to get the single element in the Couchbase document that is in the array of objects, but i am able to fetch the array of objects
i tried to fetch the array using the following query, 'select countryDetails from test';
{
"type":"countries",
"docName":"CountryData",
"countryDetails":[
{
"name":"US",
"code":"+1",
"stateInfo":[
{
"name":"Florida",
"id":"1212"
},
{
"name":"NewYork",
"id":"1214"
}
]
},
{
"name":"France",
"code":"+33",
"stateInfo":[
{
"name":"Grand Est",
"id":"5212"
},
{
"name":"Brittany",
"id":"5214"
}
]
}
]
}
i tried fetching array using, select countryDetails from test;
i like to fetch the result as [ {"name" : "US", "code" : "+1" }, {"name" : "France", "code" : "+33"}]
If you project countryDetails it projects whole sub object.
If you need to part of sub object you need to explicitly project that.
The following ARRAY construction will provide the data representation you are expecting.
SELECT ARRAY {v.name,v.code} FOR v IN t.countryDetails END AS contryDetails
FROM test AS t
WHERE t.type = "countries";
What you are trying to do does not seem to be possible. You can get closer to what you want with a query like this:
select raw countryDetails from test
But the results of this query still have the result wrapped in an extra level of array.
I create a wallpaper apps and I found this error
org.json.JSONException: No value for index
and the apps can't start
Here's my json
]
{
"wallpaper_index": "1",
"wallpaper_name": "Wallpaper1",
"wallpaper_site_name": "Wallpaper",
"wallpaper_site_url": "http://google.com",
"wallpaper_url": "https://i.imgur.com/Z0UGroX.jpg"
},
{
"wallpaper_index": "2",
"wallpaper_name": "Wallpaper2",
"wallpaper_site_name": "Wallpaper",
"wallpaper_site_url": "http://google.com",
"wallpaper_url": "https://i.imgur.com/nSx1uN8.jpg"
},
{
"wallpaper_index": "3",
"wallpaper_name": "Wallpaper2",
"wallpaper_site_name": "Wallpaper",
"wallpaper_site_url": "http://google.com",
"wallpaper_url": "https://i.imgur.com/ezhbnQR.jpg"
},
]
How to fix it ?
Thanks in advance
NOTE: Your json is not valid. It should start with the "[" symbol ( and not with "]" one ) and you should remove the comma on
second-last row. Here is the correct json after the revision:
[
{
"wallpaper_index":"1",
"wallpaper_name":"Wallpaper1",
"wallpaper_site_name":"Wallpaper",
"wallpaper_site_url":"http://google.com",
"wallpaper_url":"https://i.imgur.com/Z0UGroX.jpg"
},
{
"wallpaper_index":"2",
"wallpaper_name":"Wallpaper2",
"wallpaper_site_name":"Wallpaper",
"wallpaper_site_url":"http://google.com",
"wallpaper_url":"https://i.imgur.com/nSx1uN8.jpg"
},
{
"wallpaper_index":"3",
"wallpaper_name":"Wallpaper2",
"wallpaper_site_name":"Wallpaper",
"wallpaper_site_url":"http://google.com",
"wallpaper_url":"https://i.imgur.com/ezhbnQR.jpg"
}
]
By the way, assuming you just wrote the json wrong doing the question, the issue is that you are trying to get value for a not existing entry ( "index" ) on your json array elements.
According to your json, if you change for example the requested element to "wallpaper_index" it will work.
I have the following JSON from server:
{
"SuccessResponse": {
"Head": {
"RequestId": "",
"RequestAction": "GetMultipleOrderItems",
"ResponseType": "Orders",
"Timestamp": "2016-05-10T15:13:06-0300"
},
"Body": {
"Orders": {
"Order": [
{
"OrderId": "457634",
"OrderNumber": "256176682",
"OrderItems": {
"OrderItem": {
"OrderItemId": "712893",
"ShopId": "14690930",
"OrderId": "457634",
...
I'm using the following code to access this values:
procedure TForm1.GetOrdersPendingItems;
var
mydata : string;
obj, orderObj: ISuperObject;
orderArray: TSuperArray;
begin
mydata := GetURLAsString(GenerateApiUrl('GetMultipleOrderItems', 'OrderIdList', '[457634,457817]'));
obj := SO(mydata);
orderObj := obj['SuccessResponse.Body.Orders.Order'];
end;
With this code, if I use a simple Label1.Caption := orderObj.AsString;, it show me this:
"OrderId": "457634",
"OrderNumber": "256176682",
"OrderItems": {
"OrderItem": {
"OrderItemId": "712893",
"ShopId": "14690930",
"OrderId": "457634",
...
By the logic, the values inner of OrderItem can be access like this: orderObj['OrderItems.OrderItem'];, but if I try to access a "easy" value like OrderId, that is the first element, using orderObj['OrderId']; it returns nil and the same happens with all nodes of the orderObj...
So, the values in the orderObj.AsString can't be accessed to convert into variable...
There are a way to access the value inner of OrderItem? My objective is convert the values of OrderItem into a ClientDataSet using the following code:
orderArray := orderObj.AsArray;
TJSONDB.JsonToClientDataSet(orderArray, cdsOrdersItems);
Thanks!
Here you mention this:
By the logic, the values inner of OrderItem can be access like this: orderObj['OrderItems.OrderItem'];
This would work, indeed.But right after you wrote this contradicting the last sentence:
but if I try to access a "easy" value like OrderId, that is the first element, using orderObj['OrderId'];
By the logic, as you say, to access the values you could do:
orderObj['OrderItems.OrderItem.OrderId'];
and not orderObj['OrderId']; directly.