I'm trying to get the data from a JSON in BigQuery. This JSON is Stored in a one-column table.
So far, I've been able to get only the "variables" array, with the following:
Select JSON_QUERY_ARRAY(Column1, '$.sessions[0].variables') FROM Table
How can I get the other values/arrays (sessionMessage and events)? I can't make it work..
I've tried with:
JSON_VALUE(Column1, '$.sessions[0].conversation')
JSON_QUERY_ARRAY(Column1, '$.sessions[0].sessionMessages')
But I get only empty values (The original json has values inside this arrays..)
{
"fromDate":"2020-04-10T23:47:17.161Z",
"pageRows":151,
"sessions":[
{
"variables":[],
"sessionDate":"2020-04-10T23:47:17.161Z",
"botMessages":2,
"userHasTalked":"true",
"topics":[
"TOPIC1"
],
"sessionId":"WXXXSXSXSXXXQ_2020-01-00T23:47:17.161Z",
"platformContactId":"XXXXXXX-XXXXXXX-XXXXXXXXXXXXXX",
"sessionMessages":[.....],
"queues":[
"QUEUE1",
"QUEUE2"
],
"customerId":"SSDSDS",
"userMessages":2,
"operatorMessages":1,
"sessionMessagesQty":2,
"sessionStartingCause":"Organic",
"channelId":"IDCHANEL",
"conversation":"https://url.com",
"events":[.....]
}
],
"toDate":"2020-04-10T23:47:17.161Z",
"hasMore":true,
"pageToken":"XXXXXXXXXXXXXX"
}
There is nothing wrong with the function and JSONPath that you used, but your sample JSON file has some unexpected thing, like [.....], removing/replacing those and query below works fine:
WITH a as (select
"""
{
"fromDate":"2020-04-10T23:47:17.161Z",
"pageRows":151,
"sessions":[
{
"variables":[],
"sessionDate":"2020-04-10T23:47:17.161Z",
"botMessages":2,
"userHasTalked":"true",
"topics":[
"TOPIC1"
],
"sessionId":"WXXXSXSXSXXXQ_2020-01-00T23:47:17.161Z",
"platformContactId":"XXXXXXX-XXXXXXX-XXXXXXXXXXXXXX",
"sessionMessages":[1,2,3],
"queues":[
"QUEUE1",
"QUEUE2"
],
"customerId":"SSDSDS",
"userMessages":2,
"operatorMessages":1,
"sessionMessagesQty":2,
"sessionStartingCause":"Organic",
"channelId":"IDCHANEL",
"conversation":"https://url.com",
"events":[],
}
],
"toDate":"2020-04-10T23:47:17.161Z",
"hasMore":true,
"pageToken":"XXXXXXXXXXXXXX"
}
""" data)
SELECT JSON_VALUE(data, '$.sessions[0].conversation'),
JSON_QUERY_ARRAY(data, '$.sessions[0].sessionMessages')
FROM a;
Related
Currently my NodeJS code sends JSON back in the following format;
{"data":
[
{"audioname":"mytalk.m4a","name":"Josie ","email":"josie#gmail.com"},
{"audioname":"mytalk40.m4a","name":"Jessie James","email":"jesse#gmail.com"},
{"audioname":"mytalk.m4a","name":"Joan Bologney","email":"joan#gmail.com"}
]
}
But I'd like to get rid of the "data" and send back just;
[
{"audioname":"mytalk.m4a","name":"Josie ","email":"josie#gmail.com"},
{"audioname":"mytalk40.m4a","name":"Jessie James","email":"jesse#gmail.com"},
{"audioname":"mytalk.m4a","name":"Joan Bologney","email":"joan#gmail.com"}
]
Here's the query;
query = mysql.format(query);
connection.query(query,function(err,data){
if(err) {
res.json({"Error" : true, "Message" : "Error executing MySQL query"});
} else {
res.json({data});
}
});
If the object shown above is available though the data variable, we can pass only the data array by using:
res.json(data.data);
Where the second data references to the data array in the data variable.
Small example:
const data = {"data": [{"audioname":"mytalk.m4a","name":"Josie ","email":"josie#gmail.com"}, {"audioname":"mytalk40.m4a","name":"Jessie James","email":"jesse#gmail.com"}, {"audioname":"mytalk.m4a","name":"Joan Bologney","email":"joan#gmail.com"} ] };
console.log(data.data);
Removing the inner brackets seemed to work;
res.json(data);
I have a file composed of a single array containing multiple records.
{
"Client": [
{
"ClientNo": 1,
"ClientName": "Alpha",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "12345"
},
{
"BusinessNo": 2,
"IndustryCode": "23456"
}
]
},
{
"ClientNo": 2,
"ClientName": "Bravo",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "34567"
},
{
"BusinessNo": 2,
"IndustryCode": "45678"
}
]
}
]
}
I load it with the following code:
create or replace stage stage.test
url='azure://xxx/xxx'
credentials=(azure_sas_token='xxx');
create table if not exists stage.client (json_data variant not null);
copy into stage.client_test
from #stage.test/client_test.json
file_format = (type = 'JSON' strip_outer_array = true);
Snowflake imports the entire file as one row.
I would like the the COPY INTO command to remove the outer array structure and load the records into separate table rows.
When I load larger files, I hit the size limit for variant and get the error Error parsing JSON: document is too large, max size 16777216 bytes.
If you can import the file into Snowflake, into a single row, then you can use LATERAL FLATTEN on the Clients field to generate one row per element in the array.
Here's a blog post on LATERAL and FLATTEN (or you could look them up in the snowflake docs):
https://support.snowflake.net/s/article/How-To-Lateral-Join-Tutorial
If the format of the file is, as specified, a single object with a single property that contains an array with 500 MB worth of elements in it, then perhaps importing it will still work -- if that works, then LATERAL FLATTEN is exactly what you want. But that form is not particularly great for data processing. You might want to use some text processing script to massage the data if that's needed.
RECOMMENDATION #1:
The problem with your JSON is that it doesn't have an outer array. It has a single outer object containing a property with an inner array.
If you can fix the JSON, that would be the best solution, and then STRIP_OUTER_ARRAY will work as you expected.
You could also try to recompose the JSON (an ugly business) after reading line for line with:
CREATE OR REPLACE TABLE X (CLIENT VARCHAR);
COPY INTO X FROM (SELECT $1 CLIENT FROM #My_Stage/Client.json);
User Response to Recommendation #1:
Thank you. So from what I gather, COPY with STRIP_OUTER_ARRAY can handle a file starting and ending with square brackets, and parse the file as if they were not there.
The real files don't have line breaks, so I can't read the file line by line. I will see if the source system can change the export.
RECOMMENDATION #2:
Also if you would like to see what the JSON parser does, you can experiment using this code, I have parsed JSON on the copy command using similar code. Working with your JSON data in small project can help you shape the Copy command to work as intended.
CREATE OR REPLACE TABLE SAMPLE_JSON
(ID INTEGER,
DATA VARIANT
);
INSERT INTO SAMPLE_JSON(ID,DATA)
SELECT
1,parse_json('{
"Client": [
{
"ClientNo": 1,
"ClientName": "Alpha",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "12345"
},
{
"BusinessNo": 2,
"IndustryCode": "23456"
}
]
},
{
"ClientNo": 2,
"ClientName": "Bravo",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "34567"
},
{
"BusinessNo": 2,
"IndustryCode": "45678"
}
]
}
]
}');
SELECT
C.value:ClientNo AS ClientNo
,C.value:ClientName::STRING AS ClientName
,ClientBusiness.value:BusinessNo::Integer AS BusinessNo
,ClientBusiness.value:IndustryCode::Integer AS IndustryCode
from SAMPLE_JSON f
,table(flatten( f.DATA,'Client' )) C
,table(flatten(c.value:ClientBusiness,'')) ClientBusiness;
User Response to Recommendation #2:
Thank you for the parse_json example!
Trouble is, the real files are sometimes 500 MB, so the parse_json function chokes.
Follow-up on Recommendation #2:
The JSON needs to be in the NDJSON http://ndjson.org/ format. Otherwise the JSON will be impossible to parse because of the potential for large files.
Hope the above helps other running into similar questions!
I would like to get the single element in the Couchbase document that is in the array of objects, but i am able to fetch the array of objects
i tried to fetch the array using the following query, 'select countryDetails from test';
{
"type":"countries",
"docName":"CountryData",
"countryDetails":[
{
"name":"US",
"code":"+1",
"stateInfo":[
{
"name":"Florida",
"id":"1212"
},
{
"name":"NewYork",
"id":"1214"
}
]
},
{
"name":"France",
"code":"+33",
"stateInfo":[
{
"name":"Grand Est",
"id":"5212"
},
{
"name":"Brittany",
"id":"5214"
}
]
}
]
}
i tried fetching array using, select countryDetails from test;
i like to fetch the result as [ {"name" : "US", "code" : "+1" }, {"name" : "France", "code" : "+33"}]
If you project countryDetails it projects whole sub object.
If you need to part of sub object you need to explicitly project that.
The following ARRAY construction will provide the data representation you are expecting.
SELECT ARRAY {v.name,v.code} FOR v IN t.countryDetails END AS contryDetails
FROM test AS t
WHERE t.type = "countries";
What you are trying to do does not seem to be possible. You can get closer to what you want with a query like this:
select raw countryDetails from test
But the results of this query still have the result wrapped in an extra level of array.
Trying to figure out the best way to query a MySQL table containing a json column.
I am successfully able to get product OR port.
SELECT ip, JSON_EXTRACT(json_data, '$.data[*].product' ) FROM `network`
This will return:
["ftp","ssh"]
What I'm looking to get is something like this, or some other way to represent association and handle null values:
[["ftp",21],["ssh",22],[NULL,23]]
Sample JSON
{
"key1":"Value",
"key2":"Value",
"key3":"Value",
"data": [
{
"product":"ftp",
"port":"21"
},
{
"product":"ssh",
"port":"22"
},
{
"port":"23"
}
]
}
Can someone please help me with how to filter the results returned by the the query builder json servlet? The following is the json response,
{
"success":true,
"results":2,
"total":2,
"more":false,
"offset":0,
"hits":[
{
"SourceNodePath":"/content/en/events",
"Status":"COMPLETED",
"dateRequested":1492325940000,
"ContentType":"PAGE",
"SubmissionId":[
"016192"
],
"SourceLanguage":"en",
"TargetLanguages":[
"fr"
],
"dateCreated":1492191038787,
"dateReceived":1492191112322,
"Identifier":1492191038787,
"Initiator":"user",
"name":"2",
"Code":"201"
},
{
"SourceNodePath":"/content/en/toolbar",
"Status":"COMPLETED",
"dateRequested":1492325940000,
"ContentType":"PAGE",
"SubmissionId":[
"016190"
],
"SourceLanguage":"en",
"TargetLanguages":[
"de"
],
"dateCreated":1492190651609,
"dateReceived":1492190694082,
"Identifier":1492190651609,
"Initiator":"foo",
"name":"1",
"Code":"201"
}
]
}
I'm new to AEM development, is there any way to modify the QueryBuilder JSON Servlet so that it displays the results that has the "Initiator" value as, for example in this case, "user"? The "Initiator" takes the value of the username on AEM Sign In.
In your query you can define it like
property=Initiator
property.value=user
This article outlines all of the options, so take a minute to read through it:
https://docs.adobe.com/docs/en/aem/6-2/develop/search/querybuilder-api.html
I also assume you know you can build queries and see the differences in output with different parameters at this URL: /libs/cq/search/content/querydebug.html