Convert BigQuery rows to array of JSON - json

I wan to convert all the rows of BigQuery query output to an array of JSON.
For example: I want to convert the following output rows
Col1
Col2
ex1a
ex1b
ex2a
ex2b
Convert this to the following JSON:
{
"Col1":"ex1a",
"Col2":"ex1b"
},
{
"Col1":"ex2a",
"Col2":"ex2b"
}
]```

Use below approach
select format('[%s]', string_agg(to_json_string(t)))
from your_table t
if applied to sample data in your question - output is
Another option (with same output) is
select to_json_string(array_agg(t))
from your_table t

Related

How to extract JSON array stored as string in BigQuery

I have a JSON array that looks similar to this
[{"key":"Email","slug":"customer-email","value":"abc#gmail.com"},{"key":"Phone Number","slug":"mobile-phone-number","value":"123456789"},{"key":"First Name","slug":"first-name","value":"abc"},{"key":"Last Name","slug":"last-name","value":"xyz"},{"key":"Date of birth","slug":"date-of-birth","value":"01/01/1990"}]
But the tricky part is, this array is stored as string. So I am thinking that the first step would be to convert the string into array then unnest it then follow the method in here
I wonder if this method is doable, if so I guess the challenge that I am having is to convert string into array. If not, or if you have more efficient method please help. Thanks
Have you tried json_extract_array
select json_extract_array(
"""[{"key":"Email","slug":"customer-email","value":"abc#gmail.com"},{"key":"Phone Number","slug":"mobile-phone-number","value":"123456789"},{"key":"First Name","slug":"first- name","value":"abc"},{"key":"Last Name","slug":"last-name","value":"xyz"},{"key":"Date of birth","slug":"date-of-birth","value":"01/01/1990"}]""");
Below is for BigQuery Standard SQL
#standardSQL
SELECT
id,
JSON_EXTRACT_ARRAY(json_string) AS json_array
FROM `project.dataset.table`
if to apply to sample data from your question as in below
#standardSQL
WITH `project.dataset.table` AS (
SELECT 1 id, '[{"key":"Email","slug":"customer-email","value":"abc#gmail.com"},{"key":"Phone Number","slug":"mobile-phone-number","value":"123456789"},{"key":"First Name","slug":"first-name","value":"abc"},{"key":"Last Name","slug":"last-name","value":"xyz"},{"key":"Date of birth","slug":"date-of-birth","value":"01/01/1990"}]' json_string
)
SELECT
id,
JSON_EXTRACT_ARRAY(json_string) AS json_array
FROM `project.dataset.table`
output is
Row id json_array
1 1 {"key":"Email","slug":"customer-email","value":"abc#gmail.com"}
{"key":"Phone Number","slug":"mobile-phone-number","value":"123456789"}
{"key":"First Name","slug":"first-name","value":"abc"}
{"key":"Last Name","slug":"last-name","value":"xyz"}
{"key":"Date of birth","slug":"date-of-birth","value":"01/01/1990"}
From this point - you can use solution in How do I parse value from JSON array into columns in BigQuery that you referenced in your question

How to convert json object into array in prestodb/athena

I have a JSON object in this format data = {"1": {"col1":"a", "col2":"b"}, "2": {"col1":"c", "col2":"d"}....,"99":{"col1":"asd", "col2":"exm"}}. I would like to get all the values for col_1 and col_2 using athena. How do I achieve this using athena?
I was able to solve it by converting it to Map and then unnesting it like
select key, value from table where
unnest(cast(json_parse(data) as Map(varchar,JSON)) as t(key,value))
Ignore that it is a JSON format (since it is not simple enough) and use the regex function:
SELECT regexp_extract_all(column_data, '"col1":"([a-z]+)",'); -- [a, c, ... asd]

How to get output of a bigquery in a specific json format

I have a bigquery table in this format :
DataProvider,Id,Name,Time
ABC,f8453e99-516f-4f15-a3bd-8749089b6934,"xyz",43200
ABC,f8453e99-516f-4f15-a3bd-8749089b6934,"123",43200
ABC,00453e99-516f-4f15-a3bd-8749089b6934,"xyz",43200
I want to generate the output in this format (json) :
{"dataProviderId":"ABC","items":[{"Id":"f8453e99-516f-4f15-a3bd-8749089b6934","data":[{"Name":"xyz","Time":43200},{"Name":"xyz","Time":43200}],
{"Id":"00453e99-516f-4f15-a3bd-8749089b6934","data":[{"Name":"xyz","Time":43200}]}
In your CLI, you can use bq command with --format flag, where you can pass prettyjson format (easy-to-read JSON format).
bq query --format=prettyjson --use_legacy_sql=false 'SELECT * FROM `project_id`:dataset.table' > output.json
By using > at the end of the command, it is possible to save the output of a command to a new file. You will be able to see the output of query in output.json file.
I hope it helps.
Below is for BigQuery Standard SQL
#standardSQL
SELECT TO_JSON_STRING(t) json
FROM (
SELECT dataProvider, ARRAY_AGG(STRUCT(id, data)) items
FROM (
SELECT dataProvider, id, ARRAY_AGG(STRUCT(name, time)) data
FROM `project.dataset.table` t
GROUP BY dataProvider, id
)
GROUP BY dataProvider
) t
If to apply to sample data in your question - output is
Row json
1 {"dataProvider":"ABC","items":[{"id":"f8453e99-516f-4f15-a3bd-8749089b6934","data":[{"name":"xyz","time":43200},{"name":"123","time":43200}]},{"id":"00453e99-516f-4f15-a3bd-8749089b6934","data":[{"name":"xyz","time":43200}]}]}

MySQL Doing sum of nested JSON values

I have a field in a table called duration that contains a JSON string like this:
{
"videos": {
"en":"00:03:11",
"es":"00:03:11"
},
"audios": {
"en":"00:00:03",
"es":"00:00:03"
}
}
Is it possible to execute a query to sum all the values of both videos and audio keys using the possible langauges? Meaning that given this JSON structure, I'd like a query to return:
00:06:28
EDIT:
Don't mind the format of the values for the moment, there are ways to sum datetime values in SQL. What I'm struggling with now is to traverse the values in the JSON to actually sum them.

Select JSON array's fields from SQL view to create a column

I have an SQL Table which one of the columns contain a JSON array in the following format:
[
{
"id":"1",
"translation":"something here",
"value":"value of something here"
},
{
"id":"2",
"translation":"something else here",
"value":"value of something else here"
},
..
..
..
]
Is there any way to use an SQL Query and retrieve columns with the ID as header and the "value" as the value of the column? Instead of return only one column with the JSON array.
For example, if I run:
SELECT column_with_json FROM myTable
It will return the above array. Where I want to return
1,2
value of something here, value of something else here
You can't use SQL to retrieve columns from the JSON stored inside the table: to the database engine the JSON is just unstructured text saved in a text field.
Some relational databases, like PostgreSQL, have a JSON type and functions to support JSON query. If this is your case, you should be able to perform the query you want.
Check this for an example on how it work with PostgreSQL:
http://clarkdave.net/2013/06/what-can-you-do-with-postgresql-and-json/