I have a python script that polls an API and stores the response in a BQ table with layered, stringified JSON field that looks like so:
"{""inputs"": {""Layer1"": {""Layer2"": {""Layer3"": {""item1"": 0.7, ""item2"": 10.0, ""item3"": 0.14}}}}"
I created the following query to unnest:
ro.id,
json_extract_scalar(s,
'$.item1') AS item1,
json_extract_scalar(s,
'$.item2') AS item2,
json_extract_scalar(s,
'$.item3') AS item3
FROM
`project.dataset.table` ro
LEFT JOIN
UNNEST(json_extract_array(response)) AS r
LEFT JOIN
UNNEST (json_extract_array("Layer1")) AS o
LEFT JOIN
UNNEST (json_extract_array("Layer2")) AS sc
LEFT JOIN
UNNEST (json_extract_array("Layer3")) AS s
The query runs and builds the table as expected, but all of the data from the unnested fields is null.
I do not have control over how the script stores the data, so I can only leverage SQL to solve this problem. Are the two sets of double quotes my main issue here? If so, what is the best way to address that?
Consider below approach
SELECT
json_extract_scalar(items,'$.item1') AS item1,
json_extract_scalar(items,'$.item2') AS item2,
json_extract_scalar(items,'$.item3') AS item3,
FROM `project.dataset.table` ro,
UNNEST([struct(json_extract(response, '$.inputs.Layer1.Layer2.Layer3') as items)])
if applied to sample data in your question - output is
Related
I want to join a column with a json value. The problem is the json value is within square brackets and its a uuid. Table name that has the json column(column named json) is department and the other table name is staff. The json column value would be like below,
{"title":"Manager","alternativeTitle":null,"departmentIds":["c8098u43-7d9a-3789-gt56-r78009v4r345"]}
I would like to query the departmentIds from the json column and join it with staffdepartmentID column in the staff table.
My query for the join
from staff s
join department d on d.json ->> departmentIds::json = s.staffdepartmentID
The problem I am facing is that I dont know how to remove those square brackets. Any help is highly appreciated. Thanks
Square braquets within a json data correspond to an array.
You can access any element of the array based on its position starting with 0 for the first element : array->0
So for your query you can do :
from staff s
inner join department d
on d.json -> 'departmentIds'->>0 = s.staffdepartmentID :: text
Im having problems querying data from json fields.
I have some json format columns saved as text in my postgres database (version 10)
Sometimes i need to be able to join two tables on json values.
I have no idea how i can do this..
Here is an simple example.
In my select i want to output the fruit and the color.
I have the color_cd number inside a json in the fruits table and i can find the color inside another json in the code_table.
My wished output should be like this
Fruit_ID Name Color
1000 Pear Green
1001 Banana Yellow
Fiddle link --> https://dbfiddle.uk/?rdbms=postgres_13&fiddle=3f989db0524e288183619bab63fc9add
Your column records on table code_table has problem in json data and i fixed problem and change to below format:
{"color_cd":{"30":{"code":"30","color":"yellow"},"55":{"code":"55","color":"green"},"60":{"code":"60","color":"red"}}}
You can see query structure and result in dbfiddle
select
f.id,
f.name,
j_cd.value ->> 'color' as color
from
code_table ct
cross join jsonb_each(records::jsonb -> 'color_cd') j_cd
inner join fruits f on f.type_cd :: jsonb ->> 'color_cd' = j_cd.key::text
where
ct.name = 'color_cd'
In the database (postgres 10) i work with at the moment i have a lot of columns with json format strings (text fields)
I have a hard time to search,filter and join with values inside thoose strings.
I created an example in fiddle and i hope someone can help me solve it.
https://dbfiddle.uk/?rdbms=postgres_13&fiddle=d003c1cea832e35696260b10c6b4c047
Here is my two problems i found tricky.
In table object -> configuration field i have "date_kg_later" field. It could be [] or hold more dates. In my select i want to get the highest date if it's not empty.
In table object -> object_type_cd refers on a value inside the records string in code_table. Here i want to get the title from that string back.
My goal is an output like this:
id name object_type_name date
1000 Headphones tech 2022-04-30
1001 Pencil null null
Your first question : if the json structure of your configuration field conforms the example you provide, then a solution can be :
Before Postgres 12 :
SELECT Max((c.elt->>'date_from') :: date)
FROM object
CROSS JOIN LATERAL json_array_elements(configuration->'date_kg_later') AS c(elt)
GROUP BY your_object_table_key
From and after Postgres 12 :
SELECT Max(d :: text :: date)
FROM object
CROSS JOIN LATERAL jsonb_array_elements((configuration :: jsonb)->'date_kg_later', '$[*].date_from') AS d
GROUP BY your_object_table_key
Your second question is unclear to me, needs more explanation.
My table looks like as follows; (I populated it from excel)
I want to extract some data from another table so I use sql joins. Since my column contains comma separated values, i try to use join with Or but no success. Is this right way to do joins?
I look for quick solution for this comma separated columns.
eg:
SELECT * FROM test.types as a
inner join test.`matric as ma on (a.category= SUBSTRING_INDEX(ma.`Function Code AA`,',',1)
or a.category= SUBSTRING_INDEX(ma.`Function Code AA`,',',2)
or a.category= SUBSTRING_INDEX(ma.`Function Code AA`,',',3)
or a.category= SUBSTRING_INDEX(ma.`Function Code AA`,',',4)
or a.category= SUBSTRING_INDEX(ma.`Function Code AA`,',',5)
or a.category= SUBSTRING_INDEX(ma.`Function Code AA`,',',6))
and a.type = ma.`function Code NN` and ma.`Priority` = "T1"
You have a horrible data model. You probably cannot speed up the query very much. But you can at least simplify the code.
SELECT *
FROM test.types t JOIN
test.matric ma
ON FIND_IN_SET(t.category, REPLACE(ma.`Function Code AA`, ', ', ',')) > 0 OR
t.type = ma.`function Code NN`
WHERE ma.`Priority` = 'T1';
However, you should fix your data model!!!. Here are some issues:
Databases have very poor string processing capabilities.
Values should be stored using the correct type.
Foreign keys should be declared properly.
Such a structure prevents the database from using indexes, partitions, and the best optimization methods.
SQL has a great way to store lists. It is not called a string. It is called a table.
Datamodel
A person is represented in the database as a meta table row with a name and with multiple attributes which are stored in the data table as key-value pair (key and value are in separate columns).
Simplified data-model
Now there is a query to retrieve all users (name) with all their attributes (data). The attributes are returned as JSON object in a separate column. Here is an example:
name data
Florian { "age":25 }
Markus { "age":25, "color":"blue" }
Thomas {}
The SQL command looks like this:
SELECT
name,
json_object_agg(d.key, d.value) AS data,
FROM meta AS m
JOIN (
JOIN d.fk_id, d.key, d.value AS value FROM data AS d
) AS d
ON d.fk_id = m.id
GROUP BY m.name;
Problem
Now the problem I am facing is, that users like Thomas which do not have any attributes stored in the key-value table, are not shown with my select function. This is because it does only a JOIN and no LEFT OUTER JOIN.
If I would use LEFT OUTER JOIN then I run into the problem, that json_object_agg try's to aggregate NULL values and dies with an error.
Approaches
1. Return empty list of keys and values
So I tried to check if the key-column of a user is NULL and return an empty array so json_object_agg would just create an empty JSON object.
But there is not really a function to create an empty array in SQL. The nearest thing I found was this:
select '{}'::text[];
In combination with COALESCE the query looks like this:
json_object_agg(COALESCE(d.key, '{}'::text[]), COALESCE(d.value, '{}'::text[])) AS data
But if I try to use this I get following error:
ERROR: COALESCE types text and text[] cannot be matched
LINE 10: json_object_agg(COALESCE(d.key, '{}'::text[]), COALES...
^
Query failed
PostgreSQL said: COALESCE types text and text[] cannot be matched
So it looks like that at runtime d.key is a single value and not an array.
2. Split up JSON creation and return empty list
So I tried to take json_object_agg and replace it with json_object which does not aggregate the keys for me:
json_object(COALESCE(array_agg(d.key), '{}'::text[]), COALESCE(array_agg(d.value), '{}'::text[])) AS data
But there I get the error that null value not allowed for object key. So COALESCE does not check that the array is empty.
Qustion
So, is there a function to check if a joined column is empty, and if yes return just a simple JSON object?
Or is there any other solution which would solve my problem?
Use left join with coalesce(). As default value use '{}'::json.
select name, coalesce(d.data, '{}'::json) as data
from meta m
left join (
select fk_id, json_object_agg(d.key, d.value) as data
from data d
group by 1
) d
on m.id = d.fk_id;
name | data
---------+------------------------------------
Florian | { "age" : "25" }
Marcus | { "age" : "25", "color" : "blue" }
Thomas | {}
(3 rows)