In my project there is codegen function which produces this type of sql script:
ALTER TABLE `SomeTable`
ADD COLUMN `column_country` JSON GENERATED ALWAYS AS (`col`->>'$[*].country') STORED NOT NULL
As result in database I have following table's DDL
CREATE TABLE `SomeTable`(
`column_country` json GENERATED ALWAYS AS
(json_unquote(json_extract(`col`,'$[*].country'))) STORED NOT NULL,
)
Some how "->>" was converted to (json_unquote(json_extract(...,'...'))).
I need to wrap part "(json_unquote(json_extract(col,'$[*].country')))" to IFNULL function and to have as result:
CREATE TABLE `SomeTable`(
`column_country` json GENERATED ALWAYS AS
(json_unquote(IFNULL(json_extract(`col`,'$[*].country', "[]")))) STORED NOT NULL,
)
But how can I do it with this "->>" syntactic ? At least where I can write about it?
Related
I'll start this off by saying I know that there are more practical ways to solve this. It's more of an intellectual curiosity than anything else.
I've inherited a MySQL database where some columns are stored as varchar(5) but actually contain the literals "True" or "False". Changing the structure of the data is not an option right now due to other issues. I'm mapping the columns to an ORM (SQLAlchemy), and I want the column to be mapped to a Boolean data type in the supporting codebase using a type adapter. (I've written this adapter already; it's not the problem.)
To help make the mapping process faster, I'm writing a small query to look at the INFORMATION_SCHEMA table and build a line of Python code defining the column using the ORM's syntax. I cannot assume that the data type varchar(5) is a Boolean column - I need to inspect the contents of that column to see if there are values contained in it besides True and False.
Can I write a query that will both get the column type from INFORMATION_SCHEMA and check the actual values stored in that column?
Here is the query I have so far:
SELECT CONCAT(
"Column(""",
col.column_name,
""", ",
(CASE
WHEN col.DATA_TYPE = "int" THEN "Integer"
-- Code in question
WHEN
col.DATA_TYPE = "varchar"
AND col.CHARACTER_MAXIMUM_LENGTH = 5
AND NOT EXISTS(
-- Doesn't seem to work
SELECT DISTINCT col.COLUMN_NAME
FROM col.TABLE_NAME
WHERE col.COLUMN_NAME NOT IN ("True", "False")
)
THEN "BoolStoredAsVarchar"
WHEN col.DATA_TYPE = "varchar" THEN CONCAT("String(", col.CHARACTER_MAXIMUM_LENGTH, ")")
-- Default if it's not a recognized column type
ELSE col.DATA_TYPE
END),
"),"
) AS alchemy
FROM information_schema.columns AS col
WHERE
col.TABLE_SCHEMA = "my_schema"
AND col.TABLE_NAME = "my_table"
ORDER BY col.ORDINAL_POSITION;
Running this code gives me a permissions error: Error Code: 1142. SELECT command denied to user 'user'#'host' for table 'table_name'. Presumably it's trying to use col.TABLE_NAME as a literal instead of interpreting it.
I've also tried creating a simple stored procedure and making table_name into a variable. However, replacing the FROM clause inside the EXISTS with a variable name gives me a syntax error instead.
Again, it's easy enough to run the query myself to see what's in that column. I'd just like to know if this is possible, and if so, how to do it.
You can't do what you're trying to do in a single query.
The reason is that table names (or any other identifier) must be fixed in the query at the time it is parsed, which is before it has read any values from tables. Thus you can't read the name of a table as a string from information_schema and also read from the table with that name in the same query.
You must read the table name from information_schema and then use that result to format a second query.
This isn't a problem specific to MySQL. It's true of any SQL implementation.
I have a linking table between two tables, ja1_surveyors and ja1_stores. I'm trying to writ a stored procedure that will take three arguments, the third being a json array of store_id.
I've tried this, but I know it's not correct:
/*
========================================================================================
Set the list of stores for a surveyor in a survey. Used with template to create the list
a user sees to edit, copy and delete surveyors in a survey
Accepts three arguments:
arg_srvy_id Survey key
arg_srvr_id Surveyor key
STORE_LIST JSON value holding a list of store keys assigned to this survey/surveyor
STORE_LIST JSON should be in the form: '{store_id:val1},{store_id:val2}' etc.
========================================================================================
*/
DROP PROCEDURE IF EXISTS SURVEYOR_LINK_STORES;
DELIMITER //
CREATE PROCEDURE SURVEYOR_LINK_STORES( IN arg_srvy_id INT(11),IN arg_srvr_id INT(11),IN STORE_LIST JSON)
BEGIN
/* Remove all links for this surveyor to stores for this survey */
DELETE FROM `ja1_storesurveyor`
WHERE `lnk_strsrvr_srvy_id` = arg_srvy_id AND `lnk_strsrvr_srvr_id` = arg_srvr_id;
/* Add links between this survey and surveyor for each key in STORE_LIST */
INSERT INTO `ja1_store_surveyor`
(
`lnk_strsrvr_srvy_id`,
`lnk_strsrvr_srvr_id`,
`lnk_strsrvr_store_id`
)
SELECT
arg_srvy_id,
arg_srvr_id,
STORE_LIST->>`$.store_id`
FROM STORE_LIST;
END
DELIMITER ;
The problem seems to be the select part of the insert statement.
All of the columns are INT(11). And I'm using MySQL version 5.6.41-84.1
What am I missing?
The best way to do this is with JSON_TABLE() but it requires MySQL 8.0.
Edit: When I wrote this answer, your original question did not make it clear you were using an old version of MySQL Server.
CREATE PROCEDURE SURVEYOR_LINK_STORES(
IN arg_arg_srvy_id INT,
IN arg_arg_srvr_id INT,
IN arg_STORE_LIST JSON)
BEGIN
/* Remove all links for this surveyor to stores for this survey */
DELETE FROM `ja1_storesurveyor`
WHERE `lnk_strsrvr_srvy_id` = arg_srvy_id
AND `lnk_strsrvr_srvr_id` = arg_srvr_id;
/* Add links between this survey and surveyor for each key in STORE_LIST */
INSERT INTO `ja1_store_surveyor`
(
`lnk_strsrvr_srvy_id`,
`lnk_strsrvr_srvr_id`,
`lnk_strsrvr_store_id`
)
SELECT
arg_srvy_id,
arg_srvr_id,
j.store_id
FROM JSON_TABLE(
arg_STORE_LIST, '$[*]' COLUMNS(
store_id VARCHAR(...) PATH '$'
)
) AS j;
END
I'm guessing the appropriate data type for store_id is a varchar, but I don't know how long the max length should be.
Re your comment: MySQL 5.6 doesn't have any JSON data type, so your stored procedure won't work as you wrote it (the arg_STORE_LIST argument cannot use the JSON data type).
FYI, MySQL 5.6 past its end-of-life in February 2021, so the version you are using won't get any more bug fixes or security fixes. You should really upgrade, regardless of the JSON issue.
The equivalent code to insert multiple rows in MySQL 5.6 is a lot of work and code to write. I'm not going to write an example for such an old version of MySQL.
You can find other examples on Stack Overflow with the general principle. It involves taking the argument as a VARCHAR, not JSON, and writing a WHILE loop to picking apart substrings of the varchar.
I am using a stored procedure to INSERT into and UPDATE a number of tables. Some of the data is derived from a JSON parameter.
Although I have successfully used json_to_recordset() to extract named data from the JSON parameter, I cannot figure how to use it in an UPDATE statement. Also, I need to use some items of data from the JSON parameter a number of times.
Q: Is there a way to use json_to_recordset() to extract named data to a temporary table to allow me to reuse the data items throughout my stored procedure? Maybe I should SELECT INTO variables within the stored procedure?
Q: Failing that can anyone please provide a simple example of how to update a table using data returned from json_to_recordset(). I must also include data not from the JSON parameter such as now()::timestamp(0).
This is how I have used json_to_recordset() so far:
INSERT INTO myRealTable (
rec_timestamp,
rec_data1,
rec_data2
)
SELECT
now()::timestamp(0),
x.data1,
x.data2
FROM json_to_recordset(json_parameter) x
(
json_data1 int,
json_data2 boolean
);
Thank you.
Using Flink 1.13.1 and a pyFlink and a user-defined table aggregate function (UDTAGG) with Hive tables as source and sinks, I've been encountering an error:
pyflink.util.exceptions.TableException: org.apache.flink.table.api.TableException:
Table sink 'myhive.mydb.flink_tmp_model' doesn't support consuming update changes
which is produced by node PythonGroupAggregate
This is the SQL CREATE TABLE for the sink
table_env.execute_sql(
"""
CREATE TABLE IF NOT EXISTS flink_tmp_model (
run_id STRING,
model_blob BINARY,
roc_auc FLOAT
) PARTITIONED BY (dt STRING) STORED AS parquet TBLPROPERTIES (
'sink.partition-commit.delay'='1 s',
'sink.partition-commit.policy.kind'='success-file'
)
"""
)
What's wrong here?
I imagine you are executing a streaming query that is doing some sort of aggregation that requires updating previously emitted results. The parquet/hive sink does not support this -- once results are written, they are final.
One solution would be to execute the query in batch mode. Another would be to use a sink (or a format) that can handle updates. Or modify the query so that it only produces final results -- e.g., a time-windowed aggregation rather than an unbounded one.
I need to change a Blob field type to a Varchar(128). The data in the field will fit the target field size, it's text, and shouldn't have a problem with UTF-8.
Sample data, all data is in this format:
{"weight":"0","height":"0","width":"0","length":"0"}
I'm using Laravel Make:migrate to handle the conversion.
How should I write the SQL?
I know how to write a Laravel Migration. I also know how to alter a field. But a Blob isn't a text field nor is a Blob normally converted down to a Varchar. I've done some manual UTF-8 conversion of Blobs in the past and know you can mess up your data if you don't do it right. So my concern here is to not mess up my data with a Laravel Migrate. I don't believe the migrate 'down' method can undo corrupted data.
If my data fits the varchar target size and if the data fits the UTF-8 charset, am I good with a straight forward Alter statement:
DB::query("ALTER TABLE DB1.products CHANGE COLUMN dimensions dimensions varchar(128) DEFAULT NULL;");
You shouldn't use sql for this, just create a migration and use change method
Schema::table('table_name', function ($table) {
$table->string('column_name')->change();
});
https://laravel.com/docs/5.7/migrations#modifying-columns
Considering your comment sql would be
ALTER TABLE tablename MODIFY column_name VARCHAR(128);
Run composer install and then composer update in the console and
drop your table from the database and also delete the migration ,then create a new migration using
php artisan make:migration change_col_datatype --table=table_name
and then make changes as below
Schema::table('your_table_name', function ($table) {
$table->string('your_table_name');
});
public function down()
{
Schema::dropIfExists('tablename');
}
The SQL statment:
\DB::statement('ALTER TABLE products CHANGE COLUMN dimensions dimensions VARCHAR(128) DEFAULT NULL;');
Worked fine.