Have you heard of SQL Closures or any library that implements them ?
They allow to execute this script in SQL command window (or put it into SP):
exec closure,"
rec{select db=name from sys.databases where name like 'corp_'},{
use |db|
rec{select tbl=name from sys.tables where name like 'user_'},{
for{col},{Created,Modified},{
def_col {
|tbl|.|col| datetime not null default(getdate()) ix
}
}
def_col {|tbl|.deleted datetime ix}
}
}
"
This script will make sure that Created not null, Modified not null and Deleted indexed columns exist in all tables with prefix user_ in all databases with prefix corp_.
def_col will create new column or alter existing column to match desired definition. It will also create/recreate non-unique ascending index for each of these columns.
def_col will drop and recreate dependencies as needed (constraints, indexes, foreign keys, schema bound views and functions).
rec and for and def_col will catch errors and log them into error table or raise immediately depending on context options for easy debugging and tracking of errors during script execution should they happen.
As you can see, the script can be executed many times without failures, it's just second time it will not change anything.
Is there a more readable, supportable and compact way to achieve the same functionality in MS-SQL ?
If yes - please post example in your answer.
Is more readable, supportable and compact way available in MySql, Oracle or other major flavors of SQL language ?
I do not see any reason you could not create simple SQL Server Groups in SSMS, and register your servers against those groups and run your DDL from there. You could also do it with SQLCMD, or Powershell.
Related
I’d like to be able to create MySQL Document Store Collections via simple SQL DDL statements rather than using the X-Protocol clients.
Is there any way to do so?
Edit: I’ll try and clarify the question.
Collections are tables using JSON datatypes and functions. That much is clear.
I would like know how I can create a Collection without using the X-Protocol calls and make sure that the aforementioned collection is picked up as an actual Collection.
Judging from MySQL workbench, collection tables have a _id blob PK with an expression, a doc JSON column and a few other elements I do not recall at the moment (might be indexes, etc).
I have no means to tell via the Workbench whatever additional schema/metadata information is required for a table to be considered a Document Store Collection, or if the mere presence of an _id and doc columns are enough.
I hope this clears things up.
All "x-api" instructions are directly mapped to sql syntax. When you e.g. run db.createCollection('my_collection'), MySQL will literally just execute
CREATE TABLE `my_collection` (
`doc` json DEFAULT NULL,
`_id` varbinary(32) GENERATED ALWAYS AS
(json_unquote(json_extract(`doc`,_utf8mb4'$._id'))) STORED NOT NULL,
`_json_schema` json GENERATED ALWAYS AS (_utf8mb4'{"type":"object"}') VIRTUAL,
PRIMARY KEY (`_id`),
CONSTRAINT `$val_strict` CHECK (json_schema_valid(`_json_schema`,`doc`))
NOT ENFORCED
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
You can run the corresponding sql statements yourself if you follow that format.
The doc and _id (with their type and the given expression) are required, the _json_schema is optional, the check too (and only added since MySQL 8.0.17). Since MySQL 8, no additional columns are allowed, except generated columns that use JSON_EXTRACT on doc and which are supposed to be used in an index, see below (although they don't actually have to be used in an index).
Any table that looks like that - doc and _id with their correct type/expression and no other columns except an optional _json_schema and generated JSON_EXTRACT(doc,-columns - will be found with getCollections().
To add an index, the corresponding syntax for
my_collection.createIndex("age", {fields: [{field: "$.age", type: "int"}]})
would be
ALTER TABLE `test`.`my_collection` ADD COLUMN `$ix_i_somename` int
GENERATED ALWAYS AS (JSON_EXTRACT(doc, '$.age')) VIRTUAL,
ADD INDEX `age` (`$ix_i_somename`)
Obviously,
db.dropCollection('my_collection')
simply translates to
DROP TABLE `my_collection`
Similarly, all CRUD operations on documents have a corresponding sql DML syntax (that will actually be executed when you use them via x-api).
Is there any way to detect when an ALTER TABLE statement is executed in MySQL? For example, if the following statement were executed on some_table, is there any way to detect that the column name changed from column_name_a to column_name_b and log it in another table in the DB?
ALTER TABLE `some_table`
CHANGE COLUMN `column_name_a` `column_name_b` VARCHAR(255) NULL DEFAULT NULL;
Thanks.
To my knowledge it is unfortunately not possible to put triggers on the INFORMATION_SCHEMA tables, since they are strictly spoken views and triggers can't be made to work on views. If triggers would be possible on the INFORMATION SCHEMA, then you could have a trigger on updates of the INFORMATION_SCHEMA.COLUMNS table to identify name changes.
However, what you can do is one of the following things:
option 1) Maintain a real table with all column names. Then create a function that checks for a discrepancy between the INFORMATION_SCHEMA.COLUMNS table abd your table. If there is one, you know the name has changed. You need to copy over the new name to your column name table and do whatever else you wanted to do upon name change.
The function to check for discrepancies then must be run periodically via the mysql scheduler in order to detect name changes as quickly as possible. Note that this is not a real time solution. There will be a lag between the ÀLTER TABLE command and its detection. If this is unacceptable in your scenario you need to go with
option 2) Do not call ÀLTER TABLE directly, but wrap it in a function. Within this function you can also call other functions to achieve what you need to achieve. If may be worth while to formulate the needed steps in a higher programming language that you use to drive your application. If this is not possible, you will be limited to the possibilities that are offered in functions/procedures in the mysql environment.
Sorry to not have a simpler way of doing this for you.
i want to know if it's possible to drop a table fragment that is not letting me perform a restore with the NDB_RESTORE tool.
When i run the restore, it throws the following error:
Create table db_died_maestro/def/NDB$FKM_3194_0_mae_tipo_reg_evaluacion failed: 721: Schema object with given name already exists
Restore: Failed to restore table: db_died_maestro/def/NDB$FKM_3194_0_mae_tipo_reg_evaluacion ... Exiting
NDBT_ProgramExit: 1 - Failed
I have already drop the DB_DIED_MAESTRO database previous to run the restore, but this fragment is not being dropped along with the database.
I have check that the fragment is in the database catalog using this querys:
*select * from ndbinfo.operations_per_fragment
where fq_name like 'db_died_maestro%'*
query result
And this query:
*select * from ndbinfo.memory_per_fragment
where fq_name like '%FKM_3194_0_mae_tipo_reg_evaluacion'*
query 2 result
This fragment was created on a previous run of the NDB_RESTORE tool. Please help me.
The table is a foreign key 'mock' table (indicated by the name NDB$FKM prefix).
Foreign key mock tables are created transiently in some cases to implement the foreign_key_checks = 0 feature of MySQL. This feature requires storage engines to support unordered creation of tables with partially defined foreign key constraints which can be abritrarily enabled (without revalidation) at a later time.
Foreign key mock tables are normally entirely managed by the Ndb storage engine component of MySQL, and so should not be visible unless there has been a failure or bug of some kind.
If you can share information about activities occurring before this problem then that would help us understand how this happened and whether it can be avoided.
As a workaround it should be possible for you to use the ndb_drop_table utility to drop this table, before re-attempting the failing restore. You may have to escape the $ in the name passed as a command line argument from a shell. Probably you should check for any other NDB$FKM tables in a similar situation.
Is it possible to use any sort of logic in MySQL without using any procedures? My web hosting does not let me create any procedures so I'm looking for a workaround.
The type of thing I want to do is only add an item to a table if it doesn't already exist. Or add a column to a table if it's not already there. There are some operations that can be done such as CREATE TABLE IF NOT EXISTS and so on, but some operations I require do not have such luxuries :(
I realised late on that my lovely procs won't work and so I tried writing IF/ELSE logic as top-level queries, but for MySQL, IF ELSE blocks only seem to work inside functions/procs and not at the global scope.
Any workarounds greatfully received - I've already asked the hosting to grant me privileges to create procedures but no reply as yet...
I suppose you don't have access to the INFORMATION_SCHEMA either. You can possibly find solutions but it would be better, in my oninion, to:
Change your hosting provider. Seriously. Pay more - if needed - for a MySQL instance that you can configure to your needs. You only have a crippled DBMS if you are not allowed to create procedures and functions.
Posible workarounds for the specific task: You want to add a column if it doesn't exist.
1) Just ALTER TABLE and add the column. If it already exists, you'll get an error. You can catch that error, in your application.
2) (If you have no access to the INFORMATION_SCHEMA) maintain a version of the schema, for your database.
The best solution that I can think of would be to use an additional language with SQL. For example, you can run a query for a specific record, and based on the response that you get, you can conditionally run an INSERT statement.
For inserting a table if it doesn't exist, try using the SHOW TABLES statement and testing whether or not a name exists in the result set.
MySQL supports INSERT IGNORE. and INSERT ... ON DUPLICATE KEY UPDATE.
The following will insert a new row, but only if there is no existing row with id=10. (This assumes that id is defined as a unique or primary key).
INSERT IGNORE INTO my_table (id, col1, col2) values (10, "abc", "def");
The following will insert a new row, but if there is an existing row with id=10 (again, assuming id is unique or primary), the existing row will be updated to hold the new values, instead of inserting a new row.
INSERT INTO my_table (id, col1, col2) values (10, "abc", "def")
ON DUPLICATE KEY UPDATE col1=VALUES(col1), col2=VALUES(col2)
Also, CREATE TABLE supports the IF NOT EXISTS modifier. So you can do something like:
CREATE TABLE IF NOT EXISTS my_table ...
There are many other similar options and modifiers available in MySQL. Check the docs for more.
Originally I created a big script to create or update the database schema, to make it easier to deploy database changes from my local machine to the server.
My script was doing a lot of "if table 'abc' exists and it doesn't have a FK constraint called 'blah'" then create an FK constraint called 'blah' on table 'abc'... and so on.
I now realise it's not actually necessary to check whether a table has a certain column or constraint etc, because I can just maintain a schema-versioning system, and query the DB schema-version when my app starts, or when I navigate to a certain page.
e.g. let's say I want to add a new column to a table. It works like this:
Add a new migration script to the app code, containing the SQL required to add the column to the existing table
Increment the app's schema-version by 1
On app startup, the app queries the DB for the DB's schema-version
If DB schema-version < app schema-version, execute the SQL migration scripts between the two schema-versions, and then update the DB schema-version to be the same as the app
e.g. if the DB's schema-version is 5 and the app version is 8, the app will apply migration scripts 5-6, 6-7 and 7-8 to the DB. These can just be run without having to check anything on the DB side.
The app is therefore solely responsible for updating the DB schema and there's no need for me to ever have to execute schema change scripts on the local or remote DB.
I think it's a better system than the one I was trying to implement for my question.
I am trying to restore a DB using an SQL script, but things foreign key constraints get in the way
I am taking a MySQL DB and bringing it over to PostgreSQL.
Since the MySQL create table syntax ended up being quite different, I took another PostgreSQL DB with the same schema, but different data and restored the schema only, from that.
In other words, I now have a database with tables, constraints, sequences and all that shnaz but no data inside.
So, it's is time to restore data.
I take a backup of the MySQL DB with phpMyAdmin (data only) as an SQL script (pgAdmin does not seem to accept zip or gzip files for some reason) and run the SQL script.
Now, this is where the problems start to happen, it's only natural, I am going from MySQL to PostgreSQL, so syntax errors are bound to happen.
But, there are other non syntax related problems to, like this one:
ERROR: insert or update on table "_account" violates foreign key constraint "fk_1_account"
DETAIL: Key (accountid)=(2) is not present in table "_entity".
So, yeah, basically, a foreign constraint exists, the query is trying to insert data into the _account table, but the corresponding data has not been inserted into the _entity table yet.
How do I get around that? Is there a way to make pgAdmin3/PostgreSQL disable ALL OF the constraints, insert the data, and then re-enable the constraints?
A syntax related error I encountered, was this one:
INSERT INTO _accounttype_seq (id) VALUES (11);
The PostgreSQL equivalent of that statement (if I am correct) is
ALTER SEQUENCE _accounttype_seq INCREMENT BY 11;
But, it's a bit of a pain to run through the whole script and change all 200+ Sequence insert statements. So, I am being lazy here, but is there an easier way to deal with the sequences as well?
Or, do you guys have any suggestions for a different set of tools to make this easier?
Thanks for your time, have a good day.
Do not try to get around the foreign key constraints. That is the way to make sure the data is bad.
First look at the constraints and make sure you are inserting to the tables in the correct order. If _entity is parent of "_account, then it should be populated first.
Next you need to have the script move any failing records to an exception table. Then you can look at them and see what the data integrity issues is and if you need to throw the records away permanently or try to figure out what the missing parent value should be. If it is critical data such as orders where the customer no longer exists (possible in any system that didn't have correct fks to begin with) and you must keep the record and cannot determine what the parent value should have been, you can create an 'Unknown" record in the customer table and assign all bad orders to that customer id.
And manually changing the alter sequences shouldn't take long even if it is boring. There wil be plently of other things you need to handle manually in a conversion of this type.
I would try to find a data import tool for PostgreSQL - I live in SQL server world where I would use SSIS but you need the equivalent of SSIS for the PostgreSQL world.
Aparently the foreign keys weren't actually enforced in MySQL (maybe because of using MyISAM) or the generated SQL just does it in the wrong order.
If it's "only" the wrong order, I see two possible solutions:
edit the generated script and either move all FK definitions to the end of the script
Edit the definition of each FK constraint and set them all to initially deferred. Then run the script as one single transaction with only on commit at the very end.
Edit (because this is too much to be put as a comment)
Using SET CONSTRAINTS ALL DEFERRED will only work if the constraints have been created with the option DEFERRABLE.
To run everything in one single transaction, you have to make sure you have turned autocommit off. Then simply run the INSERTs and at the very end issue a COMMIT. A ; will only commit if you have autocommit on.
If you want to be independent of the autocommit setting, then start your script with [BEGIN][1] and make sure there is only a single COMMIT at the very end.
BEGIN DEFERRABLE
INSERT INTO table_one ... ;
INSERT INTO table_two ... ;
.....
COMMIT;