I'm using sqlalchemy declarative models to create my snowflake schema, but since sqlalchemy generates ddl using lower-case identifiers, and snowflake uses upper-case identifiers, alembic can never conclude that a table already exists in snowflake (i.e. "I looked for my_schema.my_table but only found MY_SCHEMA.MY_TABLE").
Is there an event I can use to customize schema/table/column names in DDL and just upper-case them? It seems like this is a gap in the list of sqlalchemy events.
Related
I'm looking for a method or a query to retrieve all tables that are used in a procedure.
I tried information_schema.routinesbut it contains all the definition for a procedure.
Is there any system table that contains the dependency relationship for this ?
Or how can I get table names from the definitions using other language such as Python?
Thanks a lot!!
The current version of MySQL does not implement such a view in INFORMATION_SCHEMA.
MySQL 8.0.13 added I_S.VIEW_TABLE_USAGE, which allows you to look up the tables used by a view. This was done for WorkLog #11864. That WorkLog notes compatibility with PostgreSQL and Microsoft SQL Server.
However, there is no WorkLog I can find for an hypothetical I_S.ROUTINE_TABLE_USAGE table. I checked PostgreSQL, and it has this view: https://www.postgresql.org/docs/current/infoschema-routine-table-usage.html but MySQL does not.
So to get this information automatically, you would have to query the procedure body, and parse it for table references. Not an easy task.
What is the difference between database schema and database structure?
I am learning dbms and these two words are confusing for me. Dint find much information anywhere else.
Database schema and database structure can be used interchangeably, more or less. Most software developers will understand them to mean the same thing.
Schema in particular is used in two different ways:
A schema is like a folder that contains tables. You use statements like CREATE SCHEMA <schemaname>, DROP SCHEMA <schemaname>, USE <schemaname>. In MySQL, SCHEMA and DATABASE are synonyms in most contexts.
A schema is sometimes used to refer to the full definition of all objects in your project, including the schema, tables, indexes, procedures, functions, etc.
I have a scala application that manages multiple MySQL database schemas, which includes modifying (adding, renaming, etc.) tables. The commands are issued over a connection pool that connects to a generic management database in the database server.
Because the application is designed to be cross-database, I use JOOQ to render SQL queries (execution is done via a separate JDBC module).
I experience issues with JOOQs alterTable(...).renameTo(...) DSL - consider the following example:
We have a table "TestTable" in database "TestDatabase". Let's say I want to rename that table simply to "Foo", keeping it in "TestDatabase".
This code:
...
val context = DSL.using(SQLDialect.MYSQL_5_7)
val query = context
.alterTable(table(name("TestDatabase", "TestDatabase")))
.renameTo(name("TestDatabase", "Foo"))
...
Generates: ALTER TABLE `TestDatabase`.`TestTable` RENAME TO `Foo`
However, since the connection pool I'm using is connected to my management database, it just renames the table to "Foo" and moves it to my management database. I would have expected the SQL to be: ALTER TABLE `TestDatabase`.`TestTable` RENAME TO `TestDatabase`.`Foo`. I tried a variety of alternatives to invoke the .renameTo method and convice it to use the fully qualified name, to no avail:
.renameTo(table(name(...) -> same behaviour.
.renameTo("`TestDatabase`.`Foo`") -> Escapes the name with backticks, treats it as one name instead of a qualified name.
I'm wondering if I'm missing something, if this is intended behaviour, or maybe even a bug or design shortcoming of JOOQ.
Is there a way to rename the table using fully qualified names?
Thank you!
That's a bug in jOOQ: https://github.com/jOOQ/jOOQ/issues/8042
Your workaround is close. This doesn't work:
.renameTo("`TestDatabase`.`Foo`")
As you've noticed, behind the scenes, the DSL.name() API is used to wrap the target name, because the renameTo() method doesn't implement the plain SQL templating API. You can, however, explicitly use plain SQL templating by writing as a workaround:
.renameTo(table("`TestDatabase`.`Foo`"))
We've a large and disparate data sources including oracle,db2,mysql. We also need to append few audit columns at the end.
I came across the following Java class org.apache.sqoop.hive.HiveTypes. I am planning to create a simple interpreter that accepts RDBMS DDL and spits out Hive DDL script. Any pointers on how I can achieve this?
Hive QL is more or less similar to normal RDBMS DDL. But there are certain things that it lacks and thats why it does not fully follow ANSI SQL. There is no automated process to convert it.
But you have to try running the SQL queries on Hive and wherever it violates you have to change the query according to hive.
For instance Hive takes only equality condition as join condition which is not the case in RDBMS.
For creating an interpreter yourself you first have to list down the common differences between RDBMS query construct and Hive QL construct. Whenever you encounter a RDBMS construct which according to your list will violate in hive the query gets rebuild as per hive. This replacement logic has to be coded.
I have a dataset with a lot of columns I want to import into a MySQL database, so I want to be able to create tables without specifying the column headers by hand. Rather I want to supply a filename with the column labels in it to (presumably) the MySQL CREATE TABLE command. I'm using standard MySQL Query Browser tools in Ubuntu, but I didn't see in option for this in the create table dialog, nor could I figure out how to write a query to do this from the CREATE TABLE documentation page. But there must be a way...
A CREATE TABLE statement includes more than just column names
Table name*
Column names*
Column data types*
Column constraints, like NOT NULL
Column options, like DEFAULT, character set
Table constraints, like PRIMARY KEY* and FOREIGN KEY
Indexes
Table options, like storage engine, default character set
* mandatory
You can't get all this just from a list of column names. You should write the CREATE TABLE statement yourself.
Re your comment: Many software development frameworks support ways to declare tables without using SQL DDL. E.g. Hibernate uses XML files. YAML is supported by Rails ActiveRecord, PHP Doctrine and Perl's SQLFairy. There are probably other tools that use other format such as JSON, but I don't know one offhand.
But eventually, all these "simplified" interfaces are no less complex to learn as SQL, while failing to represent exactly what SQL does. See also The Law of Leaky Abstractions.
Check out SQLFairy, because that tool might already convert from files to SQL in a way that can help you. And FWIW MySQL Query Browser (or under its current name, MySQL Workbench) can read SQL files. So you probably don't have to copy & paste manually.