Conditional drop of tables in slick - mysql

Any idea how to do a conditional drop in Slick 3.0, to prevent An exception or error caused a run to abort: Unknown table 'MY_TABLE' if for some reason it doesn't exist?
def clear = {
val operations = DBIO.seq(
myTable.schema.drop,
// other table definitions
...
)
db.run(operations)
}

I went down the MTable route, but at least in Postgres it's a big hassle.
Try
def qDropSchema = sqlu"""drop table if exists your-table-name;""";
Watch out for case-sensitivity issues with the table name. I ran into odd problems with the postgres there - don't know about mysql.

Let me try to answer your question, I think you can first check the availability of the table using MTable then drop it if it exists. More or less like below:
import scala.slick.jdbc.meta._
if (MTable.getTables("table_name").list().isEmpty) {
//do something here..
}

I did this:
val personQuery = TableQuery[PersonTable]
val addressQuery = TableQuery[AddressTable]
...
val setupAction = DBIO.seq(
sqlu"SET FOREIGN_KEY_CHECKS = 0",
sqlu"DROP TABLE IF EXISTS #${personQuery.baseTableRow.tableName}",
sqlu"DROP TABLE IF EXISTS #${addressQuery.baseTableRow.tableName}",
sqlu"SET FOREIGN_KEY_CHECKS = 1",
)
val setupFuture = db.run(setupAction)
Note how you need to use #${} not ${} otherwise slick will fire off something like:
DROP TABLE IF EXISTS 'PERSON'
Which won't work

I am currently using the Slick framework in its 3.2.0 version.
The solution I am giving may apply to earlier version of the framework however but I did not verify this point.
If the only problem is to drop the table if it exists without throwing exception you can use the combinators of the Actions for this.
I have a series of test for which I run the create/populate/drop statement for each test on H2 in memory database.
I suppose you have two tables Canal and SubCanal (SubCanal has a foreign key on Canal so that you would like to drop it first if it exists) for wich you already have declared TableQuery variables such as:
lazy val canals = TableQuery[CanalTable]
lazy val subcanals = TableQuery[SubCanalTable]
// we don't put SubCanals to check if no exeption is produced and then
// add it for further testing.
lazy val ddl = canals.schema // ++ subcanals.schema
...I provided helper methods as follows:
def create: DBIO[Unit] = ddl.create
def drop: DBIO[Unit] = ddl.drop
def popCanal = canals ++= Seq(
Canal("Chat"),
Canal("Web"),
Canal("Mail"))
The above is just creating the action but what is cool is that Slick will attempt to drop the SubCanal table and the Canal table but will encapsulate the exception in the Try[...]. So this will run smoothly:
val db = Database.forConfig("yourBaseConfig")
val res = db.run(drop)
And this will run also:
val db = Database.forConfig("yourBaseConfig")
val res1 = db.run(
create >>
popCanal >>
canals.result
)
.... some interesting computation ...
val res2 = db.run(drop)
Note: The SubCanal scheme is still commented so has never been performed for the moment and the drop is however applied and fail to this table but does not raise the exeption.
More on combining actions (combinator):
DBIO Action Doc (3.2.0)
This book is free (but you may give some money) Essential Slick

Related

SQLAlchemy ORM function works in main script but fails in Pytest

Using Postgres, SQLAlchemy 1.4 ORM, Python 3.7, Pytest.
I have a script in myproject/src/db.py and the tests for it are located in myproject/tests/.
In db.py I have a function to drop any given table, it works as expected:
async def delete_table(self, table_name):
table = self.meta.tables[table_name]
async with self.engine.begin() as conn:
await conn.run_sync(Base.metadata.drop_all(sync_engine, [table], checkfirst=True))
It gets called like:
asyncio.run(db.delete_table('user'))
In conftest.py I have a fixture like this:
#pytest.fixture(scope='function')
def test_drop_table():
def get_delete_tables(table_name):
return asyncio.run(DB.delete_table(table_name))
return get_delete_tables
In test.py I run the test like this:
#pytest.mark.parametrize('function_input, expected',
[('user', 'user'),
pytest.param('intentional failure', 'intentional failure',
marks=pytest.mark.xfail(reason='testing incorrect table name fails'))])
def test_drop_table(test_drop_table, function_input, expected):
# Drop the table
test_drop_table(function_input)
# Check that it no longer exists
with pytest.raises(KeyError) as error_info:
test_table_commits(function_input, expected)
raise KeyError
assert error_info.type is KeyError
When I run this test I get this error:
self = <postgresdb.PostgresDb object at 0x7f4bbd87cc18>, table_name = 'user'
async def delete_table(self, table_name):
> table = self.meta.tables[table_name]
E KeyError: 'user'
I verified that this table can be dropped in the main script. I then recommit the table, verify it is present, and then try to drop it with the test but will continually receive a KeyError that the table is not present even though when checking the database that table is actually present.
I'm not sure what to test or adjust in the code to get Pytest working with this function. I appreciate any help!
I think for the first time it deletes the table named user, but the second input in pytest.mark.parametrize is also the name user, so it may be throwing error. If you need to test 2 different scenarios, it's better to have 2 different test functions. By doing this, you can have all your code under with pytest.raises(KeyError) as error_info in the 2nd test function.

Django bulk update setting each to different values? [duplicate]

I'd like to update a table with Django - something like this in raw SQL:
update tbl_name set name = 'foo' where name = 'bar'
My first result is something like this - but that's nasty, isn't it?
list = ModelClass.objects.filter(name = 'bar')
for obj in list:
obj.name = 'foo'
obj.save()
Is there a more elegant way?
Update:
Django 2.2 version now has a bulk_update.
Old answer:
Refer to the following django documentation section
Updating multiple objects at once
In short you should be able to use:
ModelClass.objects.filter(name='bar').update(name="foo")
You can also use F objects to do things like incrementing rows:
from django.db.models import F
Entry.objects.all().update(n_pingbacks=F('n_pingbacks') + 1)
See the documentation.
However, note that:
This won't use ModelClass.save method (so if you have some logic inside it won't be triggered).
No django signals will be emitted.
You can't perform an .update() on a sliced QuerySet, it must be on an original QuerySet so you'll need to lean on the .filter() and .exclude() methods.
Consider using django-bulk-update found here on GitHub.
Install: pip install django-bulk-update
Implement: (code taken directly from projects ReadMe file)
from bulk_update.helper import bulk_update
random_names = ['Walter', 'The Dude', 'Donny', 'Jesus']
people = Person.objects.all()
for person in people:
r = random.randrange(4)
person.name = random_names[r]
bulk_update(people) # updates all columns using the default db
Update: As Marc points out in the comments this is not suitable for updating thousands of rows at once. Though it is suitable for smaller batches 10's to 100's. The size of the batch that is right for you depends on your CPU and query complexity. This tool is more like a wheel barrow than a dump truck.
Django 2.2 version now has a bulk_update method (release notes).
https://docs.djangoproject.com/en/stable/ref/models/querysets/#bulk-update
Example:
# get a pk: record dictionary of existing records
updates = YourModel.objects.filter(...).in_bulk()
....
# do something with the updates dict
....
if hasattr(YourModel.objects, 'bulk_update') and updates:
# Use the new method
YourModel.objects.bulk_update(updates.values(), [list the fields to update], batch_size=100)
else:
# The old & slow way
with transaction.atomic():
for obj in updates.values():
obj.save(update_fields=[list the fields to update])
If you want to set the same value on a collection of rows, you can use the update() method combined with any query term to update all rows in one query:
some_list = ModelClass.objects.filter(some condition).values('id')
ModelClass.objects.filter(pk__in=some_list).update(foo=bar)
If you want to update a collection of rows with different values depending on some condition, you can in best case batch the updates according to values. Let's say you have 1000 rows where you want to set a column to one of X values, then you could prepare the batches beforehand and then only run X update-queries (each essentially having the form of the first example above) + the initial SELECT-query.
If every row requires a unique value there is no way to avoid one query per update. Perhaps look into other architectures like CQRS/Event sourcing if you need performance in this latter case.
Here is a useful content which i found in internet regarding the above question
https://www.sankalpjonna.com/learn-django/running-a-bulk-update-with-django
The inefficient way
model_qs= ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
obj.name = 'foo'
obj.save()
The efficient way
ModelClass.objects.filter(name = 'bar').update(name="foo") # for single value 'foo' or add loop
Using bulk_update
update_list = []
model_qs= ModelClass.objects.filter(name = 'bar')
for model_obj in model_qs:
model_obj.name = "foo" # Or what ever the value is for simplicty im providing foo only
update_list.append(model_obj)
ModelClass.objects.bulk_update(update_list,['name'])
Using an atomic transaction
from django.db import transaction
with transaction.atomic():
model_qs = ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
ModelClass.objects.filter(name = 'bar').update(name="foo")
Any Up Votes ? Thanks in advance : Thank you for keep an attention ;)
To update with same value we can simply use this
ModelClass.objects.filter(name = 'bar').update(name='foo')
To update with different values
ob_list = ModelClass.objects.filter(name = 'bar')
obj_to_be_update = []
for obj in obj_list:
obj.name = "Dear "+obj.name
obj_to_be_update.append(obj)
ModelClass.objects.bulk_update(obj_to_be_update, ['name'], batch_size=1000)
It won't trigger save signal every time instead we keep all the objects to be updated on the list and trigger update signal at once.
IT returns number of objects are updated in table.
update_counts = ModelClass.objects.filter(name='bar').update(name="foo")
You can refer this link to get more information on bulk update and create.
Bulk update and Create

Use multiple DB / Schema in a single DBIO in Slick

Let's say If I have 2 Schema below.Both of them are in a same MySQL server.
master
base
The problem is I can't use more than 2 schema actions in a single Database run.
If I execute such plain sql queries via sql, It works without any problem.
def fooAction(name: String) = {
sql"""
SELECT AGE FROM MASTER.FOO_TABLE WHERE NAME = $name
""".as[String].head
}
def barAction(id: String) = {
sql"""
SELECT BAZ FROM BASE.BAR_TABLE WHERE ID = $id
""".as[String].head
}
def execute = {
//It doesn't matter which Db I use in here But Let's say this baseDb is pointing to BASE schema.
baseDb.run(for{
foo <- fooAction("sample")
bar <- barAction("sample")
} yield foo + bar)
}
But the case of code blow doesn't
class FooTableDAO #Inject() (#NamedDatabase("master") protected val dbConfigProvider: DatabaseConfigProvider) extends HasDatabaseConfigProvider[JdbcProfile] {
import dbConfig.driver.api._
val table = TableQuery[FooTable]
def fooAction(name: String) = table.filter{_.name == name}.map{_.age}.result.head
}
class BarTableDAO #Inject() (#NamedDatabase("base") protected val dbConfigProvider: DatabaseConfigProvider) extends HasDatabaseConfigProvider[JdbcProfile] {
import dbConfig.driver.api._
val table = TableQuery[BarTable]
def fooAction(id: String) = table.filter{_.id == id}.map{_.baz}.result.head
}
def execute = {
//com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'BASE.FOO_TABLE' doesn't exist
baseDb.run(for{
foo <- fooTableDAO.fooAction("sample")
bar <- barTableDAO.barAction("sample")
} yield foo + bar)
}
Since baseDb is pointing to BASE schema, It tries to find FOO_TABLE in MASTER schema. All What I want slick to do is use different schema for each query but I couldn't find the way.
Currently I do DBIO.from(db.run(**)) if another schema action is needed in a for-comprehension of DBIO action or execute each actions via run and wrap them with EitherT which is scala library named cats's monad transformer for Either to keep using for-comprehension.
Is there any way to handle more than 2 schemas in a single DBIO Action except using plain text query?
Thanks in advance.
I think (though I am not an MySQL expert) you mean schema, not a database. At least this is what I see from your SQL samples.
Can't you just use schema attribute in your Slick table mappings? Here you have complete answer for using different schemas: https://stackoverflow.com/a/41090987/2239369
Here is the relevant piece of code:
class StudentTable(tag: Tag) extends Table[Student](tag, _schemaName = Option("database2"), "STUDENT") {
...
}
(notice _schemaName attribute).
With this in mind answer to this part of the question:
Is there any way to handle more than 2 schemas in a single DBIO Action except using plain text query?
is: Yes you can.

How to call Stored Procedures and defined functions in MySQL with Slick 3.0

I have defined in my db something like this
CREATE FUNCTION fun_totalInvestorsFor(issuer varchar(30)) RETURNS INT
NOT DETERMINISTIC
BEGIN
RETURN (SELECT COUNT(DISTINCT LOYAL3_SHARED_HOLDER_ID)
FROM stocks_x_hldr
WHERE STOCK_TICKER_SIMBOL = issuer AND
QUANT_PURCHASES > QUANT_SALES);
END;
Now I have received an answer from Stefan Zeiger (Slick lead) redirecting me here: User defined functions in Slick
I have tried (having the following object in scope):
lazy val db = Database.forURL("jdbc:mysql://localhost:3306/mydb",
driver = "com.mysql.jdbc.Driver", user = "dev", password = "root")
val totalInvestorsFor = SimpleFunction.unary[String, Int]("fun_totalInvestorsFor")
totalInvestorsFor("APPLE") should be (23)
Result: Rep(slick.lifted.SimpleFunction$$anon$2#13fd2ccd fun_totalInvestorsFor, false) was not equal to 23
I have also tried while having an application.conf in src/main/resources like this:
tsql = {
driver = "slick.driver.MySQLDriver$"
db {
connectionPool = disabled
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost/mydb"
}
}
Then in my code with #StaticDatabaseConfig("file:src/main/resources/application.conf#tsql")
tsql"select fun_totalInvestorsFor('APPLE')" should be (23)
Result: Error:(24, 9) Cannot load #StaticDatabaseConfig("file:src/main/resources/application.conf#tsql"): No configuration setting found for key 'tsql'
tsql"select fun_totalInvestorsFor('APPLE')" should be (23)
^
I am also planning to call stored procedures that return one tuple of three values, via sql"call myProc(v1).as[(Int, Int, Int)]
Any ideas?
EDIT: When making
sql""""SELECT COUNT(DISTINCT LOYAL3_SHARED_HOLDER_ID)
FROM stocks_x_hldr
WHERE STOCK_TICKER_SIMBOL = issuer AND
QUANT_PURCHASES > QUANT_SALES""".as[(Int)]
results in SqlStreamingAction[Vector[Int], Int, Effect] instead of the suggested DBIO[Int] (from what I infer) suggested by the documentation
I've been running into exactly the same problem for the past week. After some extensive research (see my post here, I'll be adding a complete description of what I've done as a solution), I decided it can't be done in Slick... not strictly speaking.
But, I'm resistant to adding pure JDBC or Anorm into our solution stack, so I did find an "acceptable" fix, IMHO.
The solution is to get the session object from Slick, and then use common JDBC to manage the stored function / stored procedure calls. At that point you can use any third party library that makes it easier... although in my case I wrote my own function to set up the call and return a result set.
val db = Database.forDataSource(DB.getDataSource)
var response: Option[GPInviteResponse] = None
db.withSession {
implicit session => {
// Set up your call here... (See my other post for a more detailed
// answer with an example:
// procedure is eg., "{?=call myfunction(?,?,?,?)}"
val cs = session.conn.prepareCall(procedure.toString)
// Set up your in and out parameters here
// eg. cs.setLong(index, value)
val result = cs.execute()
val rc = result.head.asInstanceOf[Int]
response = rc match {
// Package up the response to the caller
}
}
}
db.close()
I know that's pretty terse, but as I said, see the other thread for a more complete posting. I'm putting it together right now and will post the answer shortly.

Slick 2.1 MySqlSyntaxErrorException on DDL generation

In my tests, I'm trying to setup a test database based on the schema defined in slick, which in turn is generated with the slick code generator based on an existing database. To do so, I'm extending WithApplication and redefining the around method like so:
abstract class WithDbData extends WithApplication {
def ds = DB.getDataSource("test")
def slickDb = Database.forDataSource(ds)
override def around[T: AsResult](t: => T): Result = super.around {
setup
val result = AsResult.effectively(t)
teardown
result
}
def setup = {
slickDb.withSession { implicit session =>
ddl.create
}
}
def teardown = {
slickDb.withSession { implicit session =>
ddl.drop
}
}
}
But when I run the tests, I'm getting a
MySqlSyntaxErrorException : Column length too big for column 'text' (max = 21845); use BLOB or TEXT instead (null:-2).
I'm using MySql for development and for testing, and part of the schema generated by the code generator is like so:
/** Database column text DBType(TEXT), Length(65535,true) */
val text: Column[String] = column[String]("text", O.Length(65535,varying=true))
It seems that the ddl generator is trying to create the text column as a varchar or something like that, instead of as text, as it was originally intended, because in the original database that column is of type text.
In the autogenerated slick data model I have as profile = scala.slick.driver.MySQLDriver, I've also imported scala.slick.driver.MySQLDriver.simple._ in my test class so I shouldn't have any problem because of mixing drivers.
I'm using play 2.3, slick 2.1.0, codegen 2.1.0 and play-slick 0.8.0.
I'd appreciate any light on this matter.
It's a bug in the code generator. github.com/slick/slick/issues/975
You can work around it by customizing the code generator. See http://slick.typesafe.com/doc/2.1.0/code-generation.html
One easy way to do it is override def dbType = true . You loose cross-vendor portability but get the exact types. You may have to filter Length out of def options, not sure.
Also discussed here: https://github.com/slick/slick/issues/982
You may change the column manually:
Create the table as standar varchar(255):
val text: Column[String] = column[String]("text", O.Length(255,varying=true))
After that you can change the database:
alter table your_table change `text` `text` text;
And restore the values:
val text: Column[String] = column[String]("text", O.Length(65535,varying=true))
In the other hand: you can add a custom field definition:
def recipe = column[Option[String]]("text_column", O.DBType("TEXT"))
Note: I changed the column name to text_column because text is a reserved word in MySQL.