I use spork and guard to speed up the test suite in my Rails 3.2 application. However the tests in guard behave strange after adding a new migration: they act as they don't know about the changes in schema, even though I run rake db:migrate before. They start to behave correctly when I stop guard and run rake spec. I suppose that it behaves this way, because spork/guard doesn't update schema before tests. Is it possible to configure spork to update schema on #prefork and guard to reload spork on schema changes?
You need to run rake db:test:prepare to apply migrations for testing environment.
Theoretically the following default Spork setup code should reload schema each time a test is run. Do you have it?
ActiveRecord::Schema.verbose = false
silence_stream STDOUT do
load Rails.root.join('db', 'schema.rb') # use db agnostic schema by default
load Rails.root.join('db', 'seeds.rb')
end
Related
Question:
What can cause flyway to not auto-detect migrations from the default path, and prevent resolution of migrations from a custom location, during startup only?
Given the following:
io.micronaut.flyway:micronaut-flyway uses flyway 6.4.4
Flyway, when run at application startup by micronaut, is unable to auto-detect migrations
Flyway, when run during bean initialization (i.e. in the constructor of the controller bean), is able to auto-detect migrations
Flyway is able to pick up and apply the migrations during startup during integration-testing. This gives me confidence that it is configured correctly. I can break it in expected ways by messing with the config / file location.
Migration file is certainly on the classpath during runtime on prod at the expected location, as evidenced by runtime-logs.
Context
I want to setup flyway migrations for my Kotlin-Micronaut-GoogleCloudFunction. As described in the docs, I have my migrations under src/main/resources/db/migration, named like V1__create_xyz_table.sql.
I verified that the migration is on the classpath at runtime, by logging it in the function body:
val fileContent = FunctionController::class.java.getResource("/db/migration/V1__create_xyz_table.sql").readText()
println(fileContent) // "create table xyz(id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY)"
This works, and logs the contents of the file to stdout as expected.
My integration tests run fine. Migrations are automatically detected and applied to the mysql-testcontainer instance. Data is written to and read from the dockerized DB.
However, when I start the application locally, or deploy it, the application warns me:
No migrations found. Are your locations set up correctly?
Unsurprisingly, triggering the function results in errors like "Table xyz does not exist".
Besides the actual db-credentials, my test and production setup share the following config:
# application.yml
datasources:
mysql:
url: <url>
username: <user>
password: <pw>
flyway:
datasources:
mysql:
enabled: true
Other things I have tried:
Using a Java-Based migration (same result)
Using the custom locations config (same result)
What "works":
When I autowire the datasource into the function controller, and apply the migrations inside the constructor it works: Successfully validated 1 migration.
init {
Flyway.configure().dataSource(mysqlDS).load().migrate()
}
This confirms, that all the necessary files are present and discoverable by flyway. Why would this not work during application startup?
I attached a debugger and found that different ClassLoaders are used to discover the resources:
During startup: AppClassLoader
During function execution: FunctionClassLoader
I was having the same issue. Debugging, I realized that the method that triggers the flyway migration wasn't running. This method lives inside a Micronaut BeanCreatedEventListener which listens on the creation of DataSource type beans.
What left me scratching my head was that the DataSource type bean was created successfully, which I confirmed on runtime by fetching it from the application context. So why wasn't the even listener triggering?
This is because the bean was being created before the event listener was even initialized. Why was this happening? Because I had another custom even listener in my app that injected a Jdbi bean. The jdbi bean subsequently injected the DataSource bean. This means that my custom event listener was injecting the DataSource bean, so it was impossible for it to be initialized before the bean was created.
I suggest setting a breakpoint in this method to check if it's being triggered. If it's not, it's possible the cause of your issue was similar to mine.
When i select custom setting and select sql and enter all fields
Connection test failed: Cannot find module\AppData\Local\Temp\strapi9a2b8146f759\node_modules\strapi-connector-bookshelf\lib\utils\connectivity.js'
I know It's old but I just want to reply for future people who have the same issue.
Try to use yarn create strapi-app --quickstart or using npx npx create-strapi-starter --quickstart, and It'll use the default template
and then when It finish getting dependencies and finish setting up your project go to a file called database.js and add your database set of infos.
Got this info from issue opened in github : Here
I've been following the 5 min how to for setting up an htap databse with tidb_tispark and everything works until I get to the section Launch TiSpark. My first issue occurs when executing the line:
docker-compose exec tispark-master /opt/spark-2.1.1-bin-hadoop2.7/bin/spark-shell
But I got around that by modifying the spark version to the version I found inside the container:
docker-compose exec tispark-master /opt/spark-2.3.3-bin-hadoop2.7/bin/spark-shell
My second issue occurs when executing the three line block:
import org.apache.spark.sql.TiContext
val ti = new TiContext(spark)
ti.tidbMapDatabase("TPCH_001")
When I run the last statement I get the following output
scala> ti.tidbMapDatabase("TPCH_001")
2019-07-11 16:14:32 WARN General:96 - Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark/jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-2.3.3-bin-hadoop2.7/jars/datanucleus-core-3.2.10.jar."
2019-07-11 16:14:32 WARN General:96 - Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark/jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-2.3.3-bin-hadoop2.7/jars/datanucleus-api-jdo-3.2.6.jar."
2019-07-11 16:14:32 WARN General:96 - Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark/jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-2.3.3-bin-hadoop2.7/jars/datanucleus-rdbms-3.2.9.jar."
2019-07-11 16:14:36 WARN ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectException
This doesn't prevent me from running the query:
spark.sql("select * from nation").show(30);
But when I follow the further steps of the tutorial to modify the db from MySQL, the changes are not reflected immediately in Spark. Furthermore, at some point in the future (I believe > 5 minutes later), the row that was modified stops showing up in Spark SQL queries.
I'm rather new to this kind of setup and don't really know how to debug this issue. Searches for the warnings I received weren't illuminating.
I don't know if it's helpful but when I connect MySQL this is the server version I get:
Server version: 5.7.25-TiDB-v3.0.0-rc.1-309-g8c20289c7 MySQL Community Server (Apache License 2.0)
I'm one of the main dev of TiSpark. Sorry for your bad experience with it.
Due to my docker problem, I cannot directly reproduce your issue but it seems you hit one of the bug fixed recently.
https://github.com/pingcap/tispark/pull/862/files
The tutorial document is not quite up-to-date and points to an older version. That's why it didn't work with spark 2.1.1 as in tutorial. We will update it ASAP.
Newer version of TiSpark doesn't use tidbMapDatabase anymore but hooks with catalog directly instead. Method tidbMapDatabase remains for backward compatibility. Unfortunately, the tidbMapDatabase had a bug（when we ported it from older version） that it retrieves timestamp for query only once you call the function. That causes TiSpark always uses old timestamp to do snapshot reading and newer data would never be seen by it.
In newer version of TiSpark (TiSpark 2.0+ with Spark 2.3+), databases and tables are directly hooked into catalog services and you can directly call
spark.sql("use TPCH_001").show
spark.sql("select * from nation").show
This should give you fresh data.
So try restart your Spark driver, just try the two lines of code above and see if it works.
Let me know if this fix your problem. On the other hand, we will check our docker image to make sure if it contains the fix already.
If things still get wrong, would you please help to run below code and let us know the version of TiSpark.
spark.sql("select ti_version()").show
Again, sorry for causing you trouble and thanks for trying.
EDIT
To address your comment:
The warning is due to spark itself will try to locate the database in its native catalog first and this will cause a Failed to get warning. But the failover process will delegate the search to tispark and then behave correctly. So this warning can be ignored. It's recommended that add below lines to your log4j.properties in conf folder of your spark.
log4j.logger.org.apache.hadoop.hive.metastore.ObjectStore=ERROR
We will polish the docker tutorial image soon. Thank you so much for trying.
I have some simple rspec tests that check if a particular page's title is 'xxxx'.
I added some before_filter's to my application_controller which makes db calls.
Should I be mocking these out somehow?
The tests are failing saying "mysql2::error table myapp_test.articles doesn't exist.
That sounds more like you need to run rake db:test:prepare or rake db:migrate db:test:prepare to me. Do you have something set up so your rspecs fail when they hit the database intentionally?
I have a rails app that I'm moving to another server and I figure I should use db:schema:load to create the mysql database because it's recommended. My problem is that I'm using capistrano to deploy and it seems to be defaulting to rake db:migrate instead. Is there a way to change this or is capistrano using db:migrate for a good reason?
Why to use db:schema:load
I find that my own migrations eventually do some shuffling of data (suppose I combine first_name and last_name columns into a full_name column, for instance). As soon as I do any of this, I start using ActiveRecord to sift through database records, and your models eventually make assumptions about certain columns. My "Person" table, for instance, was later given a "position" column by which people are sorted. Earlier migrations now fail to select data, because the "position" column doesn't exist yet.
How to change the default behavior in Capistrano
In conclusion, I believe deploy:cold should use db:schema:load instead of db:migrate. I solved this problem by changing the middle step which Capistrano performs on a cold deploy. For Capistrano v2.5.9, the default task in the library code looks like this.
namespace :deploy do
...
task :cold do
update
migrate # This step performs `rake db:migrate`.
start
end
...
end
I overrode the task in my deploy.rb as follows.
namespace :deploy do
task :cold do # Overriding the default deploy:cold
update
load_schema # My own step, replacing migrations.
start
end
task :load_schema, :roles => :app do
run "cd #{current_path}; rake db:schema:load"
end
end
Climbing up on the shoulders of Andres Jaan Tack, Adam Spiers, and Kamiel Wanrooij, I've built the following task to overwrite deploy:cold.
task :cold do
transaction do
update
setup_db #replacing migrate in original
start
end
end
task :setup_db, :roles => :app do
raise RuntimeError.new('db:setup aborted!') unless Capistrano::CLI.ui.ask("About to `rake db:setup`. Are you sure to wipe the entire database (anything other than 'yes' aborts):") == 'yes'
run "cd #{current_path}; bundle exec rake db:setup RAILS_ENV=#{rails_env}"
end
My enhancements here are...
wrap it in transaction do, so that Capistrano will do a proper rollback after aborting.
doing db:setup instead of db:schema:load, so that if the database doesn't already exist, it will be created before loading the schema.
That's a great answer from Andres Jaan Tack. I just wanted to add a few comments.
Firstly, here's an improved version of Andres' deploy:load_schema task which includes a warning, and more importantly uses bundle exec and RAILS_ENV to ensure that the environment is set up correctly:
namespace :deploy do
desc 'Load DB schema - CAUTION: rewrites database!'
task :load_schema, :roles => :app do
run "cd #{current_path}; bundle exec rake db:schema:load RAILS_ENV=#{rails_env}"
end
end
I have submitted a feature request to have deploy:load_schema implemented in Capistrano. In that request, I noted that the 'db:schema:load vs. db:migrate' debate has already been covered in the Capistrano discussion group, and there was some reluctance to switch the deploy:cold task to using db:schema:load over db:migrate, since if run unintentionally, the former nukes the entire database whereas the latter would probably complain and bail harmlessly. Nevertheless db:schema:load is technically the better approach, so if the risk of accidental data loss could be mitigated, it would be worth switching.
In Capistrano 3 / Rails 4, the default deploy syntax has changed. You can do this instead:
desc 'Deploy app for first time'
task :cold do
invoke 'deploy:starting'
invoke 'deploy:started'
invoke 'deploy:updating'
invoke 'bundler:install'
invoke 'deploy:db_setup' # This replaces deploy:migrations
invoke 'deploy:compile_assets'
invoke 'deploy:normalize_assets'
invoke 'deploy:publishing'
invoke 'deploy:published'
invoke 'deploy:finishing'
invoke 'deploy:finished'
end
desc 'Setup database'
task :db_setup do
on roles(:db) do
within release_path do
with rails_env: (fetch(:rails_env) || fetch(:stage)) do
execute :rake, 'db:setup' # This creates the database tables AND seeds
end
end
end
end
If you're cautious of invoking the standard deploy tasks manually in the :cold task (as they may change in upcoming version or if you have a custom deploy task), you can also simply call deploy:db_setup before running deploy.
To perform db:schema:load instead of db:setup, you can simply change the rake task, like so:
desc 'Load DB Schema'
task :db_schema_load do
...
execute :rake, 'db:schema:load'
...
end