Custom rake task for DB: Table not found - mysql

I have a custom rake task, that creates a development DB with data for various situation. The core looks like this:
namespace :db do
task setup_seed: :environment do
Rake::Task['db:drop'].invoke
Rake::Task['db:create'].invoke
Rake::Task['db:schema:load'].invoke
Rake::Task['db:migrate'].invoke
Rake::Task['db:test:prepare'].invoke
Rake::Task['db:seed'].invoke
end
end
Everything runs fine, until db:seed is invoked, because it throws an error that tables does not exist. This is my seed.rb:
puts Rails.env
# => development
puts Article.count
# rake aborted!
# Mysql2::Error: Table 'app_test.articles' doesn't exist: SHOW FULL FIELDS FROM `articles`
# /usr/src/app/db/seeds.rb:2:in `<top (required)>'
# /usr/src/app/Rakefile:16:in `block (2 levels) in <top (required)>'
# Tasks: TOP => db:seed
I noticed two strange things here:
First of all it doesn't find the table articles (or any table). When I halt at the beginning of my seed file and look into the DB (development), the tables are present, but the test db is empty
I've printed the Rails.env and it returns development. However the failure message states that it tries to load the DB app_test.articles and not app_development.articles.
So I think this two issues are related.

It looks like you are running your rake task in development environment, but db:test:prepare runs in test by default. Try invoking:
RAILS_ENV=test rake db:setup_seed

I found the solution myself. The problem was, that the task Rake::Task['db:test:prepare'].invoke changes the environment to test, so putting that line to the bottom of the script and everything works:
namespace :db do
task setup_seed: :environment do
Rake::Task['db:drop'].invoke
Rake::Task['db:create'].invoke
Rake::Task['db:schema:load'].invoke
Rake::Task['db:migrate'].invoke
Rake::Task['db:seed'].invoke
Rake::Task['db:test:prepare'].invoke # this one should be at the bottom
end
end

Related

Mysql2::Error: # MySQL client is not connected — Rails 6.0.2.2 with rspec; Ruby 2.6.4 mac os Catalina + Mojave

I'm getting this extremely strange intermittent problem with MySQL on Rails 6.0.2.2,
RSpec 4.0.0.rc1,
Ruby 2.6.4
on a two separate Macs, one is running macOS Mojave and the other is running MacOS Catalina
Interestingly, the mac running Catalina appears to do it less often, although it still does this. the failure is intermittent, seems to happen immediately after another failure in the suite, but then often continues to show up (across about half of the test suite) on subsequent runs, even if I've fixed the originally failing spec.
1) Template
Failure/Error: _query(sql, #query_options.merge(options))
ActiveRecord::StatementInvalid:
Mysql2::Error: MySQL client is not connected
# /Users/jason/.rvm/gems/ruby-2.6.4/gems/mysql2-0.5.3/lib/mysql2/client.rb:131:in `_query'
# /Users/jason/.rvm/gems/ruby-2.6.4/gems/mysql2-0.5.3/lib/mysql2/client.rb:131:in `block in query'
# /Users/jason/.rvm/gems/ruby-2.6.4/gems/mysql2-0.5.3/lib/mysql2/client.rb:130:in `handle_interrupt'
# /Users/jason/.rvm/gems/ruby-2.6.4/gems/mysql2-0.5.3/lib/mysql2/client.rb:130:in `query'
# /Users/jason/.rvm/gems/ruby-2.6.4/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block (2 levels) in <top (required)>'
# ------------------
# --- Caused by: ---
# Mysql2::Error:
# MySQL client is not connected
# /Users/jason/.rvm/gems/ruby-2.6.4/gems/mysql2-0.5.3/lib/mysql2/client.rb:131:in `_query'
I began having this problem when I set up my personal laptop (Mac OS Catalina) to do work from home. In the process, I updated our environment from Ruby 2.4.7 to 2.6.3 and gems system from 3.0.6 to 3.1.2. And, no doubt, incremental updates to our bundled gems also happened. I have not been able to identify which change resulted in the new, seemingly random and intermittent failures.
Google led me here first, and second to this discussion on a minitest issue:
https://github.com/seattlerb/minitest/issues/744
Then, looking closer at my tests, I noticed that the first test to fail always triggered an async method.
I added this to my config/environments/test.rb file:
config.active_job.queue_adapter = :inline
And all tests are passing again.
I don't know if this constitutes a fix or a work around. I'm not sure if asynchronous processes were working before or if they were previously being handled inline and a default behavior changed. The tests we currently have aren't specifically validating asynchronicity. So, for the moment, this solved my problem.
In my case I had issues while running tests for ActiveJob background jobs, and
adding config.active_job.queue_adapter = :test to config/environments/test.rb worked for me.

Not able to reset foreman admin password using foreman-rake

I am trying to migrate database to mysql from sqlite. Database migrated successfully using foreman-mysql2. After restarting foreman, I wasn't allowed to login with admin/changeme.
I then tried to reset password using the below command
foreman-rake permission-reset password=secret
But I am getting the below error
rake aborted!
Don't know how to build task 'permission:reset' (see --tasks)
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/task_manager.rb:62:in []'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:149:ininvoke_task'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:106:in block (2 levels) in top_level'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:106:ineach'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:106:in block in top_level'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:115:inrun_with_threads'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:100:in top_level'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:78:inblock in run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:176:in standard_exception_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/rake-10.5.0/lib/rake/application.rb:75:inrun'
/usr/bin/rake1.9.1:32:in `'
I am posting answer for my own question
Its not permission....
Its permissions
when we get errors like this, just type --tasks at the end of the command and see rake list, make sure the one you are trying exists in that list..,
correct command:
sudo foreman-rake permissions:reset PASSWORD=secret
On running the above command, its likely that you might get another error
ERF42-7495
"Cannot find user foreman_admin when switching context" or "Cannot find user foreman_api_admin when switching context"
To fix this, just type the below command
foreman-rake db:seed

Unit tests cannot find tables

I'm running unit tests on Rails 2.3 and I keep getting errors along these lines when I run rake test:units:
10) Error:
test_the_truth(BrokerTest):
ActiveRecord::StatementInvalid: Mysql::Error: Table 'shadow_test.users' doesn't exist: DELETE FROM `users`
/Library/Ruby/Gems/1.8/gems/after_commit-1.0.10/lib/after_commit/connection_adapters.rb:14:in `transaction'
I ran rake db:test:load prior to this, and it created the tables. However, when I check mysql after running the unit tests command, the tables are indeed missing. Something in the process of preparing for the unit tests causes the tables to go missing. Here's the debug log from running the test:
WARNING: 'task :t, arg, :needs => [deps]' is deprecated. Please use 'task :t, [args] => [deps]' instead.
at /Users/me/sources/shadow/lib/tasks/turk.rake:53
** Invoke test:units (first_time)
** Invoke db:test:prepare (first_time)
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:abort_if_pending_migrations
** Execute db:test:prepare
** Invoke db:test:load (first_time)
** Invoke db:test:purge (first_time)
** Invoke environment
** Execute db:test:purge
** Execute db:test:load
** Invoke db:schema:load (first_time)
** Invoke environment
** Execute db:schema:load
** Execute test:units
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby -I"lib:test" -I"/Library/Ruby/Gems/1.8/gems/rake-0.9.2.2/lib" "/Library/Ruby/Gems/1.8/gems/rake-0.9.2.2/lib/rake/rake_test_loader.rb" "test/unit/**/*_test.rb"
/Library/Ruby/Gems/1.8/gems/bundler-1.1.3/lib/bundler/runtime.rb:211: warning: Insecure world writable dir /usr/local/git/bin in PATH, mode 040777
NOTE: Gem.source_index is deprecated, use Specification. It will be removed on or after 2011-11-01.
Gem.source_index called from /Library/Ruby/Gems/1.8/gems/rails-2.3.14/lib/rails/gem_dependency.rb:21.
WARNING: using the built-in Timeout class which is known to have issues when used for opening connections. Install the SystemTimer gem if you want to make sure the Redis client will not hang.
/Users/me/sources/shadow/test/unit/../test_helper.rb:36: warning: already initialized constant BEANSTALK
Loaded suite /Library/Ruby/Gems/1.8/gems/rake-0.9.2.2/lib/rake/rake_test_loader
Started
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
Finished in 3.389328 seconds.
Some more background: I'm using Octopus for sharding and that may be somehow interfering with things??
Using the Octopus database sharding gem has a few issues on test case preparation. See GitHub issue: https://github.com/tchandy/octopus/issues/31.
You have a few options, the easiest of which is to disable Octopus for your development environment. Remove "- development" from your shards.yml file.
Otherwise, you can patch the issue by adding task:
task :reconnect_octopus do
if ActiveRecord::Base.connection.is_a?(Octopus::Proxy)
ActiveRecord::Base.connection.initialize_shards(Octopus.config)
end
end
task :'db:test:clone_structure' => :reconnect_octopus
or for Rails 3:
task :reconnect_octopus do
if ActiveRecord::Base.connection.is_a?(Octopus::Proxy)
ActiveRecord::Base.connection.initialize_shards(Octopus.config)
end
end
task :'db:structure:load' => :reconnect_octopus
This should stem off your issue listed above. Happy coding!

db:seed not loading models

I'm trying to seed my database with the standard db/seeds.rb method. This works fine on my development machine, but on my server, I get:
$ sudo rake db:seed RAILS_ENV=production --trace
** Invoke db:seed (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:seed
rake aborted!
uninitialized constant Permission
/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2503:in `const_missing'
/usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.4/lib/active_support/dependencies.rb:92:in `const_missing'
/path/.../.../.../.../db/seeds.rb:4
/usr/local/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/tasks/databases.rake:215:in `load'
/usr/local/lib/ruby/gems/1.8/gems/rails-2.3.4/lib/tasks/databases.rake:215
/usr/local/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
...
But when I check in the console, the model does exist:
$ script/console production
Loading production environment (Rails 2.3.4)
>> Permission
=> Permission(id: integer, ..., created_at: datetime, updated_at: datetime)
What am I forgetting?
In a comment to the blog posted linked to above and here again: http://www.builtfromsource.com/2011/02/09/getting-rake-dbseed-and-config-threadsafe-to-play-nice/
Bruce Adams mentions that one can call:
config.threadsafe! unless $rails_rake_task
to only turn on threadsafe when not running a rake task.
But as the issue is really that threadsafe turns off dependency_loading, you can simply add this line after config.threadsafe! to leave it enabled, but still load your environment as you expect.
config.dependency_loading = true if $rails_rake_task
It can be fixed by disabling threadsafe! in the environment configuration.
I just ran across a good approach to this problem in this article. I'll summarize here so people can (hopefully) find it quicker.
The idea is to turn off threadsafe in the production environment, first by editing config/environments/production.rb:
config.threadsafe! unless ENV['THREADSAFE'] == 'off'
You then set THREADSAFE=off when running rake tasks.

db:schema:load vs db:migrate with capistrano

I have a rails app that I'm moving to another server and I figure I should use db:schema:load to create the mysql database because it's recommended. My problem is that I'm using capistrano to deploy and it seems to be defaulting to rake db:migrate instead. Is there a way to change this or is capistrano using db:migrate for a good reason?
Why to use db:schema:load
I find that my own migrations eventually do some shuffling of data (suppose I combine first_name and last_name columns into a full_name column, for instance). As soon as I do any of this, I start using ActiveRecord to sift through database records, and your models eventually make assumptions about certain columns. My "Person" table, for instance, was later given a "position" column by which people are sorted. Earlier migrations now fail to select data, because the "position" column doesn't exist yet.
How to change the default behavior in Capistrano
In conclusion, I believe deploy:cold should use db:schema:load instead of db:migrate. I solved this problem by changing the middle step which Capistrano performs on a cold deploy. For Capistrano v2.5.9, the default task in the library code looks like this.
namespace :deploy do
...
task :cold do
update
migrate # This step performs `rake db:migrate`.
start
end
...
end
I overrode the task in my deploy.rb as follows.
namespace :deploy do
task :cold do # Overriding the default deploy:cold
update
load_schema # My own step, replacing migrations.
start
end
task :load_schema, :roles => :app do
run "cd #{current_path}; rake db:schema:load"
end
end
Climbing up on the shoulders of Andres Jaan Tack, Adam Spiers, and Kamiel Wanrooij, I've built the following task to overwrite deploy:cold.
task :cold do
transaction do
update
setup_db #replacing migrate in original
start
end
end
task :setup_db, :roles => :app do
raise RuntimeError.new('db:setup aborted!') unless Capistrano::CLI.ui.ask("About to `rake db:setup`. Are you sure to wipe the entire database (anything other than 'yes' aborts):") == 'yes'
run "cd #{current_path}; bundle exec rake db:setup RAILS_ENV=#{rails_env}"
end
My enhancements here are...
wrap it in transaction do, so that Capistrano will do a proper rollback after aborting.
doing db:setup instead of db:schema:load, so that if the database doesn't already exist, it will be created before loading the schema.
That's a great answer from Andres Jaan Tack. I just wanted to add a few comments.
Firstly, here's an improved version of Andres' deploy:load_schema task which includes a warning, and more importantly uses bundle exec and RAILS_ENV to ensure that the environment is set up correctly:
namespace :deploy do
desc 'Load DB schema - CAUTION: rewrites database!'
task :load_schema, :roles => :app do
run "cd #{current_path}; bundle exec rake db:schema:load RAILS_ENV=#{rails_env}"
end
end
I have submitted a feature request to have deploy:load_schema implemented in Capistrano. In that request, I noted that the 'db:schema:load vs. db:migrate' debate has already been covered in the Capistrano discussion group, and there was some reluctance to switch the deploy:cold task to using db:schema:load over db:migrate, since if run unintentionally, the former nukes the entire database whereas the latter would probably complain and bail harmlessly. Nevertheless db:schema:load is technically the better approach, so if the risk of accidental data loss could be mitigated, it would be worth switching.
In Capistrano 3 / Rails 4, the default deploy syntax has changed. You can do this instead:
desc 'Deploy app for first time'
task :cold do
invoke 'deploy:starting'
invoke 'deploy:started'
invoke 'deploy:updating'
invoke 'bundler:install'
invoke 'deploy:db_setup' # This replaces deploy:migrations
invoke 'deploy:compile_assets'
invoke 'deploy:normalize_assets'
invoke 'deploy:publishing'
invoke 'deploy:published'
invoke 'deploy:finishing'
invoke 'deploy:finished'
end
desc 'Setup database'
task :db_setup do
on roles(:db) do
within release_path do
with rails_env: (fetch(:rails_env) || fetch(:stage)) do
execute :rake, 'db:setup' # This creates the database tables AND seeds
end
end
end
end
If you're cautious of invoking the standard deploy tasks manually in the :cold task (as they may change in upcoming version or if you have a custom deploy task), you can also simply call deploy:db_setup before running deploy.
To perform db:schema:load instead of db:setup, you can simply change the rake task, like so:
desc 'Load DB Schema'
task :db_schema_load do
...
execute :rake, 'db:schema:load'
...
end