[n00b alert] I'm probably doing this all wrong... RSpec outputs this failure:
1)... #skipped irrelevant info
Failure/Error: graph.read_db('example1')
Not connected to any DB. #error msg
#./prim.rb:135:in 'read_db'
#./prim_spec.rb:171:in 'block (2 levels) in <top (required)>'
I have set up a MySQL database on the same machine. The program provides an algorithm for computing a graph's minimum spanning tree. Has methods for file I/O, database I/O using ActiveRecord, etc. All WORKS WELL except RSpec tests.
Code (irrelevant parts left out):
prim_spec.rb
describe PGraph, "online" do
before (:all) do
ActiveRecord::Base.establish_connection(
:adapter => "mysql2",
:host => "localhost",
:username => "root",
:password => "xxxxx",
:database => "rubytasks" )
#the exact same statement works perfectly when running the program itself, but fails in RSpec
end
before (:each) do
#graph = PGraph.new
end
it "should correctly retrieve data from database" do
#graph.read_db('example1') #line 171
#business part goes here
end
end
prim.rb
class PGraph
def read_db(graphID)
#the error which is raised (line 135):
raise "Not connected to any DB." unless ActiveRecord::Base.connected?
#reading goes here
end
end
Connection and PGraph manipulation is performed in ui.rb.
So, ummm, what's the correct way to access a real DB (I'm lazy) for testing (or is the problem elsewhere?)? Preferably something simple, since this is just a school assignment. And without messing with Rails or other gems.
PS: I'm using the most recent versions of all gems and server. On Windows 7 x86. Ruby 1.9.2. Thanks.
I'm guessing not everything is loaded properly when you run your rspec tests. Are all classes that setup your database connection loaded properly and with the right parameters when running rspec?
Related
I've seen lots of examples of making Docker containers for Rails applications. Typically they run a rails server and have a CMD that runs migrations/setup then brings up the Rails server.
If I'm spawning 5 of these containers at the same time, how does Rails handle multiple processes trying to initiate the migrations? I can see Rails checking the current schema version in the general query log (it's a MySQL database):
SELECT `schema_migrations`.`version` FROM `schema_migrations`
But I can see a race condition here if this happens at the same time on different Rails instances.
Considering that DDL is not transactional in MySQL and I don't see any locks happening in the general query log while running migrations (other than the per-migration transactions), it would seem that kicking them off in parallel would be a bad idea. In fact if I kick this off three times locally I can see two of the rails instances crashing when trying to create a table because it already exists while the third rails instance completes the migrations happily. If this was a migration that inserted something into the database it would be quite unsafe.
Is it then a better idea to run a single container that runs migrations/setup then spawns (for example) a Unicorn instance which in turn spawns multiple rails workers?
Should I be spawning N rails containers and one 'migration container' that runs the migration then exits?
Is there a better option?
Especially with Rails I don't have any experience, but let's look from a docker and software engineering point of view.
The Docker team advocates, sometimes quite aggressively, that containers are about shipping applications. In this really great statement, Jerome Petazzoni says that it is all about separation of concerns. I feel that this is exactly the point you already figured out.
Running a rails container which starts a migration or setup might be good for initial deployment and probably often required during development. However, when going into production, you really should consider separating the concerns.
Thus I would say have one image, which you use to run N rails container and add a tools/migration/setup whatever container, which you use to do administrative tasks. Have a look what the developers from the official rails image say about this:
It is designed to be used both as a throw away container (mount your source code and start the container to start your app), as well as the base to build other images off of.
When you look at that image there is no setup or migration command. It is totally up to the user how to use it. So when you need to run several containers just go ahead.
From my experience with mysql this works fine. You can run a data-only container to host the data, run a container with the mysql server and finally run a container for administrative tasks like backup and restore. For all three containers you can use the same image. Now you are free to access your database from let's say several Wordpress containers. This means clear separation of concerns. When you use docker-compose it is not that difficult to manage all those containers. Certainly there are already many third party containers and tools to also support you with setting up a complex application consisting of several containers.
Finally, you should decide whether docker and the micro-service architecture is right for your problem. As outlined in this article there are some reasons against. One of the core problems being that it adds a whole new layer of complexity. However, that is the case with many solutions and I guess you are aware of this and willing to except it.
docker run <container name> rake db:migrate
Starts you standard application container but don't run the CMD (rails server), but rake db:migrate
UPDATE: Suggested by Roman, the command would now be:
docker exec <container> rake db:migrate
Having the same pb publishing to a docker swarm, I put here a solution partially grabbed from others.
Rails has already a mechanism to detect concurrent migrations by using a lock on the database. But it triggers ConcurrentException where it should just wait.
One solution is then to have a loop, that whenever a ConcurrentException is thrown, just wait for 5s et then redo the migration.
This is especially important that all containers perform the migration as the migration fails, all containers must fails.
Solution from coffejumper
namespace :db do
namespace :migrate do
desc 'Run db:migrate and monitor ActiveRecord::ConcurrentMigrationError errors'
task monitor_concurrent: :environment do
loop do
puts 'Invoking Migrations'
Rake::Task['db:migrate'].reenable
Rake::Task['db:migrate'].invoke
puts 'Migrations Successful'
break
rescue ActiveRecord::ConcurrentMigrationError
puts 'Migrations Sleeping 5'
sleep(5)
end
end
end
end
And sometimes you have other processes you want to execute also one by one to perform the migration like after_party, cron setup, etc... The solution is then to use the same mechanism as Rails to embed rake tasks around a database lock:
Below, based on Rails 6 code, the migrate_without_lock performs the needed migrations while with_advisory_lock gets database lock (triggering ConcurrentMigrationError if lock cannot be acquired).
module Swarm
class Migration
def migrate
with_advisory_lock { migrate_without_lock }
end
private
def migrate_without_lock
**puts "Database migration"
Rake::Task['db:migrate'].invoke
puts "After_party migration"
Rake::Task['after_party:run'].invoke
...
puts "Migrations successful"**
end
def with_advisory_lock
lock_id = generate_migrator_advisory_lock_id
MyAdvisoryLockBase.establish_connection(ActiveRecord::Base.connection_config) unless MyAdvisoryLockBase.connected?
connection = MDAdvisoryLockBase.connection
got_lock = connection.get_advisory_lock(lock_id)
raise ActiveRecord::ConcurrentMigrationError unless got_lock
yield
ensure
if got_lock && !connection.release_advisory_lock(lock_id)
raise ActiveRecord::ConcurrentMigrationError.new(
ActiveRecord::ConcurrentMigrationError::RELEASE_LOCK_FAILED_MESSAGE
)
end
end
MIGRATOR_SALT = 1942351734
def generate_migrator_advisory_lock_id
db_name_hash = Zlib.crc32(ActiveRecord::Base.connection_config[:database])
MIGRATOR_SALT * db_name_hash
end
end
# based on rails 6.1 AdvisoryLockBase
class MyAdvisoryLockBase < ActiveRecord::AdvisoryLockBase # :nodoc:
self.connection_specification_name = "MDAdvisoryLockBase"
end
end
Then as before, do a loop to wait
namespace :swarm do
desc 'Run migrations tasks after acquisition of lock on database'
task migrate: :environment do
result = 1
(1..10).each do |i|
**Swarm::Migration.new.migrate**
puts "Attempt #{i} sucessfully terminated"
result = 0
break
rescue ActiveRecord::ConcurrentMigrationError
seconds = rand(3..10)
puts "Attempt #{i} another migration is running => sleeping #{seconds}s"
sleep(seconds)
rescue => e
puts e
e.backtrace.each { |m| puts m }
break
end
exit(result)
end
end
Then in your startup script just launch the rake tasks
set -e
bundle exec rails swarm:migrate
exec bundle exec rails server -b "0.0.0.0"
At the end, as your migrations tasks are run by all containers, they must have a mechanism to do nothing when it's already done. (like does db:migrate)
Using this solution, the order in which Swarm launches containers doesn't matter anymore AND if something goes wrong, all containers know the problem :-)
For single container id:
docker exec -it <container ID> bundle exec rails db:migrate
for multiple we can repeat the process for different container, if there number in 1000 the need to script to execute.
I have a configuration where, in addition to the local postgresql database, my Rails app also accesses a remote AWS database. My problem is that, even in tests that don't involve the remote database, the app establishes a connection to the remote database every time, so my tests run sloooooowly.
Is there a clean way to disable access to the remote database server for rspec tests that don't need it? (And enable it for those tests that do need it?)
The best I can think of is partition my rspec tests into two separate parts -- those that don't need to access the remote db and those that do -- and use environment variables to enable or disable parts of config/database.yaml accordingly.
To make this clear, my config/database.yaml file contains (in part):
# file: config/database.yaml
# Define connections to the external database
remote:
adapter: mysql
database: remote
username: <%= ENV['PRODUCTION_DB_USERNAME'] || 'root' %>
password: <%= ENV['PRODUCTION_DB_PASSWORD'] || '' %>
host: awsserver-production-mysql.abcdef1234567890.us-west-2.rds.amazonaws.com
port: 3306
test:
adapter: postgresql
encoding: unicode
database: MyApp_test
pool: 5
username: <%= ENV['POSTGRESQL_DB_USERNAME'] || 'MyApp' %>
password: <%= ENV['POSTGRESQL_DB_PASSWORD'] || '' %>
(NOTE: In case you're wondering, using mocks won't help: the remote db connection is established even before the tests start to run. And vcr only intercepts HTTP connections -- database connections use a different mechanism.)
update
I've found examples of how to dynamically establish a connection:
def connect_to_myapp_database
ActiveRecord::Base.establish_connection(:adapter => "mysql",
:database => 'myapp',
:username => ENV['MYAPP_DB_USERNAME'],
:password => ENV['MYAPP_DB_PASSWORD'],
:host => 'mayapp-mysql.abcdefg123456.us-west-2.rds.amazonaws.com',
:port => 3306,
)
end
which works fine -- I can use this to connect to the external database just for those tests that need it. But this begs the question: how do I disconnect from the external database once I've done this?
TL;DR: Use nulldb and a test-aware parent class
Use nulldb when you're testing and the real db otherwise. Here's how:
First, include this in your Gemfile:
group :development, :test do
gem 'activerecord-nulldb-adapter', :git => 'git://github.com/nulldb/nulldb.git'
end
and then do the usual bundle install
Define a base class for all models that are backed in the external database:
class ExternalModel < ActiveRecord::Base
if Rails.app.test?
establish_connection(:adapter => :nulldb)
else
establish_connection(:myapp)
end
def readonly?; true; end
end
Then all the external models inherit from ExternalModel (I should have done this from the start):
class ExternalUser < ExternalModel
...
end
When run in a test environment, it won't try to connect to the external table. Of course, attempts to access an instance of ExternalUser will fail, but you can selectively establish a connection with the external database during integration testing, or stub or mock references to the external model otherwise.
Most importantly, all my tests run really fast now.
Is there a ruby gem or such for MySQL connection pooling that isn't part of rails? I simply have a ruby script (again, I don't do anything with rails).
Seamless Database Pool is supposed to work "with any ActiveRecord application", and ActiveRecord is easy to use without Rails. I've used ActiveRecord in a plain Ruby app, and it was just a matter of configuring the logger and the database connection, something like
ActiveRecord::Base.logger = App.logger
dbconfig = YAML::load(File.open("#{APP_ROOT}/config/database.yml"))
ActiveRecord::Base.establish_connection(dbconfig[ENV["APP_ENV"]])
I haven't used Seamless Database Pool outside of Rails, but I couldn't find any connection poolers aimed at plain Ruby apps after a quick search so it might be your best bet.
If you use JRuby (with ActiveRecord and JDBC adapter) you can configure a J2EE container to handle DB connection pool and pass the pool in database.yml via jndi similar to here (example with oracle).
Also, some slides for MYSQL pools via ActiveRecord
ActiveRecord::Base.establish_connection(
:adapter => 'mysql',
:username => 'root',
:password => '123456',
:database => 'database',
:pool => 5 # <- CONN POOL
)
You could wrap the connection_pool gem around a couple single connections.
Examples from the doc (redis):
Example usage with block (faster):
#pool = ConnectionPool.new { Redis.new }
#pool.with do |redis|
redis.lpop('my-list') if redis.llen('my-list') > 0
end
Example usage replacing an existing connection (slower):
$redis = ConnectionPool.wrap { Redis.new }
def do_work
$redis.lpop('my-list') if $redis.llen('my-list') > 0
end
How do we monitor that the connectivity between the rails app and the database is established. will rails try to re-connect the connection with mysql if it closes?
AFAIK yes, you could still check periodically a special controller you would create that displays whether the DB is up or down (using ActiveRecord::Base.connected? for instance).
Edit Tue Mar 30:
I believe you could instead write a Metal and periodically check the result from it (using a rake task or whatever monitoring tool you're used to):
class ConnectivityChecker
def self.call(env)
if ActiveRecord::Base.connected?
[200, { 'Content-Type' => 'text/html' }, ["Connected."]]
else
# Leave it to Rails to deal with the request.
[500, { 'Content-Type' => 'text/html' }, ['Database is not reachable.']]
end
end
end
I am on leopard. It comes with Ruby 1.8 & Sqlite3 pre-installed. I have updated ruby to 1.9.1 & added Mysql. Here's the problem. I cannot get the path to correctly point to ruby 1.9.1. I tried to update the sym-link to no avail. I am able to get into Mysql from the terminal but I cannot connect to the server through Ruby because Sqlite3 is the default. I changed the database in my apps config file but it still doesn't work. Something is really screwing this up. I want to unistall every version of Ruby, Rails, All Gems, Mysql, Sqlite3, etc & roll all of what I want on my own. Where can I find the commands through the command line to do this? Can I just send these files to the trash manually as I find them on /usr/local/....? I am really frustrated at this point! please help.
Re-installing those packages will still not guarantee that its going to work. I would recommend go through the logs and see if you pick up some something obvious. There are lots of debugging techniques available for a Rails app, for starters see here
Here's a small ruby snippet to see if the connection to MySQL works fine, give it a try if you see the MySQL server version being printed on your terminal then you know the problem is somewhere else, do not forget to change the credentials.
#!/usr/bin/ruby -w
require "mysql"
begin
# connect to the MySQL server
dbh = Mysql.real_connect("localhost", "testuser", "testpass", "test")
# get server version string and display it
puts "Server version: " + dbh.get_server_info
rescue Mysql::Error => e
puts "Error code: #{e.errno}"
puts "Error message: #{e.error}"
puts "Error SQLSTATE: #{e.sqlstate}" if e.respond_to?("sqlstate")
ensure
# disconnect from server
dbh.close if dbh
end
Also if possible please provide some more details about the environment you are using, like
Apache + Rails + Mongrel or
Apache + Rails + Passenger etc
a snippet of your app/config/database.yml etc
If you are frustrated, take a break , relax, have a coffee :-) and then start over again....working in a frustrated state of mind is definitely not going to help solve problems.
HTH