All my previous projects used DatabaseCleaner, so I'm used to starting with an empty DB and creating test data within each test with FactoryGirl.
Currently, I'm working on a project that has a test database with many records. It is an sql file that all developers must import in their local test environments. The same DB is imported in the continuous integration server. I feel like having less control over the test data makes the testing process harder.
Some features allow their tests to focus on specific data, such as records that are associated to a certain user. In those cases, the preexisting data is irrelevant. Other features such as a report that displays projects of all clients do not allow me to "ignore" the preexisting data.
Is there any way to ignore the test DB contents in some tests (emulate an empty DB and create my own test data without actually deleting all rows in the test DB)? Maybe have two databases (both in the same MySQL server) and being able to switch between them (e.g., some tests use one DB, other tests use the other DB)?
Any other recommendations on how deal with to this scenario?
Thank you.
I would recommend preserving your test_database, and the 'test' environment as your 'clean' state. Then you could setup a separate database that you initially seed as your 'dirty' database. A before hook in your rails_helper file could also be setup with something like this:
RSpec.configure do |config|
config.before :each, type: :feature do |example|
if ENV['TEST_DIRTY'] || example.metadata[:test_dirty]
ActiveRecord::Base.establish_connection(
{
:adapter => 'mysql2',
:database => 'test_dirty',
:host => '127.0.0.1',
:username => 'root',
:password => 'password'
}
)
end
end
end
Your database.yml file will need configurations added for your 'dirty' database. But I think the key here is keeping your clean and dirty states separate. Cheers!
I have found that adding the following configuration to spec/rails_helper.rb will run all DB operations inside tests or before(:each) blocks as transactions, which are rolled back after each test is finished. That means we can do something like before(:each) { MyModel.delete_all }, create our own test data, run our assertions (which will only see the data we created) and after the end of the test, all preexisting data will still be in the DB because the deletion will be rolled back.
RSpec.configure do |config|
config.use_transactional_fixtures = true
end
Related
I have a master server which imports data into a MySQL database on a slave server, and I'm trying to keep this running until the slave server reaches near capacity. I've done this by setting up cron jobs in a chef recipe, which runs 20 different ruby files every 15 mins. Everything works fine for about an hour, then the database stops taking entries. Can anybody shed some light on this please?
Things I've done:
1)logged into database and added entries. This is fine.
2)checked cron log to see that jobs have run. This is fine.
Here is one of my import files, but I don't think there is any problem with it
require 'rubygems'
require 'mysql2'
require 'open-uri'
#my = Mysql2::Client.new(hostname,username,password,database)
con = Mysql2::Client.new(:hostname => 'ipaddress',
:username => 'username',
:password => 'password',
:database => 'database')
con.query("CREATE TABLE IF NOT EXISTS words(id INT PRIMARY KEY AUTO_INCREMENT,value VARCHAR(50));")
open('http://samplewebsite.txt').each_line do |word|
word.split(" ").each do |str|
outstring = str.gsub(/[^a-zA-Z0-9\-]/,"")
con.query("INSERT INTO words(value)
VALUES('"+outstring+"');")
end
end
puts "Done!"
con.close
Thanks in advance
Edit: After thinking about this for a little while I realized my mistake (and from people's answers below), I was hitting the same website too many times from my server, so I was getting blocked out. I've modified my recipes to go to different sites so they're more spread out and things are working fine now. Thanks to anyone wjo answered!
I'm fairly new to RSpec and have been trying to create some tests for my website, on which a user can post a reservation to the website, which is then saved to our database. I've been trying, using Rspec and Capybara, to simulate a user posting a reservation to the website. We have an existing test database, and at the end of the Rspec test want the new reservation to be written to the database, and not removed at the end of the Rspec test.
One of two things happens when we run the code: either it "works" but the new reservation can't be found in the database, or we get this error:
Failure/Error: Unable to find matching line from backtrace
ActiveRecord::StatementInvalid:
Mysql2::Error: This connection is in use by: #<Thread:0x007fb421fd6218 sleep>: SELECT `users`.* FROM `users` WHERE `users`.`id` = 6 ORDER BY `users`.`id` ASC LIMIT 1
# ./app/controllers/application_controller.rb:95:in `pass_login_status_to_js'
# ./app/middleware/search_suggestions.rb:12:in `call'
Why would this be happening? I realize that Capybara isn't generally meant to be making permanent changes to a database; is there a different program/gem you recommend?
I currently have config.use_transactional_fixtures = false, and also have added the following on the recommendation of a few websites:
class ActiveRecord::Base
mattr_accessor :shared_connection
##shared_connection = nil
def self.connection
##shared_connection || retrieve_connection
end
end
ActiveRecord::Base.shared_connection = ActiveRecord::Base.connection
To reiterate, I do want Capybara to be writing to my database (we use SQL). What can I do differently? Does it have something to do with database cleaner?
Yes, it has everything to do with database_cleaner. If you have it setup properly, it will clean your database between scenarios, to keep the tests isolated.
There are a few ways to do what you want:
You can explicitly tell database_cleaner not to clean certain tables between scenarios:
DatabaseCleaner.strategy = :transaction, {except: [:countries, :states]}
DatabaseCleaner.clean_with(:truncation, {except: [:countries, :states]})
You can add your code to a before(:each) or before(:all) block
You can add your data to one or many fixtures
There are only a few cases where you should share data between scenarios (ie. countries, states tables, which are good candidates for #3)
In any other case, I advise against sharing data between scenarios.
I'd like to dump my databases to a file.
Certain website hosts don't allow remote or command line access, so I have to do this using a series of queries.
All of the related questions say "use mysqldump" which is a great tool but I don't have command line access to this database.
I'd like CREATE and INSERT commands to be created at the same time - basically, the same performance as mysqldump. Is SELECT INTO OUTFILE the right road to travel, or is there something else I'm overlooking - or maybe it's not possible?
Use mysqldump-php a pure-PHP solution to replicate the function of the mysqldump executable for basic to med complexity use cases - I understand you may not have remote CLI and/or mysql direct access, but so long as you can execute via an HTTP request on a httpd on the host this will work:
So you should be able to just run the following purely PHP script straight from a secure-directory in /www/ and have an output file written there and grab it with a wget.
mysqldump-php - Pure PHP mysqldump on GitHub
PHP example:
<?php
require('database_connection.php');
require('mysql-dump.php')
$dumpSettings = array(
'include-tables' => array('table1', 'table2'),
'exclude-tables' => array('table3', 'table4'),
'compress' => CompressMethod::GZIP, /* CompressMethod::[GZIP, BZIP2, NONE] */
'no-data' => false,
'add-drop-table' => false,
'single-transaction' => true,
'lock-tables' => false,
'add-locks' => true,
'extended-insert' => true
);
$dump = new MySQLDump('database','database_user','database_pass','localhost', $dumpSettings);
$dump->start('forum_dump.sql.gz');
?>
With your hands tied by your host, you may have to take a rather extreme approach. Using any scripting option your host provides, you can achieve this with just a little difficulty. You can create a secure web page or strait text dump link known only to you and sufficiently secured to prevent all unauthorized access. The script to build the page/text contents could be written to follow these steps:
For each database you want to back up:
Step 1: Run SHOW TABLES.
Step 2: For each table name returned by the above query, run SHOW CREATE TABLE to get the create statement that you could run on another server to recreate the table and output the results to the web page. You may have to prepend "DROP TABLE X IF EXISTS;" before each create statement generated by the results of these queryies (!not in your query input!).
Step 3: For each table name returned from step 1 again, run a SELECT * query and capture full results. You will need to apply a bulk transformation to this query result before outputing to screen to convert each line into an INSERT INTO tblX statement and output the final transformed results to the web page/text file download.
The final web page/text download would have an output of all create statements with "drop table if exists" safeguards, and insert statements. Save the output to your own machine as a ".sql" file, and execute on any backup host as needed.
I'm sorry you have to go through with this. Note that preserving mysql user accounts that you need is something else entirely.
Use / Install PhpMySQLAdmin on your web server and click export. Many web hosts already offer you this as a service pre-configured, and it's easy to install if you don't already have it (pure php): http://www.phpmyadmin.net/
This allows you to export your database(s), as well as perform other otherwise tedious database operations very quickly and easily -- and it works for older versions of PHP < 5.3 (unlike the Mysqldump.php offered as another answer here).
I am aware that the question states 'using query' but I believe the point here is that any means necessary is sought when shell access is not available -- that is how I landed on this page, and PhpMyAdmin saved me!
please an example for use multiple data bases for my app in rails, i need create and save only one table internal and external, no all database so any idea or example explicit please??
i use mysql, the replication i dont like because is for all database and i need use only one table external.
tnks!
If I'm interpreting your question correctly, you want to know how to use a different database for one or more models in your app.
You can do this quite simply in Rails. First, you'll have to create another entry in your config/database.yml file:
production:
....
development:
...
test:
...
# Our external database
external:
adapter: mysql
host: some_host
username: some_username
password: some_password
database: some_db
Then, in your model, simply tell Rails that for this particular model, use a different connection:
class MyModel < ActiveRecord::Base
establish_connection :external
end
This will send any queries for MyModel to the external database, while all other models will use the production/development database as usual.
Please note that when you're using two different databases together, MySQL itself will not let you perform certain functions, such as joins. Otherwise, you should be fine with this approach.
well my solution is the next with help vonconrad post.
database.yml
like of vonoconrad post
MyModelconn.rb
class MyModelconnection < ActiveRecord::Base
# No corresponding table in the DB.
self.abstract_class = true
# Open a connection to the appropriate database depending
# on what RAILS_ENV is set to.
establish_connection(:connyml)
end
class User < ActiveRecord::Base
end
In users_controller.rb where i need insert into in external table with data values where i save in my Active Record of usres internal so an external swicht conection next line:
MyModelconn.connection.execute('INSERT INTO 'users' (fileld1,filed2)VALUES('#{lcfield1}','#{lcfield2}')')
and BINGO! workiT! perfect!
So dont need you again swicht back AR internal.
I am trying to switch between 2 mysql servers at runtime. I do not need to maintain both connections alive all the time.
This is what I am doing
from django.conf import settings
from django.db import connection
from django.contrib.auth.models import User
connection.close()
setattr(settings, 'DATABASE_HOST', 'mysql1.com')
list1 = User.objects.all()
connection.close()
setattr(settings, 'DATABASE_HOST', 'mysql2.com')
list2 = User.objects.all()
I have the following settings.py:
DATABASE_HOST = '' # localhost
DATABASE_NAME = test
...
The database name is the same on all servers and only the content of each tables differ.
I should get list1 != list2 as the users are different on both servers.
The issue is that I always get the list of users from the default database defined in settings.py (which is running on localhost) instead of the one from mysql 1 server and then from mysql 2 server.
Any idea what I am doing wrong here?
Laurent
My guess, from the information, would be a potential error in your set DATABASE_HOST lines (in yor pseudo code above). read: "setattr(settings..."
Other than that, I'm not sure how you've configured your database to switch based on your criteria, as you've not explained this. If you are doing it by model, it may be worth considering how Django knows this, or even using external connections (manually loading the database driver and running commands by hand prior to the render stage), and using the main.
I'd query the whole approach, but mostly because I'm not sure how you're actually differentiating the two databases, or why. Could you provide a bit more information on how you're doing this? I assume the variables you're pulling in dot-points 2 and 5 above are different. I don't need the values, I'm just making sure you've not used the old code duplication and forgotten to edit it (we've all been there).
Note: I'd post this as a comment if I could, but I think the solution may be in how you're pulling the variables. Finally, you could try adding the database name (just the server IP or whatever) to the output, if you're in 'dev'/debug (offline/non-production) mode, to check if it's actually making it to the second server.
For reference, the Django documentation explicitly states you shouldn't do this -- Altering settings at runetime.
There is a lot of talk within the Django community about the ORM supporting multiple connections/databases at once. There's a lot of good reference info out there on it. Check out this blog post: Easy Multi-Database Support for Django and this Django wiki page Multiple Database Support.
In the blog post, Eric Florenzano does something like this in his settings.py file:
DATABASES = dict(
primary = dict(
DATABASE_NAME=DATABASE_NAME,
# ...
),
secondary = dict(
DATABASE_NAME='secondary.db',
# ...
),
)