I'm trying to run an initialize script to help me with retrieving data from my config.yml file.
config.yml currently looks like this:
production: &default
log_server: log_server
deploy_to: /path/to/deploy
unicorn:
port: 8080
uid: user
gid: group
development: &dev
<<: *default
deploy_to:path/to/deploy
unicorn:
port: 80
uid: unicorn
gid: unicorngroup
Here's the config.rb file that I have so far:
require 'yaml'
require 'erb'
config = YAML.load(ERB.new(File.read("/Users/NguyenC/gomoto_server/config/api_settings.yml")).result)
If I 'puts config', the output looks like this:
{"production"=>{"log_server"=>"log_server", "deploy_to"=>"path/to/deploy", "unicorn"=>{"port"=>8080, "uid"=>"user", "gid"=>"group"}}, "development"=>{"log_server"=>"log_server", "deploy_to"=>"path/to/deploy", "unicorn"=>{"port"=>80, "uid"=>"unicorn", "gid"=>"unicorn group"}}
How then would I call the variables if I wanted to only get certain ones from either prod or development? My goal is to be able to call any variable depending on the Env that I use.
Related
I am trying to run migrations through Sequelize in Node JS on Google Cloud Run connecting to a MySQL Google Cloud SQL database. I followed
https://stackoverflow.com/a/58441728/4487248 to get the Google Cloud proxy setup. Given this log setting up the proxy connection to the database seems to have worked:
Step #2 - "migrate": Already have image (with digest): gcr.io/cloud-builders/yarn
Step #2 - "migrate": 2021/10/02 14:19:58 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
Step #2 - "migrate": 2021/10/02 14:19:58 Listening on /workspace/<MY-INSTANCE-NAME> for <MY-INSTANCE-NAME>
Step #2 - "migrate": 2021/10/02 14:19:58 Ready for new connections
Step #2 - "migrate": 2021/10/02 14:19:58 Generated RSA key in 74.706896ms
However, when I try to run migrations with yarn knex migrate:latest or ./node_modules/.bin/sequelize db:migrate I run into:
getaddrinfo ENOTFOUND /workspace/<MY-INSTANCE-NAME>
This seems to imply that the host could not be found.
Output / Logs
My cloudbuild.yaml (composed of https://stackoverflow.com/a/52366671/4487248 & https://stackoverflow.com/a/58441728/4487248):
steps:
# Install Node.js dependencies
- id: yarn-install
name: gcr.io/cloud-builders/yarn
waitFor: ["-"]
# Install Cloud SQL proxy
- id: proxy-install
name: gcr.io/cloud-builders/yarn
entrypoint: sh
args:
- "-c"
- "wget https://storage.googleapis.com/cloudsql-proxy/v1.25.0/cloud_sql_proxy.linux.amd64 -O /workspace/cloud_sql_proxy && chmod +x /workspace/cloud_sql_proxy"
waitFor: ["-"]
- id: migrate
name: gcr.io/cloud-builders/yarn
entrypoint: sh
args:
- "-c"
- "(/workspace/cloud_sql_proxy -dir=/workspace -instances=<MY-INSTANCE-NAME> & sleep 2) && ./node_modules/.bin/sequelize db:migrate"
timeout: "1200s"
waitFor: ["yarn-install", "proxy-install"]
timeout: "1200s"
My .sequelizerc (Documentation here):
const path = require('path');
module.exports = {
'config': path.resolve('config', 'config.js')
}
My config/config.js:
module.exports = {
production: {
username: process.env.PROD_DB_USERNAME,
password: process.env.PROD_DB_PASSWORD,
database: process.env.PROD_DB_NAME,
host: `/workspace/${process.env.INSTANCE_CONNECTION_NAME}`, // Replacing this line with `/workspace/cloudsql/${..}` or `/cloudsql/${..}` leads to the same error
dialect: 'mysql',
}
}
I did enable Public IP on the MySQL instance:
Setting the host to localhost and adding the instance path in socketPath in config.js fixed the issue:
module.exports = {
production: {
username: process.env.PROD_DB_USERNAME,
password: process.env.PROD_DB_PASSWORD,
database: process.env.PROD_DB_NAME,
host: localhost,
dialect: 'mysql',
dialectOptions: {
socketPath: `/workspace/${process.env.INSTANCE_CONNECTION_NAME}`,
},
}
}
TL;DR
Why does this mysqldump rake task successfully connect to MySQL and finish executing when I run it directly in the terminal (in both development environment and with RAILS_ENV=test prefix), but not when I run it inside of an RSpec test?
Long Version
I have a Rails app with the following new rake task:
EXCLUDED_TABLES = %w(
google_oauth_tokens
intuit_oauth_tokens
oauth_access_grants
oauth_access_tokens
oauth_applications
oauth_openid_requests
qbwc_sessions
tokens
)
namespace :db do
desc "Export all tables in the DB except for ones we specify into a compressed file that can be pulled down onto a developer's local machine"
task :export_allowed_tables => :environment do
directory = File.join(Rails.root, 'db', 'dump')
FileUtils.mkdir directory unless File.exists?(directory)
db = YAML::load( File.open( File.join(Rails.root, 'config', 'database.yml') ) )[ Rails.env ]
filename_to_use = File.join( directory, "#{Rails.env}_allowlisted_#{DateTime.now.to_s}.sql" )
username_flag = "-u #{ERB.new(db['username']).result}"
password_flag = "-p#{ERB.new(db['password']).result}"
excluded_tables_flag = EXCLUDED_TABLES.map{|table| "--ignore-table=#{ERB.new(db['database']).result}.#{table}"}.join(" ")
database_param = "#{ERB.new(db['database']).result}"
command = "mysqldump -v #{username_flag} #{password_flag} #{excluded_tables_flag} #{database_param} > #{filename} | gzip > #{filename}.gz"
sh command
end
I also have the following test to ensure my rake task works as expected:
require 'spec_helper'
describe 'rake db:export_allowed_tables' do
after do
files = Dir.glob("db/dump/*allowlisted*.sql")
files.each do |file|
expect(File.delete(file)).to eq 1
end
end
it 'exports the expected data' do
test_full_name = 'Foo Bar Baz Test User'
test_user = FactoryBot.create(:user, full_name: test_full_name)
expect(Dir.glob("db/dump/*.sql")).to be_empty
MyApp::Application.load_tasks
Rake::Task['db:export_allowed_tables'].invoke
expect(Dir.glob("db/dump/*.sql")).not_to be_empty
end
end
The task works fine when I run rake db:export_allowed_tables, and it even works fine when I prefix the command with RAILS_ENV=test. My SQL file shows up as expected, with 7000+ lines of code ending in Dump completed on 2020-05-22 8:44:29, and in my terminal the script starts off with:
mysqldump: [Warning] Using a password on the command line interface can be insecure.
-- Connecting to localhost...
-- Retrieving table structure for table access_control_analytics_reports_claims...
-- Sending SELECT query...
-- Retrieving rows...
...
However, when I run my spec file, the script hangs on the following line:
mysqldump: [Warning] Using a password on the command line interface can be insecure.
-- Connecting to localhost...
And while the .sql file is created in the expected folder, it's empty when I open it (which is to be expected if a connection to localhost failed).
I'm also able to open my Rails console in the terminal (again, both with and without the RAILS_ENV=test prefix) and execute the following without any problems connecting to localhost:
MyApp::Application.load_tasks
Rake::Task['db:export_allowed_tables'].invoke
I am able to create that instance of User, so I know the test environment database is up and running. And I'm able to run the rake task even when MySQL is running, so it doesn't seem to be an issue of the MySQL port being in-use when I run the task. What could be causing my test failure?
Here's my database.yml file, for reference:
default: &default
adapter: <%= ENV["MYAPP_DB_ADAPTER"] || "mysql2_annotator" %>
username: <%= ENV["DATABASE_USERNAME"] || ENV["MYSQL_USER"] || ENV['MYAPP_USER'] || [*FILTERED*] %>
password: <%= ENV["DATABASE_PASSWORD"] || ENV["MYSQL_PASSWORD"] || ENV['MYAPP_PASSWORD'] || [*FILTERED*] %>
database: <%= ENV['MYAPP_DB'] || ENV["DATABASE"] || ENV["MYSQL_DATABASE"] || "myapp_dev#{ENV['MYAPP_ENV_NUMBER']}" %>
host: <%= ENV["MYSQL_HOST"] || ENV['DATABASE_HOST'] || 'localhost' %>
port: <%= ENV['MYAPP_DB_PORT'] || ENV['DATABASE_PORT'] || ENV["MYSQL_PORT"] || 3306 %>
encoding: utf8mb4
collation: utf8mb4_unicode_ci
strict: true
reconnect: true
variables:
sql_mode: "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"
pool: 20
timeout: 20000
checkout_timeout: 20
development:
<<: *default
test:
<<: *default
database: <%= ENV['MYAPP_TEST_DB'] || "myapp_test" %><%= ENV['TEST_ENV_NUMBER'] %>
I am relatively new to Ruby on Rails and I am trying to build a robust system with plenty of wonderful tests. I am currently trying to run a test on my database queries but I am having an error when I run the query.
Disclaimer -> I know that this query is a bit messy, but I am more familiar with SQL queries and this was the best method for me.
def setup
#parts = Part.find_by_sql("SELECT DISTINCT num AS Part_Number,description AS Description, QTYONHAND as On_Hand, QTYONORDER as On_Order,
QTYALLOCATED as Allocated, QTYNOTAVAILABLETOPICK as Not_Avail_Pick FROM (part JOIN qtyinventorytotals ON part.id = qtyinventorytotals.PARTID) WHERE part.id in
(SELECT PARTID FROM qtyinventorytotals);").to_a
end
This is just the setup function for the test case and the error is:
Run options: --seed 34672
# Running:
E
Error:
PartTest#test_parts_pull_from_db:
ActiveRecord::NotNullViolation: Mysql2::Error: Field 'activeFlag' doesn't have a default value: INSERT INTO `part` (`id`) VALUES (980190962)
bin/rails test test/models/part_test.rb:11
Finished in 7.505619s, 0.1332 runs/s, 0.0000 assertions/s.
1 runs, 0 assertions, 0 failures, 1 errors, 0 skips
Here is my database.yml file with the database. I am running a local MySql and querying from the localhost.
# MySQL. Versions 5.1.10 and up are supported.
#
# Install the MySQL driver
# gem install mysql2
#
# Ensure the MySQL gem is defined in your Gemfile
# gem 'mysql2'
#
# And be sure to use new-style password hashing:
# http://dev.mysql.com/doc/refman/5.7/en/old-client.html
#
default: &default
adapter: mysql2
encoding: utf8
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
username: root
password: root
host: localhost
development:
<<: *default
database: xxxxxxxx
# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
<<: *default
database: xxxxxxxx
username: root
password: root
# As with config/secrets.yml, you never want to store sensitive information,
# like your database password, in your source code. If your source code is
# ever seen by anyone, they now have access to your database.
#
# Instead, provide the password as a unix environment variable when you boot
# the app. Read http://guides.rubyonrails.org/configuring.html#configuring-a-database
# for a full rundown on how to provide these environment variables in a
# production deployment.
#
# On Heroku and other platform providers, you may have a full connection URL
# available as an environment variable. For example:
#
# DATABASE_URL="mysql2://myuser:mypass#localhost/somedatabase"
#
# You can use this database configuration with:
#
# production:
# url: <%= ENV['DATABASE_URL'] %>
#
production:
<<: *default
database: xxxxxxxx
username: root
password: root
Again, I am new to Ruby on Rails and all of the wonderful Ruby Magic that happens. I have a feeling that the error is in the database.yml file but I haven't found any resources to steer my in the right direction. Any assistance would be greatly appreciated so that I can really get testing.
Also, I am only reading from this database and there will be no writing, updating or deleting.
EDIT:
Here is the full test class:
require 'test_helper'
class PartTest < ActiveSupport::TestCase
def setup
#parts = Part.find_by_sql("SELECT DISTINCT num AS Part_Number,description AS Description, QTYONHAND as On_Hand, QTYONORDER as On_Order,
QTYALLOCATED as Allocated, QTYNOTAVAILABLETOPICK as Not_Avail_Pick FROM (part JOIN qtyinventorytotals ON part.id = qtyinventorytotals.PARTID) WHERE part.id in
(SELECT PARTID FROM qtyinventorytotals);").to_a
end
# ActiveRecord will return a nil object if there is an error with the DB pull.
test "parts_pull_from_db" do
assert_not #parts.nil?
end
end
New
When I run this query in the console, the result is
Part Load (21549.0ms) SELECT DISTINCT num AS Part_Number,description AS Description, QTYONHAND as On_Hand, QTYONORDER as On_Order,
QTYALLOCATED as Allocated, QTYNOTAVAILABLETOPICK as Not_Avail_Pick FROM (part JOIN qtyinventorytotals ON part.id = qtyinventorytotals.PARTID) WHERE part.id in
(SELECT PARTID FROM qtyinventorytotals);
=> [#<Part id: nil>, #<Part id: nil>, #<Part id: nil>, #<Part id: nil>, #<Part id: nil>, #<Part id: nil>, #<Part id: nil>, #<Part id: nil>, #<Part id: nil>, #<Part id: nil>, #<Part id: nil
>, #<Part id: nil>, #<Part id: nil>, ...]
When I run the rails server, the application successfully pulls the data from the database and renders the data nicely. Why am I getting these results when running from the rails console or the testing environment and not from running the application?
I am using sphinx search in my rails project. I have sphinx.yml file, i-e.
defaults: &defaults
enable_wildcard: 1
min_prefix_len: 2
enable_star: 1
max_matches: 25000
development:
<<: *defaults
test:
<<: *defaults
production:
<<: *defaults
pid_file: "RAILS_ROOT/log/searchd.pid"
searchd_file_path: "RAILS_ROOT/config/db/sphinx"
indexer_binary_name: "/usr/local/bin/indexer"
searchd_binary_name: "/usr/local/bin/searchd"
port: 9314
In my controller's action, i have follwing code.
Application.search(query,:with => options,:order => "updated_at DESC",:max_matches=> 25_000,
:match_mode => :extended).page(params[:page]).per(11)
i am getting error like this.
searchd error (status: 1): per-query max_matches=25000 out of bounds (per-server max_matches=1000)
Any good suggestion?? please.
You need to increase max_matches in sphinx.conf and restart searchd process. max_matches is a searching setting , has nothing to do with indexing process.
I have model to store database connection parameters (host, database name, username, password) and filling it by form. Before create or update I need to check if connection be good with entered parameters. I create validate :check_connection validator:
# don`t change primary connection
class Remote < ActiveRecord::Base; end
def check_connection
return if errors.any? || (!new_record? && password.empty?)
begin
Remote.establish_connection(
adapter: 'mysql2',
host: host,
username: username,
password: password,
database: database,
connect_timeout: 5,
reconnect: false
)
# maybe need to make some sql request? did try it
rescue Exception => e
errors.add :connection, 'Could not connect to database'
end
end
When I try enter accessible host (localhost), code like above works good. But if host like 192.168.1.1 (unaccessible) page freezing after submit form. I see attempts to connect every second and it did not stop after ESC (stop loading page) at browser (I see attempts at tcpdump on network interface). And attempts did not stop..
So, how can I validate connection to database? And if connection can not be established page must will not be load long time.
I did use gem 'mysql2' and bundle install 0.3.11 version. This version ignore connect_timeout and bug fixed at newer version. After I try 0.3.12b4 (gem 'mysql2', '~> 0.3.12b4') all works fine.
Variable connect_timeout is a global variable. Therefore, mysql2 maybe ignore it.
on mysql5.6:
mysql[(none)]> set connect_timeout = 123;
ERROR 1229 (HY000): Variable 'connect_timeout' is a GLOBAL variable and should be set with SET GLOBAL
I set timeout variables when initializing mysql2 but it's not reflected. A README of mysql2 says that you can set the *timeout options, but I think the README is outdated or broken.
on mysql2 0.3.14(gem):
client = Mysql2::Client.new(
host: 'localhost',
database: 'test',
username: 'root',
password: '',
connect_timeout: 3,
read_timeout: 3,
write_timeout: 3,
wait_timeout: 3);
client.query('show variables like "%timeout%"').map{|r| [r["Variable_name"], r["Value"]] }
=> [["connect_timeout", "10"],
["delayed_insert_timeout", "300"],
["innodb_lock_wait_timeout", "50"],
["innodb_rollback_on_timeout", "OFF"],
["interactive_timeout", "28800"],
["lock_wait_timeout", "31536000"],
["net_read_timeout", "30"], # Maybe older mysql has read_timeout?
["net_write_timeout", "60"], # Maybe older mysql has write_timeout?
["slave_net_timeout", "3600"],
["wait_timeout", "28800"]]
If you use ActiveRecord, you can set only wait_timeout variable by database.yml.
in database.yml:
development:
adapter: mysql2
encoding: utf8
charset: utf8
database: test
pool: 5
username: root
password:
host: localhost
connect_timeout: 3
read_timeout: 3
write_timeout: 3
wait_timeout: 3
A result of ActiveRecord 4.0.1:
> ActiveRecord::Base.connection.execute('show variables like "%timeout%"').to_a
=> [["connect_timeout", "10"],
["delayed_insert_timeout", "300"],
["innodb_flush_log_at_timeout", "1"],
["innodb_lock_wait_timeout", "50"],
["innodb_rollback_on_timeout", "OFF"],
["interactive_timeout", "28800"],
["lock_wait_timeout", "31536000"],
["net_read_timeout", "30"],
["net_write_timeout", "60"],
["rpl_stop_slave_timeout", "31536000"],
["slave_net_timeout", "3600"],
["wait_timeout", "3"]]
ActiveRecord set wait_timeout variable in abstract_mysql_adapter.rb.
see:
abstract_mysql_adapter.rb
https://github.com/rails/rails/blob/master/activerecord/lib/active_record/connection_adapters/abstract_mysql_adapter.rb
mysql2_adapter.rb
https://github.com/rails/rails/blob/master/activerecord/lib/active_record/connection_adapters/mysql2_adapter.rb