Chef Creating dynamic Json File - json

I got a problem in generate a json file using a template inside a chef template.
I need to generate a json file like below, called plugin.json:
{
"agents": [
{
"name" : "sqlhost",
"host" : "localhost",
"metrics" : "status,newrelic",
"user" : "root",
"passwd" : "mysecret"
}
]
}
This file will be used by the Newrelic plugin
Template File plugin.json.erb:
<%= Chef::JSONCompat.to_json_pretty(#name) %>
I've created a recipe like this one:
mysql-plugin.rb
# Default Variables
newrelic = decrypt_databag('newrelic')
node.default['newrelic_npi']['license_key'] = newrelic['license_key']
plugin = 'com.newrelic.plugins.mysql.instance'
plugin_json_path = "/opt/newrelic-npi/plugins/com.newrelic.plugins.mysql.instance/newrelic_mysql_plugin-2.0.0/config/plugin.json"
node.default['newrelic_npi']['name'] = "localhost1"
execute 'fetch' do
cwd node['newrelic_npi']['install_path']
user node['newrelic_npi']['user']
command "./npi fetch #{plugin} -y"
not_if do
::File.exist? "#{node['newrelic_npi']['install_path']}/plugins/#{plugin}"
end
end
execute 'prepare' do
cwd node['newrelic_npi']['install_path']
user node['newrelic_npi']['user']
command "./npi prepare #{plugin}"
end
template "#{plugin_json_path}" do
source 'plugin.json.erb'
owner "root"
group "root"
mode "0644"
variables :name => node['newrelic_npi']['name']
action :create
end
execute 'add-service' do
cwd node['newrelic_npi']['install_path']
user node['newrelic_npi']['user']
command "sudo ./npi add-service #{plugin} --start" # needs root privileges
not_if do
::File.exist? "/etc/init.d/newrelic_plugin_#{plugin}"
end
end
At the Template block what should I do to make a Json file like plugin.json?
template "#{plugin_json_path}" do
source 'plugin.json.erb'
owner "root"
group "root"
mode "0644"
variables :name => node['newrelic_npi']['name']
action :create
end
Thanks in Advance.

First. You should be setting any default node variables inside default.rb in your attributes folder. Like this:
default['newrelic_npi']['license_key'] = newrelic['license_key']
default['newrelic_npi']['name'] = "localhost1"
Node attributes are reset before compilation and will pull from that file as well as Ohai. Setting node attributes inside the recipe means they aren't going to be available for your template (you should review this Chef Client Overview).
Next, if you just need to set the hostname or another specific portion of the template file you can do this inside your erb file:
{
"agents": [
{
"name" : "sqlhost",
"host" : "<%= #name %>",
"metrics" : "status,newrelic",
"user" : "root",
"passwd" : "mysecret"
}
]
}
Your current erb file is trying to convert #name (which you try to set to localhost1) to json, not set the whole file like your example.

To generate a correct plugin.json:
{
"agents": [
{
"name" : <%= Chef::JSONCompat.to_json_pretty(#name) %>,
"host" : <%= Chef::JSONCompat.to_json_pretty(#host) %>,
"metrics" : <%= Chef::JSONCompat.to_json_pretty(#metrics) %>,
"user" : <%= Chef::JSONCompat.to_json_pretty(#user) %>,
"passwd" : <%= Chef::JSONCompat.to_json_pretty(#passwd) %>
}
]
}
At the template Block I did like:
template node['newrelic_npi']['plugin_json_path'] do
source 'plugin.json.erb'
owner 'root'
group 'root'
mode '0644'
variables(
:name => node['newrelic_npi']['name'],
:host => node['newrelic_npi']['host'],
:metrics => node['newrelic_npi']['metrics'],
:user => node['newrelic_npi']['user'],
:passwd => node['newrelic_npi']['myql_admin_password']
)
action :create
end

Related

Files saved by carrierwave is being gotten visible to users not logged in

Current Detail
[Env]
nginx puma mysql
Rails == 5.1.5
I mounted carrierwave to upload pictures and files in a post.
[Case]
Users not logged in type the file URL in browser, then they can access and view the file.
[Ideal]
Only users logged in is accessible to the file.
The file path is "uploads/post/images/1234(post_id)/sample.png".
So far, I locate uploads directory under public, app/assets/, and root directory in vain.
Any answers or suggestions are appreciated.
Source
Rails.application.routes.draw do
get 'users/index'
get 'users/show'
get 'posts/index'
devise_for :users, module: :users
resources :users, :only => [:index, :show]
get "/" => "posts#index"
get "posts/like_ranking" => "posts#like_rank"
get "posts/post_count_ranking" => "posts#post_count"
get "posts/tags_search" => "posts#tags_search"
get "posts/new" => "posts#new"
get "posts/:id/reply" => "posts#new"
post "posts/create" => "posts#create"
get "posts/:id" => "posts#show"
get "posts/:id/edit" => "posts#edit"
post "posts/:id/update" => "posts#update"
post "posts/:id/destroy" => "posts#destroy"
get 'tags/:tag', to: 'posts#index', as: :tag
get "users/:id/likes" => "users#likes"
get "users/:id/reply" => "users#reply"
resources :posts, only: %w(index)
resources :posts, shallow: true do
resources :likes, only: [:create, :destroy]
end
end
"
class ImageUploader < CarrierWave::Uploader::Base
# Choose what kind of storage to use for this uploader:
storage :file
# Override the directory where uploaded files will be stored.
# This is a sensible default for uploaders that are meant to be mounted:
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
end
All assets under the public folder is served by nginx. In order to make it available for logged users you can first upload files to other directory than public and create action which serve files by Rails app from that directory and check if users logged before serving.

rspec routing with subdomain

I'm having a Rails 5 api only app using rspec and versioned this way :
app
- controllers
- api
- v1
- users_controller.rb
My api/v1/users_controller.rb :
module Api::V1
class UsersController < ApiController
My config\routes.rb :
Rails.application.routes.draw do
# For details on the DSL available within this file, see http://guides.rubyonrails.org/routing.html
constraints subdomain: 'api' do
scope module: 'api' do
namespace :v1 do
resources :users
end
end
end
end
When I check the routes with rails routes it shows me.
Prefix Verb URI Pattern Controller#Action
v1_users GET /v1/users(.:format) api/v1/users#index {:subdomain=>"api"}
POST /v1/users(.:format) api/v1/users#create {:subdomain=>"api"}
v1_user GET /v1/users/:id(.:format) api/v1/users#show {:subdomain=>"api"}
PATCH /v1/users/:id(.:format) api/v1/users#update {:subdomain=>"api"}
PUT /v1/users/:id(.:format) api/v1/users#update {:subdomain=>"api"}
DELETE /v1/users/:id(.:format) api/v1/users#destroy {:subdomain=>"api"}
My spec file :
require "rails_helper"
RSpec.describe Api::V1::UsersController, type: :routing do
describe "routing" do
it "routes to #index" do
expect(:get => "/v1/users").to route_to("api/v1/users#index")
end
it "routes to #create" do
expect(:post => "/v1/users").to route_to("api/v1/users#create")
end
it "routes to #show" do
expect(:get => "/v1/users/1").to route_to("api/v1/users#show", :id => "1")
end
it "routes to #update via PUT" do
expect(:put => "/v1/users/1").to route_to("api/v1/users#update", :id => "1")
end
it "routes to #update via PATCH" do
expect(:patch => "/v1/users/1").to route_to("api/v1/users#update", :id => "1")
end
it "routes to #destroy" do
expect(:delete => "/v1/users/1").to route_to("api/v1/users#destroy", :id => "1")
end
end
end
But when I'm testing my routes with rspec it fails as it.
bundle exec rspec spec/routing/users_routing_spec.rb
FFFFF
Failures:
1) Api::V1::UsersController routing routes to #index
Failure/Error: expect(:get => "/v1/users").to route_to("api/v1/users#index")
No route matches "/v1/users"
# ./spec/routing/users_routing_spec.rb:7:in `block (3 levels) in <top (required)>'
I don't understand why. Any idea ?
You have to specify "subdomain" for your spec.
before do
Rails.application.routes.default_url_options[:host] = 'test.host'
end
it "routes to #index" do
expect(:get => v1_users_url).to route_to('v1/users#index', subdomain: 'api')
end

How to integrate ElasticSearch with MySQL?

In one of my project, I am planning to use ElasticSearch with MySQL.
I have successfully installed ElasticSearch. I am able to manage index in ES separately. but I don't know how to implement the same with MySQL.
I have read a couple of documents but I am a bit confused and not having a clear idea.
As of ES 5.x , they have given this feature out of the box with logstash plugin.
This will periodically import data from database and push to ES server.
One has to create a simple import file given below (which is also described here) and use logstash to run the script. Logstash supports running this script on a schedule.
# file: contacts-index-logstash.conf
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
jdbc_user => "user"
jdbc_password => "pswd"
schedule => "* * * * *"
jdbc_validate_connection => true
jdbc_driver_library => "/path/to/latest/mysql-connector-java-jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
statement => "SELECT * from contacts where updatedAt > :sql_last_value"
}
}
output {
elasticsearch {
protocol => http
index => "contacts"
document_type => "contact"
document_id => "%{id}"
host => "ES_NODE_HOST"
}
}
# "* * * * *" -> run every minute
# sql_last_value is a built in parameter whose value is set to Thursday, 1 January 1970,
# or 0 if use_column_value is true and tracking_column is set
You can download the mysql jar from maven here.
In case indexes do not exist in ES when this script is executed, they will be created automatically. Just like a normal post call to elasticsearch
Finally i was able to find the answer. sharing my findings.
To use ElasticSearch with Mysql you will require The Java Database Connection (JDBC) importer. with JDBC drivers you can sync your mysql data into elasticsearch.
I am using ubuntu 14.04 LTS and you will require to install Java8 to run elasticsearch as it is written in Java
following are steps to install ElasticSearch 2.2.0 and ElasticSearch-jdbc 2.2.0 and please note both the versions has to be same
after installing Java8 ..... install elasticsearch 2.2.0 as follows
# cd /opt
# wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.2.0/elasticsearch-2.2.0.deb
# sudo dpkg -i elasticsearch-2.2.0.deb
This installation procedure will install Elasticsearch in /usr/share/elasticsearch/ whose configuration files will be placed in /etc/elasticsearch .
Now lets do some basic configuration in config file. here /etc/elasticsearch/elasticsearch.yml is our config file
you can open file to change by
nano /etc/elasticsearch/elasticsearch.yml
and change cluster name and node name
For example :
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: servercluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: vps.server.com
#
# Add custom attributes to the node:
#
# node.rack: r1
Now save the file and start elasticsearch
/etc/init.d/elasticsearch start
to test ES installed or not run following
curl -XGET 'http://localhost:9200/?pretty'
If you get following then your elasticsearch is installed now :)
{
"name" : "vps.server.com",
"cluster_name" : "servercluster",
"version" : {
"number" : "2.2.0",
"build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
"build_timestamp" : "2016-01-27T13:32:39Z",
"build_snapshot" : false,
"lucene_version" : "5.4.1"
},
"tagline" : "You Know, for Search"
}
Now let's install elasticsearch-JDBC
download it from http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/2.3.3.1/elasticsearch-jdbc-2.3.3.1-dist.zip and extract the same in /etc/elasticsearch/ and create "logs" folder also there ( path of logs should be /etc/elasticsearch/logs)
I have one database created in mysql having name "ElasticSearchDatabase" and inside that table named "test" with fields id,name and email
cd /etc/elasticsearch
and run following
echo '{
"type":"jdbc",
"jdbc":{
"url":"jdbc:mysql://localhost:3306/ElasticSearchDatabase",
"user":"root",
"password":"",
"sql":"SELECT id as _id, id, name,email FROM test",
"index":"users",
"type":"users",
"autocommit":"true",
"metrics": {
"enabled" : true
},
"elasticsearch" : {
"cluster" : "servercluster",
"host" : "localhost",
"port" : 9300
}
}
}' | java -cp "/etc/elasticsearch/elasticsearch-jdbc-2.2.0.0/lib/*" -"Dlog4j.configurationFile=file:////etc/elasticsearch/elasticsearch-jdbc-2.2.0.0/bin/log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"
now check if mysql data imported in ES or not
curl -XGET http://localhost:9200/users/_search/?pretty
If all goes well, you will be able to see all your mysql data in json format
and if any error is there you will be able to see them in /etc/elasticsearch/logs/jdbc.log file
Caution :
In older versions of ES ... plugin Elasticsearch-river-jdbc was used which is completely deprecated in latest version so do not use it.
I hope i could save your time :)
Any further thoughts are appreciated
Reference url : https://github.com/jprante/elasticsearch-jdbc
The logstash JDBC plugin will do the job:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/testdb"
jdbc_user => "root"
jdbc_password => "factweavers"
# The path to our downloaded jdbc driver
jdbc_driver_library => "/home/comp/Downloads/mysql-connector-java-5.1.38.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
schedule => "* * * *"
statement => "SELECT" * FROM testtable where Date > :sql_last_value order by Date"
use_column_value => true
tracking_column => Date
}
output {
stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "test-migrate"
"document_type" => "data"
"document_id" => "%{personid}"
}
}
To make it more simple I have created a PHP class to Setup MySQL with Elasticsearch. Using my Class you can sync your MySQL data in elasticsearch and also perform full-text search. You just need to set your SQL query and class will do the rest for you.

Rails connect to remote db

How to properly connect to remote db?
Now i have
def db_params
{:adapter => "mysql2",
:host => "host",
:username => "name",
:password => "pass",
:database => "mydb"}
end
def connect_to_remote_db
ActiveRecord::Base.establish_connection(db_params)
end
When i write connect_to_remote_db it seems ok
I know that remote db has table 'Team'
but when i write Team
in console it returns me uninitialized constant Team
How to handle it properly?
When you call Team ActiveRecord's primary connection is looked up, hence the error.
You could probably wrap that in class.
Since I had dealt with similar situation, you could have that connection in database.ymlitself and use.
development:
adapter: mysql2
other stuff...
db_2:
adapter: mysql2
other stuff..
Then create a class
class Team < ActiveRecord::Base
establish_connection(:db_2)
self.table_name = "teams"
end
from - https://stackoverflow.com/a/26574386/2231236
You need to create model in your application ( of that remote db table) and establish connection. Example:
team.rb
class Team< ActiveRecord::Base
establish_connection "remote_db"
end
If you have multiple table you want to use from that remote db, you can make module and just include it in every model for remote db table.
Module example:
module RemoteConnection
extend ActiveSupport::Concern
included do
establish_connection "remote_db"
end
end
and than
class Team< ActiveRecord::Base
include RemoteConnection
end
Use database.yml file to store connections:
...
remote_db:
:adapter => "mysql2",
:host => "host",
:username => "name",
:password => "pass",
:database => "mydb"
...

500 Error in production adapter rails when starting new user session

I have been having this issue ever since I deployed and i can't figure it out.
I'll give some information and let me know if you need anything else! Thanks!
I, [2013-09-08T12:44:31.935143 #19456] INFO -- : Started POST "/sessions" for {IP ADDRESS} at 2013-09-08 12:44:31 -0700
I, [2013-09-08T12:44:31.937969 #19456] INFO -- : Processing by SessionsController#create as HTML
I, [2013-09-08T12:44:31.938102 #19456] INFO -- : Parameters: {"utf8"=>"✓", "authenticity_token"=>"{AUTHENTICITY TOKEN}", "email"=>"mike#test.com", "password"=>"[FILTERED]", "commit"=>"Log In"}
I, [2013-09-08T12:44:31.941064 #19456] INFO -- : Completed 500 Internal Server Error in 3ms
F, [2013-09-08T12:44:31.943631 #19456] FATAL -- :
ActiveRecord::StatementInvalid (Could not find table 'users'):
app/controllers/sessions_controller.rb:6:in `create'
Obviously it's telling me that the "users" table doesn't exist, but that BS, because it does. Perhaps it can't find the table? Which i ALSO think is wierd, because I created the table using Rails migrations.
Here is my production Adapter just for reference:
production:
adapter: mysql
database: {DATABASENAME}
username: {USERNAME}
password: {PASSWORD}
host: localhost
port: 3306
Here is my seeds file:
User.create([{ email: 'mike#test2.com' }, { password_digest: 'password' }])
And my user model:
class User < ActiveRecord::Base
has_secure_password
validates_uniqueness_of :email
end
And my sessions controller (handles the login):
class SessionsController < ApplicationController
def new
end
def create
user = User.find_by_email(params[:email])
if user && user.authenticate(params[:password])
session[:user_id] = user.id
redirect_to root_url, notice: "Logged in!"
else
flash.now.alert = "Email or password is invalid"
render "new"
end
end
def destroy
session[:user_id] = nil
redirect_to root_url, notice: "Logged out!"
end
end
I created the user directly in the database, so the issue isn't that the user doesnt exist, the log file is saying the table 'users' doesnt exist, but that is false as well...i really don't know whats going on...
OH, BTW this all works in development. Login, user creation, everything. I was using sqlite3 for development and switched to mysql for production, just screwed everything up....
Any help is appreciated!
For future reference, this is the fix if anyone else is having an issue with their seeds not working:
This is what I was using:
User.create([{ email: 'mike#test2.com' }, { password_digest: 'password' }])
This is what it should be:
users =User.create!(:email 'mike#test2.com', :password 'password', :password_confirmation 'password')
users.save
Using
Create!
instead of just plain
Create
enables validation, and the seed will fail and explain the problem. In my case, it was that I was using password_digest, which is the column name, instead of password and password_confirmation.
Also, save the seed creation in a variable (users in my case), then save said variable by doing this:
users.save
Simple as that!
Hope this helps a fellow newbie in the future!