require 'os'
require 'socket'
require 'mysql2'
require 'etc'
require 'time'
def getBits
return OS.bits.to_s
end
def getPCName
return Socket.gethostname
end
def getUser
return Etc.getlogin
end
info = Hash["SYSBITS " => getBits,"PCN " => getPCName,"USER " => getUser]
for h in info.keys
print h.chomp
end
for b in info.values
print b.chomp
end
connection = Mysql2::Client.new(:host => "localhost", :username => "root",:password => "",:database => "ruby")
result = connection.query("INSERT INTO `mydata`(`#{h}`) VALUES ('#{b}')")
So the idea here is to make Hash of information returned and then use Hash keys as columns and hash values as rows in mysql query but it doesn't work.
I added separation manually because I can't see any other way.
This is the error I'm getting:
_query': Unknown column 'USER ' in 'field list' (Mysql2::Error)
I'm not using sqlite3 gem
I'm using mysql2 gem
I'm retrieving data from MySQL database given that it meets the condition of a certain event type and severity. However, it returns only one row instead of an array of results. It really puzzles me. Shouldnt .map return an array?
result = connect.query("SELECT * FROM data WHERE event_type = 'ALARM_OPENED' AND severity = '2'")
equipments = result.map do |record|
[
record['sourcetime'].strftime('%H:%M:%S'),
record['equipment_id'],
record['description']
]
end
p equipments
I had misread your question...I think what you are looking for is in here.
UPDATE
You can use each instead, like this:
#!/usr/bin/env ruby
require 'mysql2'
connect= Mysql2::Client.new(:host => '', :username => '', :password => '', :database => '')
equipments = []
result = connect.query("SELECT * FROM data WHERE event_type = 'ALARM_OPENED' AND severity = '2'", :symbolize_keys => true).each do |row|
equipments << [
row[:sourcetime].strftime('%H:%M:%S'),
row[:equipment_id],
row[:description]
]
end
puts "#equipments {equipments}"
EDITED:
I forgot to add .each at the end of the query. So it was returning the initialized empty array instead.
You must need to change your sql statement :
result = connect.query("SELECT * FROM data WHERE event_type = 'ALARM_OPENED' AND severity = '2'", :as => :array)
I'm trying to use EM::Synchrony to speed up my queries by making them async. Following along with the examples from the github page here I'm making 2 asynchronous queries:
EventMachine.synchrony do
db = EventMachine::Synchrony::ConnectionPool.new(size: 2) do
Mysql2::EM::Client.new(
:host => config[:server],
:username => config[:user],
:password => config[:pwd],
:database => config[:db_name]
)
end
multi = EventMachine::Synchrony::Multi.new
multi.add :a, db.aquery("
select count(distinct userid) from my_table
where date = '2013-09-28'
")
multi.add :b, db.aquery("
select count(distinct userid) from my_table
where date = '2013-09-27'
")
res = multi.perform
puts res
# p "Look ma, no callbacks, and parallel MySQL requests!"
# p res.responses[:callback][0]
EventMachine.stop
end
> #<EventMachine::Synchrony::Multi:0x00000001eb8da8>
My question is how do I setup a callback to actually get the values returned by the queries? What I would like to do is once the queries are finished, aggregate them back together and write to another table or csv or whatever. Thanks.
Maybe you don't need Synchrony? They are async anyway. https://github.com/brianmario/mysql2#eventmachine
But if you do need then probably the answer is
res.responses[:callback][:a]
res.responses[:callback][:b]
https://github.com/igrigorik/em-synchrony/blob/master/lib/em-synchrony/em-multi.rb#L17
What I found is that the following code will give me the results from the queries:
res.responses[:callback].each do
|obj|
obj[1].callback do
|rows|
rows.each do
|row|
puts row.inspect
end
end
end
$ruby async_mysql.rb
{"count(distinct ui)"=>159}
{"count(distinct ui)"=>168}
i want to upload CSV files through the activeadmin panel.
on the index page from the resource "product" i want a button next to the "new product" button with "import csv file".
i dont know where to start.
in the documentation is something about collection_action, but with the code below i have no link at the top.
ActiveAdmin.register Post do
collection_action :import_csv, :method => :post do
# Do some CSV importing work here...
redirect_to :action => :index, :notice => "CSV imported successfully!"
end
end
anyone here who use activeadmin and can import csv data?
Continuing from Thomas Watsons great start to the answer which helped me get my bearings before figuring the rest of it out.
The code blow allows not just CSV upload for the example Posts model but for any subsequent models thereafter. all you need to do is copy the action_item ands both collection_actions from the example into any other ActiveAdmin.register block and the functionality will be the same. hope this helps.
app/admin/posts.rb
ActiveAdmin.register Post do
action_item :only => :index do
link_to 'Upload CSV', :action => 'upload_csv'
end
collection_action :upload_csv do
render "admin/csv/upload_csv"
end
collection_action :import_csv, :method => :post do
CsvDb.convert_save("post", params[:dump][:file])
redirect_to :action => :index, :notice => "CSV imported successfully!"
end
end
app/models/csv_db.rb
require 'csv'
class CsvDb
class << self
def convert_save(model_name, csv_data)
csv_file = csv_data.read
CSV.parse(csv_file) do |row|
target_model = model_name.classify.constantize
new_object = target_model.new
column_iterator = -1
target_model.column_names.each do |key|
column_iterator += 1
unless key == "ID"
value = row[column_iterator]
new_object.send "#{key}=", value
end
end
new_object.save
end
end
end
end
note: this example does a check to see whether or not the first column is an ID column, it then skips that column as rails will assign an ID to the new object (see example CSV below for reference)
app/views/admin/csv/upload_csv.html.haml
= form_for :dump, :url=>{:action=>"import_csv"}, :html => { :multipart => true } do |f|
%table
%tr
%td
%label{:for => "dump_file"}
Select a CSV File :
%td
= f.file_field :file
%tr
%td
= submit_tag 'Submit'
app/public/example.csv
"1","TITLE EXAMPLE","MESSAGE EXAMPLE","POSTED AT DATETIME"
"2","TITLE EXAMPLE","MESSAGE EXAMPLE","POSTED AT DATETIME"
"3","TITLE EXAMPLE","MESSAGE EXAMPLE","POSTED AT DATETIME"
"4","TITLE EXAMPLE","MESSAGE EXAMPLE","POSTED AT DATETIME"
"5","TITLE EXAMPLE","MESSAGE EXAMPLE","POSTED AT DATETIME"
note: quotations not always needed
Adding a collection_action does not automatically add a button linking to that action. To add a button at the top of the index screen you need to add the following code to your ActiveAdmin.register block:
action_item :only => :index do
link_to 'Upload CSV', :action => 'upload_csv'
end
But before calling the collection action you posted in your question, you first need the user to specify which file to upload. I would personally do this on another screen (i.e. creating two collection actions - one being a :get action, the other being your :post action). So the complete AA controller would look something like this:
ActiveAdmin.register Post do
action_item :only => :index do
link_to 'Upload posts', :action => 'upload_csv'
end
collection_action :upload_csv do
# The method defaults to :get
# By default Active Admin will look for a view file with the same
# name as the action, so you need to create your view at
# app/views/admin/posts/upload_csv.html.haml (or .erb if that's your weapon)
end
collection_action :import_csv, :method => :post do
# Do some CSV importing work here...
redirect_to :action => :index, :notice => "CSV imported successfully!"
end
end
#krhorst, I was trying to use your code, but unfortunately it sucks on big imports. It eat so much memory =( So I decided to use own solution based on activerecord-import gem
Here it is https://github.com/Fivell/active_admin_import
Features
Encoding handling
Support importing with ZIP file
Two step importing (see example2)
CSV options
Ability to prepend CSV headers automatically
Bulk import (activerecord-import)
Ability to customize template
Callbacks support
Support import from zip file
....
Based on ben.m's excellent answer above I replaced the csv_db.rb section suggested with this:
require 'csv'
class CsvDb
class << self
def convert_save(model_name, csv_data)
begin
target_model = model_name.classify.constantize
CSV.foreach(csv_data.path, :headers => true) do |row|
target_model.create(row.to_hash)
end
rescue Exception => e
Rails.logger.error e.message
Rails.logger.error e.backtrace.join("\n")
end
end
end
end
While not a complete answer I did not want my changes to pollute ben.m's answer in case I did something egregiously wrong.
expanding on ben.m's response which I found very useful.
I had issues with the CSV import logic (attributes not lining up and column iterator not functioning as required) and implemented a change which instead utilizes a per line loop and the model.create method. This allows you to import a .csv with the header line matching the attributes.
app/models/csv_db.rb
require 'csv'
class CsvDb
class << self
def convert_save(model_name, csv_data)
csv_file = csv_data.read
lines = CSV.parse(csv_file)
header = lines.shift
lines.each do |line|
attributes = Hash[header.zip line]
target_model = model_name.classify.constantize
target_model.create(attributes)
end
end
end
end
So your imported CSV file can look like this (use to match up with model attributes):
importExample.csv
first_name,last_name,attribute1,attribute2
john,citizen,value1,value2
For large excel which takes time on normal process, I created a gem that process Excel sheets using an active job and display results using action cable(websockets)
https://github.com/shivgarg5676/active_admin_excel_upload
Some of the solutions above worked pretty well. I ran into challenges in practice that I solved here below. The solved problems are:
Importing CSV data with columns in different orders
Preventing errors caused by hidden characters in Excel CSVs
Resetting the database primary_key so that the application can continue to add records after the import
Note: I took out the ID filter so I could change IDs for what I'm working on, but most use cases probably want to keep it in.
require 'csv'
class CsvDb
class << self
def convert_save(model_name, csv_data)
csv_file = csv_data.read
csv_file.to_s.force_encoding("UTF-8")
csv_file.sub!("\xEF\xBB\xBF", '')
target_model = model_name.classify.constantize
headers = csv_file.split("\n")[0].split(",")
CSV.parse(csv_file, headers: true) do |row|
new_object = target_model.new
column_iterator = -1
headers.each do |key|
column_iterator += 1
value = row[column_iterator]
new_object.send "#{key.chomp}=", value
end
new_object.save
end
ActiveRecord::Base.connection.reset_pk_sequence!(model_name.pluralize)
end
end
end
I want to return about 90k items in a JSON document but I'm getting this error when I make the call:
Timeout::Error in ApisController#api_b
time's up!
Rails.root: /root/api_b
I am simply running "rails s" with the default rails server.
What's the way to make this work and return the document?
Thanks
#bs.each do |a|
puts "dentro do bs.each"
#final << { :Email => a['headers']['to'], :At => a['date'], :subject => a['headers']['subject'], :Type => a['headers']['status'], :Message_id => a['headers']['message_id'] }
end
Being #bs the BSON object from MongoDB. The timeout is in "#final << ..."
If you are experiencing timeouts from rails and it is possible to cache the data (e.g. the data changes infrequently), I would generate the response in the background using resque or delayed_job and than have Rails dump that to the client. Or if the data cannot be cached, use a lightweight Rack handler like Sinatra and Metal to generate the responses.
Edited to reflect sample data
I was able to run the following code in a Rails 3.0.9 instance against a high performance Mongo 1.8.4 instance. I was using Mongo 1.3.1, bson_ext 1.3.1, webrick 1.3.1 and Ruby 1.9.2p180 x64. It did not time out but it took some time to load. My sample Mongo DB has 100k records and contains no indexes.
before_filter :profile_start
after_filter :profile_end
def index
db = #conn['sample-dbs']
collection = db['email-test']
#final = []
#bs = collection.find({})
#bs.each do |a|
puts "dentro do bs.each"
#final << { :Email => a['headers']['to'], :At => a['date'], :subject => a['headers']['subject'], :Type => a['headers']['status'], :Message_id => a['headers']['message_id'] }
end
render :json => #final
end
private
def profile_start
RubyProf.start
end
def profile_end
RubyProf::FlatPrinter.new(RubyProf.stop).print
end
A more efficient way to dump out the records would be
#bs = collection.find({}, {:fields => ["headers", "date"]})
#final = #bs.map{|a| {:Email => a['headers']['to'], :At => a['date'], :subject => a['headers']['subject'], :Type => a['headers']['status'], :Message_id => a['headers']['message_id'] }}
render :json => #final
My data generator
100000.times do |i|
p i
#coll.insert({:date =>Time.now(),:headers => {"to"=>"me#foo.com", "subject"=>"meeeeeeeeee", "status" => "ffffffffffffffffff", "message_id" => "1234634673"}})
end