Can someone explain to me how I can trigger Celery tasks through Logstash?
Is it possible?
If I try to do that in PHP through the 'php-amqplib' library it works fine: (without using Logstash)
$connection = new AMQPStreamConnection(
'rabbitmq.local',
5672,
'guest',
'guest'
);
$channel = $connection->channel();
$channel->queue_declare(
'celery',
false,
true,
false,
false
);
$taskId = rand(1000, 10000);
$props = array(
'content_type' => 'application/json',
'content_encoding' => 'utf-8',
);
$body = array(
'task' => 'process_next_task',
'lang' => 'py',
'args' => array('ktest' => 'vtest'),
'kwargs' => array('ktest' => 'vtest'),
'origin' => '#'.'mytest',
'id' => $taskId,
);
$msg = new AMQPMessage(json_encode($body), $props);
$channel->basic_publish($msg, 'celery', 'celery');
According to the Celery docs:
http://docs.celeryproject.org/en/latest/internals/protocol.html
I'm trying to send the request in the json format, this is my Logstash filter:
ruby
{
remove_field => ['headers', '#timestamp', '#version', 'host', 'type']
code => "
event.set('properties',
{
:content_type => 'application/json',
:content_encoding => 'utf-8'
})
"
}
And Celery answer is:
[2017-05-05 14:35:09,090: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!
{content_type:None content_encoding:None delivery_info:{'exchange': 'celery', 'routing_key': 'celery', 'redelivered': False, 'consumer_tag': 'None4', 'delivery_tag': 66} headers={}}
Basically, Celery is not able to decode my message format or better... I'm not able to set the request in the JSON format :)
It's driving me crazy, thank you in advance for any clues :)
Forgot it, this is my output plugin in Logstash
rabbitmq
{
key => "celery"
exchange => "celery"
exchange_type => "direct"
user => "${RABBITMQ_USER}"
password => "${RABBITMQ_PASSWORD}"
host => "${RABBITMQ_HOST}"
port => "${RABBITMQ_PORT}"
durable => true
persistent => true
codec => json
}
From the information provided in this question, you can't.
When you're playing with the event in the ruby filter, you're actually playing with what will be put in the body of the message, while you'd like to set the rabbitmq headers and properties of your message.
Till that functionality has been tackled, I do not think you'll be able to achieve it unless of course you implement it yourself. After all, the plugin is available on github.
As Olivier said, right now is not possible but I've created a pull request to the official project.
https://github.com/logstash-plugins/logstash-output-rabbitmq/pull/59
If you are looking for a working version take a look to my clone:
https://github.com/useless-stuff/logstash-output-rabbitmq
You should be seriously scared about that code :)
I'm completely far away to be a Ruby developer
But it works :)
Related
Locally everything works great:
Storage::fake('local');
$file = UploadedFile::fake()->image('watch.jpg');
$watchData = [
'model' => 'beatae unde adipisci',
'size' => '50',
'number' => 'm98300',
'metal' => 'deserunt',
'rrp' => '92840',
'status' => 'discontinued',
'brand_id' => '2',
'new_image' => $file,
];
$response = $this->post('/watch', $watchData);
Storage::disk('local')->assertExists('mg/jc-int/watch-images/1.jpg');
When I try to run this same test as part of my deployment workflow in GitHub Actions:
1) Tests\Feature\WatchImagesTest::test_a_watch_image_can_be_created
Unable to find a file or directory at path [mg/jc-int/watch-images/1.jpg].
Failed asserting that false is true.
I've tried a few things but I think this is more that I'm not understand how either Storage::fake() works or how GitHub actions handles filepaths?
Found the issue.
In my controller that handles the image upload, I'm grabbing a folder from my config:
$folder = config('site.DO_SUBFOLDER');
This value is taken from my .env and wasn't being included in the deployment pipeline.
snappy.php get this code
'pdf' => [
'enabled' => true,
'binary' => env('"C:\Program Files\wkhtmltox\bin\wkhtmltopdf-amd64.exe"'),
'timeout' => false,
'options' => [],
'env' => [],
],
and when i try this code for test html to pdf in web.php
Route::get('/', function () {
$snappy = App::make('snappy.pdf');
$snappy->generateFromHtml('<h1>hello</h1>','exemple.pdf');
return view('welcome');
});
The problem is the executable path.
If Snappy don't find the executable, he get this message.
Please mark the file as executable (chmod +x on linux)
You can see this documentation
The problem is your binary path, if you are using Windows you must use the path like this:
'binary' => '"C:\Program Files\wkhtmltox\bin\wkhtmltopdf-amd64.exe"'
And make sure the path to the .exe is correct.
I need a brief explanation of how can I convert MySQL data to Elastic Search using logstash.
can anyone explain the step by step process about this
This is a broad question, I don't know how much you familiar with MySQL and ES. Let's say you have a table user. you may just simply dump it as csv and load it at your ES will be good. but if you have a dynamic data, like the MySQL just like a pipeline, you need to write a Script to do those stuff. anyway you can check the below link to build your basic knowledge before you ask How.
How to dump mysql?
How to load data to ES
Also, since you probably want to know how to covert your CSV to json file, which is the best suite for ES to understand.
How to covert CSV to JSON
You can do it using the jdbc input plugin for logstash.
Here is a config example.
Let me provide you with a high level instruction set.
Install Logstash, and Elasticsearch.
In Logstash bin folder copy jar ojdbc7.jar.
For logstash, create a config file ex: config.yml
#
input {
# Get the data from database, configure fields to get data incrementally
jdbc {
jdbc_driver_library => "./ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:#db:1521:instance"
jdbc_user => "user"
jdbc_password => "pwd"
id => "some_id"
jdbc_validate_connection => true
jdbc_validation_timeout => 1800
connection_retry_attempts => 10
connection_retry_attempts_wait_time => 10
#fetch the db logs using logid
statement => "select * from customer.table where logid > :sql_last_value order by logid asc"
#limit how many results are pre-fetched at a time from the cursor into the client’s cache before retrieving more results from the result-set
jdbc_fetch_size => 500
jdbc_default_timezone => "America/New_York"
use_column_value => true
tracking_column => "logid"
tracking_column_type => "numeric"
record_last_run => true
schedule => "*/2 * * * *"
type => "log.customer.table"
add_field => {"source" => "customer.table"}
add_field => {"tags" => "customer.table" }
add_field => {"logLevel" => "ERROR" }
last_run_metadata_path => "last_run_metadata_path_table.txt"
}
}
# Massage the data to store in index
filter {
if [type] == 'log.customer.table' {
#assign values from db column to custom fields of index
ruby{
code => "event.set( 'errorid', event.get('ssoerrorid') );
event.set( 'msg', event.get('errormessage') );
event.set( 'logTimeStamp', event.get('date_created'));
event.set( '#timestamp', event.get('date_created'));
"
}
#remove the db columns that were mapped to custom fields of index
mutate {
remove_field => ["ssoerrorid","errormessage","date_created" ]
}
}#end of [type] == 'log.customer.table'
} #end of filter
# Insert into index
output {
if [type] == 'log.customer.table' {
amazon_es {
hosts => ["vpc-xxx-es-yyyyyyyyyyyy.us-east-1.es.amazonaws.com"]
region => "us-east-1"
aws_access_key_id => '<access key>'
aws_secret_access_key => '<secret password>'
index => "production-logs-table-%{+YYYY.MM.dd}"
}
}
}
Go to bin, Run as
logstash -f config.yml
I using Memcache to cache data in project yii2.
with config:
'cache' => [
'class' => 'yii\caching\MemCache',
'servers' => [
[
'host' => 'localhost',
'port' => 11211,
],
],
],
With code: return false
\Yii::$app->cache->set('abc', 'value of abc', 20); // 20 seconds
$data = \Yii::$app->cache->get('abc'); var_dump($data); // return false
But if i set:
\Yii::$app->cache->set('abc', 'value of abc', -20); // -20 seconds $data = \Yii::$app->cache->get('abc'); var_dump($data); // return "value of abc"
So why this? Thanks.
Can you increment the seconds?
Pls try this and let me know the results:
\Yii::$app->cache->set('abc', 'value of abc', 10000);
$data = \Yii::$app->cache->get('abc'); var_dump($data);
A bit late, but in case someone else have same problem, here are my thoughts...
It might depend on whether MemCached or MemCache is installed. If config uses MemCache (like in the question) but MemCached is installed, it will work using get() but set() is different after the two first parameters, ie duration.
i am using simfony2.x and i want to use knpsnappybundle budle to create pdf file from html. so i had use this configuration,
knp_snappy:
pdf:
enabled: true
binary: "\"C:\\Program Files\\wkhtmltopdf\\bin\\wkhtmltopdf.exe\""
options: []
image:
enabled: true
binary: "\"C:\\Program Files\\wkhtmltopdf\\bin\\wkhtmltoimage.exe\""
options: []
then after in my contoller i am using this service like
$html = $this->renderView('MyBundle:MyController:view.html.twig', array(
'project' => $project,
'answers' => $answers
));
return new Response(
$this->get('knp_snappy.pdf')->getOutputFromHtml($html), 200, array(
'Content-Type' => 'application/pdf',
'Content-Disposition' => 'attachment; filename="test.pdf"',
'encoding' => 'UTF-8',
)
);
it create pdf file for me but pdf contain wrong design at the time of page break, this is the screen cap of my pdf
and here is my actual Html (from which, this pdf is generated)
so can anyone help me to avoid this issue of html page break ?
thanks