load dynamic data from MySQL table in ElasticSearch using JDBC driver - mysql

I have get dyanamically data from MySQL tables in my elasticSearch index. For that i have used following link for but not get propper result:
I have used following code:
echo '{
"type":"jdbc",
"jdbc":{
"url":"jdbc:mysql://localhost:3306/CDFL",
"user":"root",
"password":"root",
"useSSL":"false",
"sql":"SELECT * FROM event",
"index":"event",
"type":"event",
"autocommit":"true",
"metrics": {
"enabled" : true
},
"elasticsearch" : {
"cluster" : "servercluster",
"host" : "localhost",
"port" : 9300
}
}
}' | java -cp "/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib/*" -"Dlog4j.configurationFile=file:////etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin/log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"
and for that get solution i have used following link:
ElasticSearch how to integrate with Mysql
https://github.com/jprante/elasticsearch-jdbc
Fetching changes from table with ElasticSearch JDBC river
https://github.com/logstash-plugins/logstash-input-jdbc

I have got a answer for that question:
make one file in root directory called event.sh and following code in that file
event.sh
curl -XDELETE 'localhost:9200/event'
bin=/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin
lib=/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib
echo '{
"type":"jdbc",
"jdbc":{
"url":"jdbc:mysql://localhost:3306/CDFL",
"user":"root",
"password":"root",
"useSSL":"false",
"sql":"SELECT * FROM event",
"index":"event",
"type":"event",
"poll" : "6s",
"autocommit":"true",
"metrics": {
"enabled" : true
},
"elasticsearch" : {
"cluster" : "servercluster",
"host" : "localhost",
"port" : 9300
}
}
}' | java -cp "/etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/lib/*" -"Dlog4j.configurationFile=file:////etc/elasticsearch/elasticsearch-jdbc-2.3.4.0/bin/log4j2.xml" "org.xbib.tools.Runner" "org.xbib.tools.JDBCImporter"
echo "sleeping while importer should run..."
sleep 10
curl -XGET 'localhost:9200/event/_refresh'
and run that file in cmd type following command:
sh elasticSearch/event.sh
that is work fine

Related

How to find public IP address of MySQL persistent on Openshift 3 web console?

I deployed Wildfly application and MySQL persistent on Openshift web console and tried to connect MySQL persistent with jdbc driver of Eclipse outside. However I can not find the public IP address at all on web console.
How can I find public IP address of MySQL persistent or how to configure the specific IP address into MySQL persistent? I attach an image of both services on Openshift.
[
UPDATED
On Eclipse IDE, I opened the log part of MySQL pod. And I found the IP addresses of MySQL service:
"readinessProbe" : {
"exec" : {"command" : [
"/bin/sh",
"-i",
"-c",
"MYSQL_PWD=\"$MYSQL_PASSWORD\" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'"
]},
"initialDelaySeconds" : 5,
"timeoutSeconds" : 1,
"periodSeconds" : 10,
"successThreshold" : 1,
"failureThreshold" : 3
},
....
"phase" : "Running",
"conditions" : [
{
"type" : "Initialized",
"status" : "True",
"lastTransitionTime" : "2017-04-02T06:35:00Z"
},
{
"type" : "Ready",
"status" : "True",
"lastTransitionTime" : "2017-04-03T16:47:27Z"
},
{
"type" : "PodScheduled",
"status" : "True",
"lastTransitionTime" : "2017-04-02T06:35:00Z"
}
],
"hostIP" : "172.31.14.159",
"podIP" : "10.1.72.72",
"startTime" : "2017-04-02T06:35:00Z",
"containerStatuses" : [{
"name" : "mysql",
"state" : {"running" : {"startedAt" : "2017-04-03T16:47:07Z"}},
"lastState" : {"terminated" : {
"exitCode" : 255,
"reason" : "Error",
"startedAt" : "2017-04-02T06:36:28Z",
....
I tried to connect MySQL pod with the hostIP, 172.31.14.159 or podIP, 10.1.72.72. But connection failed. And then I found the following MySQL generation commands in the log contents:
"exec" : {"command" : [
"/bin/sh",
"-i",
"-c",
"MYSQL_PWD=\"$MYSQL_PASSWORD\" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'"
]},
So, I tried to connect the mysql database service with the ip 127.0.0.1. And the connection was SUCCESSFUL.
Now I am confused what this 127.0.0.1 address is, my local PC or MySQL pod of Openshift container. How can I generate MySQL persistent with the HostIP, not with 127.0.0.1? I am afraid I missed the some procedure.
Your mysql pod havn't a public ip address, but you can use port forwarding.
With Eclipse:
How-To:
https://blog.openshift.com/getting-started-eclipse-jboss-tools-openshift-online-3/
Download:
http://marketplace.eclipse.org/content/jboss-tools-luna
With Openshift CLI:
$ oc port-forward <pod> [<local_port>:]<remote_port> [...[<local_port_n>:]<remote_port_n>]
such as
$ oc port-forward <pod> 3306:5000
Now you can connect the URL jdbc:mysql://127.0.0.1:5000/database. The mysql pod listen to on your local port 5000.
https://docs.openshift.com/container-platform/3.3/dev_guide/port_forwarding.html

userData are not getting executed after launching the instance by cloudformation

I created a AWS cloudformation, which creates a launch configuration and an autoscaling group. In the user Data in the launch Config I have configured the file system mount target, and I installed the cloudwatch agent:
Code EDITED
"LaunchConfig":{
"Type":"AWS::AutoScaling::LaunchConfiguration",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"files" : {
"/etc/cwlogs.cfg": {
"content": { "Fn::Join" : ["", [
"[general]",
"state_file = /var/awslogs/state/agent-state",
"[/var/log/syslog]",
"file = /tmp/",
"log_group_name = ecs-dataloader",
"log_stream_name = ECS-loader",
"datetime_format = %b %d %H:%M:%S"
]]},
"mode": "000755",
"owner": "root",
"group": "root"
},
"/etc/ecs/ecs.config": {
"content": { "Fn::Join" : ["", [
"ECS_CLUSTER=", { "Ref" : "ClusterName" }
]]},
"mode": "000755",
"owner": "root",
"group": "root"
}
},
"commands": {
"Update": {
"command": "yum -y update"
},
"InstallNfs":{
"command": "yum -y install nfs-utils"
},
"CreatFolder": {
"command": "mkdir -p /efs-mount-point/"
},
"EditPerms": {
"command": "chown ec2-user:ec2-user /efs-mount-point/"
},
"MountPoint": {
"command": { "Fn::Join" : ["", [
"AZ=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)\n",
"echo LABEL=cloudimg-rootfs / ext4 defaults,discard 0 0\n",
"$AZ.",{ "Fn::FindInMap" : [ "FileSystemMap", {"Ref" : "EnvParam"}, "FileSystemID"] },
".efs.",{ "Ref" : "AWS::Region" },".amazonaws.com:/ /efs-script-import-tmp nfs4 nfsvers=4.1 0 0 >> /etc/fstab"
]]}
},
"Mount": {
"command": "mount -a -t nfs4"
},
"CloudWatchAgent": {
"command": { "Fn::Join" : ["", [
"curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O\n",
"python ./awslogs-agent-setup.py --region ",{"Ref" : "AWS::Region"},"\n",
"chmod +x ./awslogs-agent-setup.py ./awslogs-agent-setup.py -n -r",
{"Ref" : "AWS::Region"}," -c /etc/cwlogs.cfg"
]]}
}
},
"services" : {
"sysvinit" : {
"awslogs" : { "enabled" : "true", "ensureRunning" : "true" }
}
}
}
}
},
"Properties":{
"ImageId":{ "Fn::FindInMap":[ "AWSRegionToAMI", { "Ref":"AWS::Region" }, "AMIID" ] },
"SecurityGroups":[ { "Ref":"EcsSecurityGroup" } ],
"InstanceType": {"Ref":"InstanceType" },
"IamInstanceProfile":{ "Ref":"EC2InstanceProfile" },
"KeyName":{ "Fn::FindInMap" : [ "KeyPairMapping", {"Ref" : "EnvParam"}, "Key"] },
"UserData":{ "Fn::Base64" : {
"Fn::Join" : ["", [
"#!/bin/bash -xe\n",
"/opt/aws/bin/cfn-init -v ",
" --stack ", { "Ref": "AWS::StackName" },
" --resource LaunchConfig",
" --region ", { "Ref" : "AWS::Region" },"\n"
]]}
}
}
}
The image details : "eu-west-1": { "AMIID":"ami-ba346ec9" },
After running the template, the resources got created successfuly. So I connected to my instance that got created by the autoscaling group via SSH to see if the userData was properly run and set.
Unfortunately, After checking, this is what I found in the /etc/fstab file:
$ cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults,discard 0 0
$ cat /etc/ecs/ecs.config
cat: /etc/ecs/ecs.config: No such file or directory
The instance is not connected to the file system, the file that I tried to create in the cloudformation::init /etc/cwlogs.cfg does not exist either (it's the cloudwatch agent config file) . Can any one tell me what is wrong in the user data that it didn't get executed?
I tried to check the log files but :
$ cat /var/log/cfn-init.log
cat: /var/log/cfn-init.log: No such file or directory
What is the problem here ?
EDIT
$ cat /var/log/cloud-init-ouput.log
...
Cloud-init v. 0.7.6 running 'modules:final' at Fri, 17 Feb 2017 11:43:42 +0000. Up 44.66 seconds.
+ yum install -y aws-cfn-bootstrap/opt/aws/bin/cfn-init -v --stack Mystack --resource LaunchConfig --region eu-west-1
Loading "priorities" plugin
Loading "update-motd" plugin
Config time: 0.009
Command line error: no such option: --stack
Feb 17 11:43:43 cloud-init[2814]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Feb 17 11:43:43 cloud-init[2814]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Feb 17 11:43:43 cloud-init[2814]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 0.7.6 finished at Fri, 17 Feb 2017 11:43:43 +0000. Datasource DataSourceEc2. Up 45.18 seconds
User Data log files are located at:
Linux cloud-init: /var/log/cloud-init.log
Windows EC2Config: C:\cfn\log\cloud-init.log
Check to see whether anything is in the log file. If not, then something's wrong with passing the User Data script from the template. (Why do you have the initial empty quotes in the Join?)
cfn-init is only installed by default on Amazon Linux AMI, so if you're using any other Image ID to launch your EC2 instance you need to ensure that it's installed correctly before invoking it. See my previous answer to the question, "Installing packages using apt-get in CloudFormation file" for more info.
Here is how I resolved the problem: I Update the cloud-init in the user data before calling the meta-data and instead of installing the cloudwatch agent in the metadata, I did in the userdata.
"UserData":{ "Fn::Base64" : {
"Fn::Join" : ["", [
"#!/bin/bash -xe\n",
"yum -y install aws-cfn-bootstrap\n",
"/opt/aws/bin/cfn-init -v",
" --stack ", { "Ref": "AWS::StackName" },
" --resource LaunchConfig",
" --region ", { "Ref" : "AWS::Region" },"\n",
"# Get the CloudWatch Logs agent\n",
"wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py\n",
"# Install the CloudWatch Logs agent\n",
"python ./awslogs-agent-setup.py -n -r ", { "Ref" : "AWS::Region" }, " -c /etc/cwlogs.cfg || error_exit 'Failed to run CloudWatch Logs agent setup'\n",
"service awslogs start"
]]}

How to index couchdb from elasticsearch server with the help of elasticsearch river plugin and hence get JSON data

I am working on graphical representation of data. The graph accepts JSON data,hence I need to fetch the required data from couchdb. I am using elasticsearch server for indexing couchdb and hence retrieve required data.
I am using elasticsearch river plugin to make couchdb and elasticsearch server together.
I have Created the CouchDB Database 'testdb' and created some test documents for the same.
Setup elasticsearch with the database.
On testing the same by writing CURl GET command with default search criteria, we must get 'total hits' more than 0 and the 'hits' must have some response value for searched criteria.
But we are getting 'total hits' as 0 and 'hits':[] (i.e. null)
Procedures I followed.
1) Downloaded and installed couchdb latest version
2) Verified CouchDB is running
curl localhost:5984
I got response that starts with:
{"couchdb":"Welcome"...
3) Downloaded ElasticSearch and installed service
service.bat install
curl http://127.0.0.1:9200
I got response as
{ "ok" : true, "status" : 200,.....
4) Installed the CouchDB River Plugin for ElasticSearch 1.4.2
plugin -install elasticsearch/elasticsearch-river-couchdb/2.4.1
5) To Create the CouchDB Database and ElasticSearch Index
curl -X PUT "http://127.0.0.1:5984/testdb"
6) To Create some test documents:
curl -X PUT "http://127.0.0.1:5984/testdb/1" -d "{\"name\":\"My
Name 1\"}"
curl -X PUT "http://127.0.0.1:5984/testdb/2" -d
"{\"name\":\"My Name 2\"}"
curl -X PUT
"http://127.0.0.1:5984/testdb/3" -d "{\"name\":\"My Name 3\"}"
curl
-X PUT "http://127.0.0.1:5984/testdb/4" -d "{\"name\":\"My Name 4\"}"
7) To Setup ElasticSearch with the Database
curl -X PUT "127.0.0.1:9200/_river/testdb/_meta" -d "{ \"type\" :
\"couchdb\", \"couchdb\" : { \"host\" : \"localhost\", \"port\" :
5984, \"db\" : \"testdb\", \"filter\" : null }, \"index\" : {
\"index\" : \"testdb\", \"type\" : \"testdb\", \"bulk_size\" :
\"100\", \"bulk_timeout\" : \"10ms\" } }"
8) To test it
curl "http://127.0.0.1:9200/testdb/testdb/_search?pretty=true"
on testing we should get this
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : 1.0,
"hits" : [ {
"_index" : "testdb",
"_type" : "testdb",
"_id" : "4",
"_score" : 1.0, "_source" : {"_rev":"1-7e9376fc8bfa6b8c8788b0f408154584","_id":"4","name":"My Name 4"}
}, {
"_index" : "testdb",
"_type" : "testdb",
"_id" : "1",
"_score" : 1.0, "_source" : {"_rev":"1-87386bd54c821354a93cf62add449d31","_id":"1","name":"My Name"}
}, {
"_index" : "testdb",
"_type" : "testdb",
"_id" : "2",
"_score" : 1.0, "_source" : {"_rev":"1-194582c1e02d84ae36e59f568a459633","_id":"2","name":"My Name 2"}
}, {
"_index" : "testdb",
"_type" : "testdb",
"_id" : "3",
"_score" : 1.0, "_source" : {"_rev":"1-62a53c50e7df02ec22973fc802fb9fc0","_id":"3","name":"My Name 3"}
} ]
}
}
But I got something like this
{
"error" : "IndexMissingException[[testdb] missing]",
"status" : 404
}
This curl string doesn't need the additional testb. This:
curl "http://127.0.0.1:9200/testdb/testdb/_search?pretty=true"
Should be this:
curl 'http://localhost/testdb/_search?pretty=true'
You can view all your by running the following and ensuring your search is against one of your indices:
curl -X GET 'localhost:9200/_cat/indices'

Error will be Raised when parse data into elasticsearch from mysql

curl -XPUT localhost:9200/_river/my_jdbc_river/_meta -d
"{
"type" : "jdbc",
"jdbc" :
{
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://localhost:3306/springtest",
"user" : "root",
"password" : "root",
"sql" : "select * from register",
"index" : "my_register",
"type" : "my_register_type"
}
}"
the connection successfull but error tab will generated like
error: NoClassSettingsException[Failed to load class with value [jdbc]]; nested:
ClassNotFoundException[jdbc];
I presume you need to add
"driver": "com.mysql.jdbc.Driver"
to your jdbc definition.
Also, check that you have done all the steps mentioned there: https://github.com/jprante/elasticsearch-river-jdbc/wiki/Quickstart
(especially pp.4 and 5, related to registering MySQL driver with your ElasticSearch instance)

"IndexMissingException[[jdbc] missing]", "status" : 404

i want to index and search mysql database using elastic search & I followed this tutorial
elasticsearch-river-jdbc
At first i downloaded elastic search and installed river-jdbc in its plugin folder.
then added mysql-jdbc inside ES_HOME/elasticsearch-0.90.1/plugins/river-jdbc/
Then started elasticsearch and Started another terminal window,
and created a new JDBC river with name my_jdbc_river with this curl command
curl -XPUT 'localhost:9200/_river/my_jdbc_river/_meta' -d '{
"type" : "jdbc",
"jdbc" : {
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://localhost:3306/bablool",
"user" : "root",
"password" : "babloo",
"sql" : "select * from details"
},
"index" : {
"index" : "jdbc",
"type" : "jdbc"
}
}'
then when I run this command: curl -XGET 'localhost:9200/jdbc/jdbc/_search?pretty&q=*'
and im getting following error:
"error": "IndexMissingException[[jdbc] missing]", "status" : 404
plz help me....
I'm bumped into the same error following the tutorial mentioned in the question... after reading again the documentation, I noticed that I forgot to restart the ES after install the river_jdbc plugin.
This tutorial is also interesting if you starting to learn about indexing SQL databases.