UnavailableShardsException - mysql

I want to index and search mysql database using elasticsearch & I followed this tutorial
https://github.com/jprante/elasticsearch-river-jdbc/wiki/Quickstart
At first I downloaded elasticsearch and installed river-jdbc in its plugin folder. then added mysql-jdbc inside ES_HOME/plugins/river-jdbc/ Then started elasticsearch and Started another terminal window, and created a new JDBC river with name my_jdbc_river with this curl command
curl -XPUT 'localhost:9200/_river/my_jdbc_river/_meta' -d '{
"type" : "jdbc",
"jdbc" : {
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://localhost:3306/bablool",
"user" : "root",
"password" : "babloo",
"sql" : "select * from details"
},
"index" : {
"index" : "jdbc",
"type" : "jdbc"
}
}'
I'm getting the following error:-
then when I run this command: curl -XGET 'localhost:9200/jdbc/jdbc/_search?pretty&q=*'
and I'm getting following error:
"error": "IndexMissingException[[jdbc] missing]", "status" : 404
And when I give this in my browser:
http://localhost:9201/_search?q=*
Im getting like this:
{
"took": 51,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.0,
"hits": [
{
"_index": "_river",
"_type": "my_jdbc_river",
"_id": "_meta",
"_score": 1.0,
"_source": {
"type": "jdbc",
"jdbc": {
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://localhost:3306/bablool",
"user": "root",
"password": "babloo",
"sql": "select * from details"
},
"index": {
"index": "jdbc",
"type": "jdbc"
}
}
}
]
}
}
Is mysql dB indexed? How can I Search in my Db?

I encountered similar problem and this is how I managed to solve the issue:
Firstly, I checked all the indices via http://localhost:9200/_cat/indices?v
Deleted all indices with health status as red (there was just one index _river with health status red)
This is how you delete in index curl -XDELETE 'localhost:9200/_river/'
Redo step 7 in link you mentioned https://github.com/jprante/elasticsearch-river-jdbc/wiki/Quickstart
Hope it solves your problem as well :) Good luck!!

Related

Why is my JSON request body showing as invalid when earlier, a similar request is working?

I have the following code snippet where I'm trying to send notification data to a discord webhook, but it is returning {"code": 50109, "message": "The request body contains invalid JSON."}%.
Here is the non-working code (bolded the JSON request part):
if [ "$DISCORD_WEBHOOK_URL" != "" ]; then
rclone_sani_command="$(echo $rclone_command | sed 's/\x1b\[[0-9;]*[a-zA-Z]//g')" # Remove all escape sequences
# Notifications assume following rclone ouput:
# Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 0.0s
transferred_amount=${rclone_sani_command#*Transferred: }
transferred_amount=${transferred_amount%% /*}
** send_notification() {
output_transferred_main=${rclone_sani_command#*Transferred: }
output_transferred_main=${output_transferred_main% Errors*}
output_errors=${rclone_sani_command#*Errors: }
output_errors=${output_errors% Checks*}
output_checks=${rclone_sani_command#*Checks: }
output_checks=${output_checks% Transferred*}
output_transferred=${rclone_sani_command##*Transferred: }
output_transferred=${output_transferred% Elapsed*}
output_elapsed=${rclone_sani_command##*Elapsed time: }
notification_data='{
"username": "'"$DISCORD_NAME_OVERRIDE"'",
"avatar_url": "'"$DISCORD_ICON_OVERRIDE"'",
"content": null,
"embeds": [
{
"title": "Rclone Local Backup: Finished!",
"color": 65408,
"fields": [
{
"name": "Directories synced",
"value": "'"$SOURCE_DIR"' to '"$DESTINATION_DIR"'"
},
{
"name": "Transferred",
"value": "'"$output_transferred_main"'"
},
{
"name": "Errors",
"value": "'"$output_errors"'"
},
{
"name": "Checks",
"value": "'"$output_checks"'"
},
{
"name": "Transferred",
"value": "'"$output_transferred"'"
},
{
"name": "Elapsed time",
"value": "'"$output_elapsed"'"
},
{
"name": "Finished",
"value": "Finished backup on '"$END_DATE"' at '"$END_TIME"'"
},
],
"thumbnail": {
"url": null
}
}
]
}'
curl -H "Content-Type: application/json" -d "$notification_data" $DISCORD_WEBHOOK_URL
}**
if [ "$transferred_amount" != "0" ]; then
send_notification
fi
fi
rm -f "$LOCK_FILE"
trap - SIGINT SIGTERM
exit
fi
Earlier in the same script, the following code snippet does correctly send a notification to discord:
touch "$LOCK_FILE"
echo "Starting to backup $SOURCE_DIR to $DESTINATION_DIR on $START_DATE at $START_TIME" | tee -a $LOG_FILE
**send_notification_start() {
notification_data='{
"username": "'"$DISCORD_NAME_OVERRIDE"'",
"avatar_url": "'"$DISCORD_ICON_OVERRIDE"'",
"content": null,
"embeds": [
{
"title": "Rclone Local Backup: Started!",
"color": 4094126,
"fields": [
{
"name": "Started",
"value": "Started backup on '$START_DATE' at '$START_TIME'"
},
{
"name": "Directories being synced",
"value": "'$SOURCE_DIR' to '$DESTINATION_DIR'"
}
],
"thumbnail": {
"url": null
}
}
]
}'
curl -H "Content-Type: application/json" -d "$notification_data" $DISCORD_WEBHOOK_URL
}**
I've tried messing with the quotes and other characters to see if that is the issue but I am unable to figure out why this JSON request body is incorrect.
Running either of those two JSON snippets themselves (bolded) through a JSON validator I just get that it is incorrectly parsing right the beginning (even though the latter is working fine).
I am pretty new to using JSON but I wanted to just get a notification sent to discord when my script finishes backing up with some details on what exactly occurred.

Facing Issues while adding custom JSON

Hi I am new to solr and trying to add an custom JSON.I am following the link https://cwiki.apache.org/confluence/display/solr/Transforming+and+Indexing+Custom+JSON. First I created a core using below command.solr create -c my_collection -d data_driven_schema_configs. after that I added the below content from cygwin to my core.But when I am enquiring through http://localhost:8983/solr/my_collection/select?wt=json&q=* it is saying no doc.{"responseHeader":{"status":0,"QTime":1,"params":{"q":"*","wt":"json"}},"response":{"numFound":0,"start":0,"docs":[]}}
curl 'http://localhost:8983/solr/my_collection/update/json/docs'\
'?split=/exams'\
'&f=first:/first'\
'&f=last:/last'\
'&f=grade:/grade'\
'&f=subject:/exams/subject'\
'&f=test:/exams/test'\
'&f=marks:/exams/marks'\
-H 'Content-type:application/json' -d '
{
"first": "John",
"last": "Doe",
"grade": 8,
"exams": [
{
"subject": "Maths",
"test" : "term1",
"marks" : 90},
{
"subject": "Biology",
"test" : "term1",
"marks" : 86}
]
}'
The query should be *:*, not just *. Just check in Admin UI and do default search.

How to index couchdb from elasticsearch server with the help of elasticsearch river plugin and hence get JSON data

I am working on graphical representation of data. The graph accepts JSON data,hence I need to fetch the required data from couchdb. I am using elasticsearch server for indexing couchdb and hence retrieve required data.
I am using elasticsearch river plugin to make couchdb and elasticsearch server together.
I have Created the CouchDB Database 'testdb' and created some test documents for the same.
Setup elasticsearch with the database.
On testing the same by writing CURl GET command with default search criteria, we must get 'total hits' more than 0 and the 'hits' must have some response value for searched criteria.
But we are getting 'total hits' as 0 and 'hits':[] (i.e. null)
Procedures I followed.
1) Downloaded and installed couchdb latest version
2) Verified CouchDB is running
curl localhost:5984
I got response that starts with:
{"couchdb":"Welcome"...
3) Downloaded ElasticSearch and installed service
service.bat install
curl http://127.0.0.1:9200
I got response as
{ "ok" : true, "status" : 200,.....
4) Installed the CouchDB River Plugin for ElasticSearch 1.4.2
plugin -install elasticsearch/elasticsearch-river-couchdb/2.4.1
5) To Create the CouchDB Database and ElasticSearch Index
curl -X PUT "http://127.0.0.1:5984/testdb"
6) To Create some test documents:
curl -X PUT "http://127.0.0.1:5984/testdb/1" -d "{\"name\":\"My
Name 1\"}"
curl -X PUT "http://127.0.0.1:5984/testdb/2" -d
"{\"name\":\"My Name 2\"}"
curl -X PUT
"http://127.0.0.1:5984/testdb/3" -d "{\"name\":\"My Name 3\"}"
curl
-X PUT "http://127.0.0.1:5984/testdb/4" -d "{\"name\":\"My Name 4\"}"
7) To Setup ElasticSearch with the Database
curl -X PUT "127.0.0.1:9200/_river/testdb/_meta" -d "{ \"type\" :
\"couchdb\", \"couchdb\" : { \"host\" : \"localhost\", \"port\" :
5984, \"db\" : \"testdb\", \"filter\" : null }, \"index\" : {
\"index\" : \"testdb\", \"type\" : \"testdb\", \"bulk_size\" :
\"100\", \"bulk_timeout\" : \"10ms\" } }"
8) To test it
curl "http://127.0.0.1:9200/testdb/testdb/_search?pretty=true"
on testing we should get this
{
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : 1.0,
"hits" : [ {
"_index" : "testdb",
"_type" : "testdb",
"_id" : "4",
"_score" : 1.0, "_source" : {"_rev":"1-7e9376fc8bfa6b8c8788b0f408154584","_id":"4","name":"My Name 4"}
}, {
"_index" : "testdb",
"_type" : "testdb",
"_id" : "1",
"_score" : 1.0, "_source" : {"_rev":"1-87386bd54c821354a93cf62add449d31","_id":"1","name":"My Name"}
}, {
"_index" : "testdb",
"_type" : "testdb",
"_id" : "2",
"_score" : 1.0, "_source" : {"_rev":"1-194582c1e02d84ae36e59f568a459633","_id":"2","name":"My Name 2"}
}, {
"_index" : "testdb",
"_type" : "testdb",
"_id" : "3",
"_score" : 1.0, "_source" : {"_rev":"1-62a53c50e7df02ec22973fc802fb9fc0","_id":"3","name":"My Name 3"}
} ]
}
}
But I got something like this
{
"error" : "IndexMissingException[[testdb] missing]",
"status" : 404
}
This curl string doesn't need the additional testb. This:
curl "http://127.0.0.1:9200/testdb/testdb/_search?pretty=true"
Should be this:
curl 'http://localhost/testdb/_search?pretty=true'
You can view all your by running the following and ensuring your search is against one of your indices:
curl -X GET 'localhost:9200/_cat/indices'

Lost results in ElasticSearch index with autoupdate

I have problem with data being lost when using a jdbc driver to update elasticsearch from a mysql database. My river is below:
curl -XPUT 'http://localhost:9200/_river/river_index_type/_meta' -d '
{
"type": "jdbc",
"jdbc": {
"strategy": "simple",
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://localhost/tt",
"user": "user",
"password": "password",
"sql": "SELECT p.product_id AS _id, ... FROM product p ... WHERE ...",
"poll": "5m",
"autocommit": true
},
"index": {
"type": "type",
"index": "index"
}
}
Initially everything works fine, but later, instead of 1200 results in my index i only have 800-900, and each five minutes the count changes. I don't understand what the problem is. Any help would be appreciated.

How to update multiple documents in Solr with JSON?

How to update multiple documents in Solr 4.5.1 with JSON? I tried this but it does not work:
POST /solr/mycore/update/json:
{
"commit": {},
"add": {
"overwrite": true,
"doc": [{
"thumbnail": "/images/404.png",
"url": "/404.html?1",
"id": "demo:/404.html?1",
"channel": "demo",
"display_name": "One entry",
"description": "One entry is not enough."
}, {
"thumbnail": "/images/404.png",
"url": "/404.html?2",
"id": "demo:/404.html?2",
"channel": "demo",
"display_name": "Another entry",
"description": "Another entry is required."
}
]
}
}
Solr expects one "add"-key in the JSON-structure for each document (which might seem weird, if you think about the original meaning of the key in the object), since it maps directly to the XML format when doing the indexing - and this way you can have metadata for each document by itself.
{
"commit": {},
"add": {
"doc": {
"id": "321321",
"name": "barfoo"
}
},
"add": {
"doc": {
"id": "123123",
"name": "Foobar"
}
}
}
.. works. I think allowing an array as the element referenced by "add" would make more sense, but I haven't dug further into the source or know the reasoning behind this.
I understand that (at least) from versions 4.0 and older of solr, this has been fixed. Look at http://wiki.apache.org/solr/UpdateJSON.
In ./exampledocs/books.json there is an example of a json file with multiple documents.
[
{
"id" : "978-0641723445",
"cat" : ["book","hardcover"],
"name" : "The Lightning Thief",
"author" : "Rick Riordan",
"series_t" : "Percy Jackson and the Olympians",
"sequence_i" : 1,
"genre_s" : "fantasy",
"inStock" : true,
"price" : 12.50,
"pages_i" : 384
}
,
{
"id" : "978-1423103349",
"cat" : ["book","paperback"],
"name" : "The Sea of Monsters",
"author" : "Rick Riordan",
"series_t" : "Percy Jackson and the Olympians",
"sequence_i" : 2,
"genre_s" : "fantasy",
"inStock" : true,
"price" : 6.49,
"pages_i" : 304
},
...
]
While #fiskfisk answer is still a valid JSON, it is not easy to be serializable from a data structure. This one is.
elachell is correct that the array format will work if you are just adding documents with the default settings. Unfortunately, that won't work if, for instance, you need to add a custom boost to some of the documents or change the overwrite setting. You then have to use the full object structure with an "add" key for each of them, which as they pointed out, makes this frustratingly annoying to try to serialize from most languages which don't allow the same key more than once in an object:
{
"commit": {},
"add": {
"doc": {
"id": "321321",
"name": "barfoo"
},
"boost": 2.0
},
"add": {
"doc": {
"id": "123123",
"name": "Foobar"
},
"boost": 1.5,
"overwrite": false
}
}
Update for SOLR 8.8 (and maybe lower).
The following JSON works for /update/json:
{
'add': [
{'id': '123', 'field1': 'foo'},
{'id': '124', 'field1': 'foo'}
],
'delete': ['111', '106']
}
Another option if you are on Solr 4.10 or later is to use a custom JSON structure and tell Solr how to index it (not sure how to add boosts with this method either, but it's a nice option if you already have a data struct in JSON and don't want to convert it over to Solr's format). Here's the Solr documentation on this option:
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-TransformingandIndexingCustomJSON