I didn't find exactly what I'm looking for and no one to explain me, so I ask here. Hope this is not a duplicate…
I have an application that run on NodeJS with the ORM sequelize and a MySQL database. The project start one year ago.
I would like, now, to increase performances etc, install ElasticSearch. But I'm not sure if I can.
Can I just tell my elasticserver, where my database is, and magically, it does the mapping alone ?
Or Have I to insert my data on elasticSearch ?
All my infrastructure is hosted on AWS, I'm using RDS. May be it will help ?
In fact I don't know where to put the elasticSearch layer ? Do you have any ideas to help me.
Because, I already worked with Symfony and the FosElasticaBundle, but it works pretty well that I don't know how.
UDPATE:
you can use the ES http api
your existing data in your database can you bulk index
You have to handle changes in your database, so you can use the observe pattern, such as created(), saved(), deleted(), so you can in these methods handle the ES actions to created, update or delete a document
Elasticsearch (ES) is a speed machine with a lot of possibilities (advanced queries, stats, etc.), not a database (read why here: https://www.elastic.co/blog/found-elasticsearch-as-nosql).
It's distributed (can work/share work via multiple servers)
It's document based (json documents, for example with nested, parent/child relationships)
It's mapping based
It's indexing based
More here: https://www.elastic.co/products/elasticsearch
You can use types in ES (like tables in your database).
ES uses indexed data, based on the mapping. Therefore, you still need a database and index the data (denormalized) to ES. You can query, filter, aggregate, analyze the data in your index.
Related
I have a wordpress website with mysql database of real estate properties listed for sale and an elasticsearch database of addresses I have curated.
Can the mysql query the elasticsearch database to find if there is a matching address then send back to wordpress to move the property into a "xyz" category?
If not does anyone know a way for this type of process to happen?
Thanks
I don't know of any way MySQL can interact directly with ElasticSearch.
Typically you would develop code to interact with both ElasticSearch and MySQL in turn. This would be done with two separate queries. The first to query ElasticSearch to find a matching address, and then a second query against MySQL, using the information returned by the first query as a parameter for the MySQL query.
WordPress in this context is a client application for both MySQL and ElasticSearch. But each has its own protocol and connector. MySQL and ElasticSearch don't share data or queries.
Update: I found a way to use an ElasticSearch data source via MySQL's FEDERATED engine through a proxy that translates MySQL's protocol into ODBC to the ElasticSearch node. This is probably way too complex for your needs, but it is possible. See https://www.cdata.com/kb/tech/elasticsearch-odbc-mysql-federated-tables.rst
By analogy, this is like asking "can I drive my car while making my car drive another car?" Yes, you can — just weld your car's steering rods to some long metal struts that operate the hands of a puppet version of you sitting in the other car.
I don't recommend it.
I am new to Elasticsearch. I was using MySQL Full Text features till now.
I want my MySQL database as my primary database and want to use Elasticsearch alongside as a search engine in my website. I got several problems when thinking about it. The main problem is Syncing between MySQL database and Elastic search.
Some say to use Logstash. But even though I use it, would I need to write separate functions in my program to database transactions and Elasticsearch indexing?
You will need to run periodic job doing full reindex and/or send individual document updates for ES indexing. Logstash sounds like ill-suited thing for the purpose. You need just the usual ES API to index stuff.
I am using MySQL to store all of my data.
My application is an e-commerce application developed in Node.JS. It has many tables ( users, sellers, stores, products, productoptions etc. )
My goal now is to implement the ability to search for products and stores. I have been doing a lot of research on Search Engines, Full Text Searches etc.
After quite a lot of research, I decided to go with ElasticSearch to handle searching in my application. Now, I am well aware ElasticSearch has many features, but I want to know how to utilize them best in this kind of application.
In my opinion, I have 2 options:
Use MySQL alongside ElasticSearch. This, at least to me, sounds like the best option. The problem with this is synchronizing MySQL and ES. I did a lot of research about this topic, and found many methods, such as manually updating ES ( once I perform a CRUD operation on MySQL, do the same thing for ES ), or even use something like go-mysql-elasticsearch. Will these 2 even be good at performing such a task? go-mysql-elasticsearch doesn't seem too hard to implement but I am not sure if it's the way to go.
Use ElasticSearch as the main database. Now, I know ElasticSearch is intended to mainly be used as a Search Engine, but it is indeed able to be used as a database. This would then eliminate the problem of having to sync MySQL and ElasticSearch. The biggest drawback with this method would be the fact that I would have to rewrite a lot of my code to switch from MySQL to ES entirely.
Since this is an e-commerce application the data will be updated often ( adding new products, editing them, registering new users etc. )
So to conclude, which option should I go with?
Is there a third option?
I have tons of data present in MySQL in form of different database, and their respective tables. They all are related to each other. But when I have to do analysis in data, I have to create different scripts, that combine data, merge it and show me as a result, but this takes a lot of time, and effort too. I love elasticsearch for its speed and visualization of data via kibana, therefore I have decided to move my entire MySQL data in real time to elasticsearch, keeping data in MySQL too. But I want a scalable strategy, and process that migrates that data to elasticsearch.
Suggest the best tool, or methods to do the job.
Thank you.
Prior to Elasticsearch 2.x you could write your own Elasticsearch _river plugin that you can install into elasticsearch. You can control how often you want this said data you've munged with your scripts to be pulled in by the _river (Note: this is not truly recommended).
You may also use your favourite Queuing Message Broker tool such as ActiveMQ to push your data into elasticsearch
If you want full control to meet your need for real time migration of data you may also write a simple app that makes use of elasticsearch REST end point, by simply writing to it via REST. You can even do bulk POST
Make use of any of the elasticsearch tools such as beat, logstash that are great at shipping almost any type of data into elasticsearch
For other alternatives of munging your data to a flat file, or if you want to maintain relationships see this post here
Does it make sense to use a combination of MySQL and MongoDB. What im trying to do basically is use MySQl as a "raw data backup" type thing where all the data is being stored there but not being read from there.
The Data is also stored at the same time in MongoDB and the reads happen only from mongoDB because I dont have to do joins and stuff.
For example assume in building NetFlix
in mysql i have a table for Comments and Movies. Then when a comment is made In mySQL i just add it to the table, and in MongoDB i update the movies document to hold this new comment.
And then when i want to get movies and comments i just grab the document from mongoDb.
My main concern is because of how "new" mongodb is compared to MySQL. In the case where something unexpected happens in Mongo, we have a MySQL backup where we can quickly get the app fallback to mysql and memcached.
On paper it may sound like a good idea, but there are a lot of things you will have to take into account. This will make your application way more complex than you may think. I'll give you some examples.
Two different systems
You'll be dealing with two different systems, each with its own behavior. These different behaviors will make it quite hard to keep everything synchronized.
What will happen when a write in MongoDB fails, but succeeds in MySQL?
Or the other way around, when a column constraint in MySQL is violated, for example?
What if a deadlock occurs in MySQL?
What if your schema changes? One migration is painful, but you'll have to do two migrations.
You'd have to deal with some of these scenarios in your application code. Which brings me to the next point.
Two data access layers
Your application needs to interact with two external systems, so you'll need to write two data access layers.
These layers both have to be tested.
Both have to be maintained.
The rest of your application needs to communicate with both layers.
Abstracting away both layers will introduce another layer, which will further increase complexity.
Chance of cascading failure
Should MongoDB fail, the application will fall back to MySQL and memcached. But at this point memcached will be empty. So each request right after MongoDB fails will hit the database. If you have a high-traffic site, this can easily take down MySQL as well.
Word of advice
Identify all possible ways in which you think 'something unexpected' can happen with MongoDB. Then use the most simple solution for each individual case. For example, if it's data loss you're worried about, use replication. If it's data corruption, use delayed replication.