I would like to get some feedback from anyone who's had experience indexing MySQL data with ElasticSearch for full-text searching. How did you accomplish this? I've been researching this a bit and unfortunately I've noticed that ElasticSearch has no official plugin to accomplish this although I've come across three different 3rd party tools:
elasticsearch-river-jdbc
go-mysql-elasticsearch
elasticsearch-river-mysql
I'm unsure which one would be best in terms of performance although I suspect the Go tool might have an advantage due to it's compiled nature and the fact that it uses the mysql binary logs. I would appreciate any advice or examples anyone could provide me with.
Thanks!
UPDATE:
you can now use Logstash to do it..here is an useful blog about it click ere
well you can use mysql UDF functions in mysql triggers to execute external scripts, which will index the newly updated or inserted data in elasticsearch. thats one way. check this to see how to's on mysql UDF
Related
I want to implete the full text search(FTS) queries in my node js application. The database I am using is MySQL. I know that MySQL does have inbuild support for FTS but sadly it does not support Singular/Plurals, Synonyms and Inflectional words.
There other FTS libraries available that can work with MySQl. Following are the two I am interested in
Lucene
Sphinx Search
I am very much sure that Shpinx Search has npm package and can be used with node js. I am not sure if Lucene can be used with node js ?
Please let me know if lucene can be used with node js if so provide the documentation for the same.
Thanks !
You have several alternative like query-engine and several other tools available for Lucene.
Also, if you want to use FTS with node, you could have a look to Norch like suggest this answer on a look-alike topic.
Best,
Maintainer of search-index and Norch here. They might be what you are looking for. You can even use your MySQL database as a back end if you want.
https://github.com/fergiemcdowall/search-index (the lib)
https://github.com/fergiemcdowall/norch (the server)
We have a rails app that has a MySQL backend, each client has one DB and the schema is identical. We use a custom gem to change the DB based on the URL of the request (This is some legacy code that we are trying to move away from)
We need to capture some changes from those MySQL databases (Changes in inventory, some order information, etc) transform and store in a single MongoDB database (multitenant data store), this data will be used for analytics at first, but our idea is to move everything there.
There was something in place to do this, using AR callbacks and Rabbit, but to be honest it wasn't working correctly and it looked like it was more trouble to fix it than to start over with a fresh approach.
We did some research and found some tools to do ETL but they are overkill for our needs.
Does anyone have some experience with a similar problem?
Recommendations on how to architect and implement this simple ETL
Pentaho provides change-data-capture option which can solve Data-synchronization problems.
If by Overkill you mean Setup, Configuration, then Yes that is the common problem with ETL tools and PENTAHO is the easiest among them.
If you can provide more details, I'll be glad to provide an elaborate answer.
I don't know completely what I want, but surely someone has had the same need, and has solved it in a far better manner than I could:
I'm looking for some mechanism to extract the data definition of a mySQL table from the database and allow it to be queried for the list of columns and their definitions, as part of a routine to dynamically construct DML? It would also be good to have the table parameters (e.g. ENGINE, INDEX, etc.) available, too.
Our databases aren't particularly advanced, and I certainly don't have an encyclopedic knowledge of SQL DDL, so what I came up with probably wouldn't be of much use to anyone else. Is there something already out there in Perl - preferably object-oriented - to do this, at least for mySQL?
Yes, there's a Perl package SQL::Translator, part of a toolset called SQLFairy. It parses SQL DDL from an SQL script or from a live database instance. It supports several RDBMS, including MySQL.
Then it offers tools to do schema conversions, schema diffs, and a bunch of other cool stuff.
http://metacpan.org/pod/SQL::Translator
http://sqlfairy.sourceforge.net/
I found the docs are better than most Perl projects, but still I had to read the code to understand how to use it in the way I wanted to.
The DBI interface has a set of "Catalog Methods": http://metacpan.org/pod/DBI#Catalog-Methods.
There is a similar StackOverflow question you can look at: How do I get schemas from Perl's DBI?
Does anyone know of a quick and easy test to see if a query is properly formatted for both MySQL & MSSQL. Perhaps other database types as well, such as SQL Server? I only have access to MySQL at this time.
Info: I'm working on an Open Source project called JJWDesign Google Maps for SugarCRM. Some of the queries use the SugarCRM classes; others I have to write custom. For example, some are special distance calculations against the geocode information stored in the tables.
http://www.sugarforge.org/projects/jjwgooglemaps/
More importantly, while there is an accepted syntax, each flavour of database has it's own specific functions, features and things you can do.
The best you can do is to make do with the most basic of features. Oracle has different functions for datetime compared to mysql compared to db2. While I would love to assist in a 'free as in beer' project, you really will need to check each function to see if it is the same across all major vendors. General functions most often are, so abs() will be fairly consistent, but others simply won't.
You're talking about a SQL parser so by definition it either isn't going to be quick and easy or it will do only the simplest checking.
Each RDBMS has its own flavour of SQL too so you'd really be limited to testing whether it was ANSI SQL.
In this thread someone pointed me to use sqlalchemy-migrate to help with a fast-changing web application using sqlalchemy.
However a Do It Yourself method was also recommended consisting in manually writing CSV columns for the new database schema, and finally import them.
The problem is that I can't find real-world examples of sqlalchemy-migrate. Ressources that I have found at best decribe adding a single column or a column rename. The official documentation essentially describes the API and it's hard to see how to use migrate effectively. From the doc I cannot even know if migrate could help changing the database engine, from sqlite to mysql for example, while the DIY solution would to the job.
I really want to see code that would make some non-trivial transformations of a database schema and proving that migrate is really a useful tool.
Where can I find good examples/tutorials for sqlalchemy-migrate ?
Thanks !
Don't forget about google code search when looking for real work examples of code. For instance the follow search:
http://www.google.com/codesearch?hl=en&lr=&q=%22from+migrate+import%22+lang:python&sbtn=Search
Will pull up a number of real migration scripts. It basically looks for Python files with "from migrate import" in the file.
Work through some of these and see if you can follow what they're doing.