We are looking at ways of improving "search" functionality in our large business application which currently uses SQL Like syntax to do it. So we started evaluating Solr server and were able to index few of our database tables and search. But I am newbie and wanted to know if
1) We have large number of tables in our application. Is there any utility that generates schema xml in solr using the database tables?
2) Our current search lists the database row that meets the search criteria (this was written using SQL 'like' and takes lot of time to generate the search results). We want to simulate the exact functionality using solr. Is that possible?
For importing a database into SOLR, you might want to look into DataImportHandler.
There will be a fair amount of configuration required for it, defining what tables and columns to import, what should be stored, and how it should be indexed.
Related
Is it possible to search for something that is in two databases? For example, I want to do a "starts with" search on a column in Postgres as well as a column in MySQL where one is "name" and one is "email"
Copying over data is not reliable as new data will be created in both databases constantly.
Yes, it is possible. For the "starts with" part, you should be able to use the standard Postgres string functions, of which starts_with is one, and indexing on the desired columns.
Getting the data from MySQL is the more complicated part.
You would most likely want to use a foreign data wrapper (e.g. FDW) from Postgres to access the MySQL data, and then handle the unioning of it (or other desired processing) with the Postgres data for returning the combined data set.
You could write your own FDW if you have particularly specific requirements, or you could try an open source one, such as this one from EnterpriseDB. EnterpriseDB is a Postgres consultancy and offers their own Postgres version, but the doc on the Github page for this says it is compatible with base Postgres as well as their own version.
My Application's data is stored in both MongoDB and MYSQL. Is there any way to write queries in Metabase that will help in fetching data from both MYSQL and Mongo Databases?
I'm assuming here that by "My application's data" you are refering to the data sources, not to the Metabase internal databasa (aka metadata).
As of now (0.32.10), there is no way to query data from two different data sources.
What you can do, though, is to setup two (or more, depending on your need) different questions, and add them to a dashboard - that way, you will be able to show the data in a "data source agnostic" way.
If I didn't not get it wrong, I think this will be somewhat possible in the incoming version (0.33), scheduled for the next weeks (I believe), using the new join system (which will allow you to join data from different data sources, given a certain common key - say, "order_id" or something like that).
You can get more info, and even test the new version (which is currently in the RC2) at
this link.
Intro
I've searched all around about this problem, but I didn't really found a source of knowledge about this, so I'm sorry if this problem seems basic to you, but for me is rather quite intriguing due the fact that I'm having hard time to guess what keywords to use on google in order to retrieve proper info.
Problem Description :
As a matter of fact, i have to issues that i don't know how to deal in a MySQL instance installed in a laptop in a windows environment:
I have a DB in MySQL with 50 tables, of with 15 or 20 tables are tables with original data. The other tables were tables that i generated from the original data tables, in order to properly create tables that would allow me to analyze data in PowerBI. The original data tables are fed by dumps from a ERP Database.
My issue is the following:
How would one automate the process of receiving cumulative txt/csv files (via pen-drive or any other transfer mechanism), store those files into a folder and then update the existing tables with the new information? Is there any reference of best practices to deal with such a scenario?
How can i maintain the good shape of my database with the successive data integration, I mean, how can I make my database scalable and responsive?
Can you point me some sources that would help me with this?
At the moment I imported data into tables, in 2 steps:
1st - I created the table structure with the Workbench import wizard help ( I had to do it this way because the tables have a lot of fields - dozens of them, literally, and those fields need to be in the database). I also inserted primary keys and indexes in those tables;
2nd - I Managed to load the data from the files into those tables, using LOAD DATA IN FILE command.
Some of the fields of the tables created with the import wizard, were created as data type text, with is not necessary in this scenario. I would like to revert those fields to data type NVARCHAR(255) or something, However there are a lot of field to alter the data type and in multiple tables at this point, and i was wondering if i can write a query to do the job of creating all the ALTER TABLES statements i need.
So my issue here is: is it safe to alter the data type in multiple fields in multiple columns (in this case i would like to change fields with datatype text to NAVARCHAR(255))? What is the best way to do this? Can you point me to some sources or best practices for this, please?
Thank you, in advance, for your help.
Cheers
You need a scripting language, not a UI. See mysql commandline tool, the shell of your OS, etc, etc.
DROP DATABASE and reCREATE it
LOAD DATA
Massage the data to get the columns cleaner than what the load data provided
Sic the BI tool on the data.
If you want to discuss Step 3, we need details about what transformations are needed between step 2 and step 4. That includes providing the format or schema for steps 2 and 4.
Anyone who have worked on elastic search. need input as mentioned in subject. I have one requirement in which sql data is migrated to elastic Search now there is ton of data so I can not check each and every field manually. So what is best way to test migrated data with sql.
Try to implement 2 scripts which will result exactly the same output from mysql and elasticsearch. Make sure, formatting, ordering matches. At the end you should be able to diff the files or do md5sum on them to compare.
I am new to Solr and i need to know whether i am thinking correct about Solr and MySQL relation or not.
We index data from MySQL to Solr for once and then all add,edit,delete, update queries etc are implemented on Solr and MySQL got no change meanwhile. If we need to update MySQL as well, we will have to export (or some thing like that) from Solr to MySQL to keep MySQL up to date.
Am i thinking Right?
We only need to index those tables of MySQL on Solr which need search, not all MySQL data?
Am i thinking Right again?
Qn 1. Usual case is that your main data store is MySQL, so Solr is the one that lags. One can either use the data import handler or write custom indexing programs to get data from MySQL to Solr.
Qn 2. Along with the fields you want to search (which are indexed fields), you can also keep non-indexed stored fields in Solr. This will help you build your data from Solr itself, without doing a secondary DB query.