I'm new to elasticsearch.I have learnt how to give different queries and get search results with the understanding that each document is stored in json format.Is it possible to insert records that were obtained from an sql query on a relational database?If it is possible,how is it done? by converting each record into json format?
You need to build an index in elasticsearch similar to the way you've got your tables in the RMDBS, this can be done in a lot of ways and it really depends on what data you would need to access via elasticsearch. You shouldnt just dump your complete RMDBS data into ES.
If you search around you may find bulk data importers/synchronisers/rivers(deprecated) for your RMDBS to ES, some of these can run in the background and keep the indexes in ES upto date with your RMDBS.
You can create your own code as well which updates ES whenever any data is changed in your RMDBS. Look into the API for your platform Elastic Search Client APIhttps://www.elastic.co/guide/en/elasticsearch/client/index.html
Related
I am trying to do it with client lib using python.
the problem I am facing is that the TIMESTAMP on the JSON files are on Unix epoch TIMESTAMP format and big query can't detect that:
according to documentation:
so I wonder what to do?
I thought about changing the JSON format manually before I load it into BigQuery table?
Or maybe looking for an auto conversion from the BigQuery side?
I wondered across the internet and could not find anything useful yet.
Thanks in advance for any support.
You have 2 solutions
Either you update the format before the BigQuery integration
Or you update the format after the BigQuery integration
Before
Before means updating your JSON (manually or by script) or to update it by the process that load the JSON into BigQuery (like Dataflow).
I personally don't like this, file handling are never funny and efficient.
After
In this case, you let BigQuery loading your JSON file into a temporary table and convert your UNIX timestamp into a Number or a String. Then, perform a request into this temporary table, convert the field in the correct timestamp format, and insert the data in the final table.
This way is smoother and easier (a simple SQL query to write). However, it implies cost to read all the loaded data (to write them then)
I am very new to ELK and want to know if there is a way around to join two databases from different sources (ie. MYSQL and Postgres) and indexing it to a single index in elasticsearch using logstash.
As I am able to achieve the same with the help of pyspark. But I want to achieve the same thing using log stash if it's possible!
Also, suggest some other feasible ways to achieve the same apart from the spark and logstash.
Thanks in Advance!
You can definitely achieve this by sourcing data from one database using a jdbc input and then joining it with data coming from another database that is sourced with either the jdbc_static (if the data doesn't change too often) or jdbc_streaming (if the data changes more often) filters.
My Application's data is stored in both MongoDB and MYSQL. Is there any way to write queries in Metabase that will help in fetching data from both MYSQL and Mongo Databases?
I'm assuming here that by "My application's data" you are refering to the data sources, not to the Metabase internal databasa (aka metadata).
As of now (0.32.10), there is no way to query data from two different data sources.
What you can do, though, is to setup two (or more, depending on your need) different questions, and add them to a dashboard - that way, you will be able to show the data in a "data source agnostic" way.
If I didn't not get it wrong, I think this will be somewhat possible in the incoming version (0.33), scheduled for the next weeks (I believe), using the new join system (which will allow you to join data from different data sources, given a certain common key - say, "order_id" or something like that).
You can get more info, and even test the new version (which is currently in the RC2) at
this link.
Anyone who have worked on elastic search. need input as mentioned in subject. I have one requirement in which sql data is migrated to elastic Search now there is ton of data so I can not check each and every field manually. So what is best way to test migrated data with sql.
Try to implement 2 scripts which will result exactly the same output from mysql and elasticsearch. Make sure, formatting, ordering matches. At the end you should be able to diff the files or do md5sum on them to compare.
We are looking at ways of improving "search" functionality in our large business application which currently uses SQL Like syntax to do it. So we started evaluating Solr server and were able to index few of our database tables and search. But I am newbie and wanted to know if
1) We have large number of tables in our application. Is there any utility that generates schema xml in solr using the database tables?
2) Our current search lists the database row that meets the search criteria (this was written using SQL 'like' and takes lot of time to generate the search results). We want to simulate the exact functionality using solr. Is that possible?
For importing a database into SOLR, you might want to look into DataImportHandler.
There will be a fair amount of configuration required for it, defining what tables and columns to import, what should be stored, and how it should be indexed.