I have an Orion context broker and i use cygnus(0.6) to send data to my cosmos account.
Cygnus creates correctly the txt file with the data on my cosmos, but it does not create
the tables for hive. I would query my data through an hive client.
Is there a configuration paramater to set in cygnus.conf to do that?
How can i solve?
I found myself with the same issue and found the problem. Check this: Cygnus release/0.6.0 does not create Hive extended tables
Related
Hi I am working on a project to display more visualization in my dashboard from the Airflow data. I am struggling on how can I query Airflow database directly to retrieve such data. Is there a way how can I retrieve data from Airflow database? Thanks
Since I did not set up any MySQL connection airflow was using SQLlite3 and that is why I was struggling to connect to the database. This was fixed by create a Airflow database in MySQL.
I have installed CDH 5.16 in a RHEL 7 server and installed kafka separately.
I am trying to load data from mysql to HDFS or Hive table on real time basis(CDC approach). That is if some data is updated or added in mysql table ,it should be immediately reflected in HDFS or Hive table.
Approach i have come up with:
Use kafka-connect to connect to mysql server and push table data to a kafka topic
and write a consumer code in spark-stream which reads the data from topic
and store it in HDFS.
One problem with this approach is, hive table on top of these files should
be refreshed periodically for the update to be reflected.
I also came to know of Kafka-Hive integration in HDP 3.1. Unfortunately i am using Hadoop 2.6.0. So cant leverage this feature.
Is there any other better way achieve this?
I am using Hadoop 2.6.0 and CDH 5.16.1
I am trying to connect JHipster's MySQL database with a HIVE database so that JHipster can be receive data from HIVE tables. I want to ask you what is the best approach for this problem as I have just began using JHipster.
Thank you !
what do you mean when you say JHipster's MySQL database with a HIVE database? I asume that you want to use the MySQL database for authentification and authorization and then for diferent bussines logic you will want to retrive data from HIVE. If you are using your JHipster project with a relational database, then jhipster is using spring data jpa in order to access the database. Since HIVE is a NoSQL database, my recomandation it will be to have a look at spring-hadoop for an easy access on HIVE database and create a service which is macking use of sopring-data-jpa.
How to configure Cygnus to save data on SQL database?
After installing Orion and Cosmos, I try to configure cygnus and save data on mysql database. Is it possible?
With Cygnus you can save all the entities you want to by making a subscription to OrionCB. Yo can save it in differents formats including MySQL.
Anyway, you should visit this Cygnus guide and read it before asking about. The guide is pretty clear about how cygnus work.
I'm running a cygnus instance in a Fi-Lab vm. I've compiled it from branch release/0.6.0. Everything is working except that Hive extended tables for context updates are not created. Cygnus is succesfully receiving updates from Orion, they are forwarded to Cosmos and the HDFS files are created.
The cygnus log says Creating Hive external table=... and does not raise any error. Then, if I log into the cosmos head node, I can see the HDFS files created, but if I use the Hive console, the tables are not there.
If I type SHOW TABLES; in the Hive console, I can see some tables that end with _row and _column, so I guess it is working for others.
Any hint? Should I use another version of cygnus?
Finally I found the problem. Flume is using libthrift 0.7 but cygnus needs 0.9. The solution is in the README (at the end of section Installing Cygnus and its dependencies (from sources)) but I skipped it. You have to manually overwrite the jar file in the Flume bin distribution with the 0.9 version.