I have successfully connected Remote hive metastore with apache Drill. I am able to show databases of remote hdfs and also able to see the table structure from database. But while querying on the database it is giving error
Error: SYSTEM ERROR: UnknownHostException: remotename
Here is my configuration of apache hive
{
"type": "hive",
"enabled": true,
"configProps": {
"hive.metastore.uris": "thrift://myremoteIP:PortofThrift",
"hive.metastore.warehouse.dir": "/tmp/drill_hive_wh",
"fs.default.name": "hdfs://IP address of remote:port of hdfs from /",
"hive.metastore.sasl.enabled": "false"
}
}
here are the successful queries
jdbc:drill:zk=local> describe data_mcsc_mcsc_bill_info;
and result.
COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+------------------------+--------------------+--------------+
| tran_dt | CHARACTER VARYING | YES |
| tran_tm | CHARACTER VARYING | YES |
| bill_id | CHARACTER VARYING | YES |
| policy_number | CHARACTER VARYING | YES |
| policy_start_date | CHARACTER VARYING | YES |
| policy_end_date | CHARACTER VARYING | YES |
More details will be required to provide a complete answer to your question. I can provide some debugging tips here.
Verify that the machines running your namenode and metastore are accessible from the machine you are running Drill on. You can use the telnet command to verify that a socket can be opened. If this fails you have a firewall / connectivity issue.
Validate that Drill can talk to your HDFS cluster by putting a csv file on HDFS and adding a storage plugin configuration for your HDFS cluster to Drill. Validate that you can query the file from Drill. If this fails you may have not specified the correct port for your HDFS namenode or there may be some other HDFS related issue.
If these debugging tips are insufficient please subscribe to the Apache Drill dev and user lists. You can look at the information here on how to do this http://drill.apache.org/mailinglists/. You can then send your question to the user list, and the Drill team can provide more interactive help with your issue there. Also please include the following information if you send your question to the user list:
Your version of Drill.
Whether you are running a drill cluster or you are running a simple standalone node.
The version of Hive.
The Distribution of HDFS you are using (e.g. Big Top, Hortonworks, Cloudera).
Related
I am making some POC using debezium version 0.9 and I am running a standalon docker-ized mysql database using the embedded Debezium configuration. I am running into this issue:
Caused by: java.lang.IllegalStateException: Cannot read the binlog filename and position via 'SHOW MASTER STATUS'. Make sure your server is correctly configured
I suspect it's because the standalone configuration of mysql or the embedded configuration, but not sure how to proceed and fix. Any hint?
As Bill pointed out in the above comment - Debezium connector uses binary log to do its job so it needs to be enabled.
Debezium MySQL requires enabling the server binlog. In the case of RDS
MySQL, the log_bin property is managed directly by AWS and is set to
OFF by default. When Debezium MySQL executes the SHOW MASTER STATUS
command during a snapshot, the result set is empty and an exception is
thrown:
https://debezium.io/documentation/faq/
You can check if binlog is enabled with show master status and that is the error you are getting from debezium when it's disabled. The following is an example when binlog is enabled, when it's disabled there would be no rows returned.
mysql> show master status;
+------------------+----------+--------------+------------------+--------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+--------------------------------------------+
| mysql-bin.000001 | 170796 | | | bf9ae151-baac-11ec-919f-42010a8a0014:1-599 |
+------------------+----------+--------------+------------------+--------------------------------------------+
For Google Cloud SQL MySQL 5.x you need to enable point-in-time recovery
https://cloud.google.com/sql/docs/mysql/backup-recovery/pitr#enablingpitr
I just upgraded Piwik to the latest version via the integrated updater, on the first step of the updater wizard there was a file permission error. I continued to the next step which upgraded the database without doing anything about the permissions.
It upgraded the database to the latest version and I also ran the console command to upgrade it which finished with a success message.
For some reason (probably unrelated to the piwik upgrade) my mysqld crashed and I had to restart it. Now I'm running into a problem with my Piwik database.
If I list the tables with show tables; they all show up but I can't list them:
MariaDB [piwik]> show columns from piwik_user;
ERROR 1146 (42S02): Table 'piwik.piwik_user' doesn't exist
MariaDB [piwik]> check table piwik_user;
+------------------+-------+----------+----------------------------------------+
| Table | Op | Msg_type | Msg_text |
+------------------+-------+----------+----------------------------------------+
| piwik.piwik_user | check | Error | Table 'piwik.piwik_user' doesn't exist |
| piwik.piwik_user | check | status | Operation failed |
+------------------+-------+----------+----------------------------------------+
This is also the output phpmyadmin shows if I click on any of the tables.
Is there a chance to repair that database or is it lost for good? This is running on my backup server so unfortunately there are no backups, this will change in the future.
Thanks
I have a drupal website that is calling the static PDO __contruct for a mysql connection, on a remote server at first and then changed to a local server to remove network latency for sanity checking.
On the local db call I am getting the following generated from xhprof and an extremely slow page load.
Function Name | Calls | Calls% | Incl. Wall Time(ms) | IWall% | Excl. Wall Time(ms) | EWall%
PDO::__construct | 6 | 0.0% | 120,084,724 | 91.6% | 120,084,724 | 91.6%
The php version is 5.4 on debian wheezy. The website is on a nginx and php5-fpm stack. The MySQL version is 5.5.
The tables are MyISAM but were originally InnoDB and had the same issue.
Does anyone know what could be causing this delay in the connection?
We've got a client application that connects to our online MySQL database (5.1.44-community-log) thru a ODBC connector (the server is a managed* dedicated webserver). This works very nice. However I can't get it to work using SSL. This is what I've done so far:
1. MySQL server
I've got the server manager* set up MySQL with SSL, this is 'proven by':
mysql> SHOW VARIABLES LIKE '%ssl%';
which results is this response:
+---------------+---------------------------------+
| Variable_name | Value |
+---------------+---------------------------------+
| have_openssl | YES |
| have_ssl | YES |
| ssl_ca | /***/mysql-cert/ca-cert.pem |
| ssl_capath | |
| ssl_cert | /***/mysql-cert/server-cert.pem |
| ssl_cipher | |
| ssl_key | /***/mysql-cert/server-key.pem |
+---------------+---------------------------------+
Question: is the server configured right? I'm guessing it is...
2. Certificates
I've purchased real certificates (via my server manager). These are in the directory shown above. I've also downloaded the client-cert.pem, client-key.pem and ca-cert.pem from that directory.
3. MySQL user with REQUIRE [SSL|X509]
I've created a new user and then granted it access from any location (for testing) with SSL:
GRANT USAGE ON *.* TO 'somevaliduser'#'%' IDENTIFIED BY PASSWORD 'somevalidpass' REQUIRE X509
4. ODBC Client
I've (just downloaded and) installed : mysql-connector-odbc-5.1.8-winx64.msi (64-bit) as my machine is a 64-bit Windows 7 machine (so that's not what's wrong).
And I've created a User DSN configuring it like this (no options set on tabs), which shows it connecting to the server (however not using - nor requesting to do so - SSL) successfully (using some valid user which doesn't requires SSL):
So the connection is able to establish, now try using SSL.
This is configured like this, which is like I've read about on MySQL.com. So I'm not 100% sure the options set are right.
As you can see it results in a error HY000. Turning on tracing (within the ODBC configuration) also shows this error.
Can anyone give me a hint on how to make this work? Even if you know about just a part of the solution?
I solved the problem. Because I tried several things at a time I don't know what did the trick:
I've had the server manager re-create the certificates: I bought some but I found out that those couldn't be used to SSL-encrypt the connection. So for now I'm using OpenSSL certificates. I've had them re-create the certificates with 4) Create your client .... server. They must be unique. (as mentioned here) in mind.
I guess the checkbox 'Verify SSL Certificate' only applies when you buy a certificate and a thrid party service should check the validity of the certificate. Uncheck that box!
Only fill out the fields:
'SSL Key' (c:\path_to\client-key.pem)
'SSL Certificate' (c:\path_to\client-cert.pem)
'SSL Certificate Authority' (c:\path_to\ca-cert.pem)
Please note:
The port is still the same (for me).
The logs - as Michal Niklas proposed - didn't show any usefull information.
I've toggled on 'Use compression' which is said to improve performance.
I am using Ubuntu 12.04 LTS with MySQL
Ver 5.5.22-0ubuntu1 for debian-linux-gnu on x86_64 ((Ubuntu)) and OpenSSL OpenSSL 1.0.1 14 Mar 2012
I created the certificates following the tutorial on
http://www.thomas-krenn.com/de/wiki/MySQL_Verbindungen_mit_SSL_verschl%C3%BCsseln
(The tutorial is in German, but this is not important here).
When trying to connect with
mysql -u root -p --ssl-ca=/etc/mysql/ca-cert.pem --ssl-cert=/etc/mysql/client-cert.pem --ssl-key=/etc/mysql/client-key.pem --protocol=tcp
I always got an error message SSL connection error: protocol version mismatch
This lead me to the site
http://bugs.mysql.com/bug.php?id=64870
which confirms (for me) that there is a bug.
To make a long story short. In the end I created all certificates using my MacOS X Lion, copied the certificates to the server and client and it worked immediately!
When I got the Linux side working, Windows worked immediatly, too!
As mentioned above, you just have to set client-key, client-cert and ca-cert!
I'm injecting a stress test into my web app that connects to a mysql server and I'm monitoring the show processlist of mysql.
When the load is high (high swap i/o) I get many processes like that:
| 97535 | db| localhost | userA | Sleep | 515 | | NULL
| 97536 | db| localhost | userA | Sleep | 516 | | NULL
| 97786 | db| localhost | userA | Sleep | 343 | | NULL
| 97889 | db| localhost | userA | Sleep | 310 | | NULL
But I can't understand why are they still there and are not killed? This eventually leads to my app using all max_connections and stop processing incoming requests...
Any idea what are those processes and what are they doing there :) ?
Those are idle connections being held by a client. You should make sure that whatever client library you are using (JDBC, ...) is configured to not keep unused connections open so long, or that your # clients * max # of connections isn't too big.
My guess is that you are using persistent connections, e.g. pconnect in php:
[..] when connecting, the function would first try to find a (persistent) link that's already open with the same host, username and password. If one is found, an identifier for it will be returned instead of opening a new connection
and
[..] the connection to the SQL server will not be closed when the execution of the script ends. Instead, the link will remain open for future use
I had a similar situation, and was using Codeigniter with pconnect turned on. After turning it to off (see how) every connection was closed down properly after use, and my MySQL processlist was empty.
Performance: The above does not argue about performance, but simply tries to explain why you might see a lot of Sleeping connections in MySQL. It might not be negative, with regard to performance, to have the connections stay active.
More info at: http://www.mysqlperformanceblog.com/2006/11/12/are-php-persistent-connections-evil/