I am making some POC using debezium version 0.9 and I am running a standalon docker-ized mysql database using the embedded Debezium configuration. I am running into this issue:
Caused by: java.lang.IllegalStateException: Cannot read the binlog filename and position via 'SHOW MASTER STATUS'. Make sure your server is correctly configured
I suspect it's because the standalone configuration of mysql or the embedded configuration, but not sure how to proceed and fix. Any hint?
As Bill pointed out in the above comment - Debezium connector uses binary log to do its job so it needs to be enabled.
Debezium MySQL requires enabling the server binlog. In the case of RDS
MySQL, the log_bin property is managed directly by AWS and is set to
OFF by default. When Debezium MySQL executes the SHOW MASTER STATUS
command during a snapshot, the result set is empty and an exception is
thrown:
https://debezium.io/documentation/faq/
You can check if binlog is enabled with show master status and that is the error you are getting from debezium when it's disabled. The following is an example when binlog is enabled, when it's disabled there would be no rows returned.
mysql> show master status;
+------------------+----------+--------------+------------------+--------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+--------------------------------------------+
| mysql-bin.000001 | 170796 | | | bf9ae151-baac-11ec-919f-42010a8a0014:1-599 |
+------------------+----------+--------------+------------------+--------------------------------------------+
For Google Cloud SQL MySQL 5.x you need to enable point-in-time recovery
https://cloud.google.com/sql/docs/mysql/backup-recovery/pitr#enablingpitr
Related
I have successfully connected Remote hive metastore with apache Drill. I am able to show databases of remote hdfs and also able to see the table structure from database. But while querying on the database it is giving error
Error: SYSTEM ERROR: UnknownHostException: remotename
Here is my configuration of apache hive
{
"type": "hive",
"enabled": true,
"configProps": {
"hive.metastore.uris": "thrift://myremoteIP:PortofThrift",
"hive.metastore.warehouse.dir": "/tmp/drill_hive_wh",
"fs.default.name": "hdfs://IP address of remote:port of hdfs from /",
"hive.metastore.sasl.enabled": "false"
}
}
here are the successful queries
jdbc:drill:zk=local> describe data_mcsc_mcsc_bill_info;
and result.
COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+------------------------+--------------------+--------------+
| tran_dt | CHARACTER VARYING | YES |
| tran_tm | CHARACTER VARYING | YES |
| bill_id | CHARACTER VARYING | YES |
| policy_number | CHARACTER VARYING | YES |
| policy_start_date | CHARACTER VARYING | YES |
| policy_end_date | CHARACTER VARYING | YES |
More details will be required to provide a complete answer to your question. I can provide some debugging tips here.
Verify that the machines running your namenode and metastore are accessible from the machine you are running Drill on. You can use the telnet command to verify that a socket can be opened. If this fails you have a firewall / connectivity issue.
Validate that Drill can talk to your HDFS cluster by putting a csv file on HDFS and adding a storage plugin configuration for your HDFS cluster to Drill. Validate that you can query the file from Drill. If this fails you may have not specified the correct port for your HDFS namenode or there may be some other HDFS related issue.
If these debugging tips are insufficient please subscribe to the Apache Drill dev and user lists. You can look at the information here on how to do this http://drill.apache.org/mailinglists/. You can then send your question to the user list, and the Drill team can provide more interactive help with your issue there. Also please include the following information if you send your question to the user list:
Your version of Drill.
Whether you are running a drill cluster or you are running a simple standalone node.
The version of Hive.
The Distribution of HDFS you are using (e.g. Big Top, Hortonworks, Cloudera).
I just upgraded Piwik to the latest version via the integrated updater, on the first step of the updater wizard there was a file permission error. I continued to the next step which upgraded the database without doing anything about the permissions.
It upgraded the database to the latest version and I also ran the console command to upgrade it which finished with a success message.
For some reason (probably unrelated to the piwik upgrade) my mysqld crashed and I had to restart it. Now I'm running into a problem with my Piwik database.
If I list the tables with show tables; they all show up but I can't list them:
MariaDB [piwik]> show columns from piwik_user;
ERROR 1146 (42S02): Table 'piwik.piwik_user' doesn't exist
MariaDB [piwik]> check table piwik_user;
+------------------+-------+----------+----------------------------------------+
| Table | Op | Msg_type | Msg_text |
+------------------+-------+----------+----------------------------------------+
| piwik.piwik_user | check | Error | Table 'piwik.piwik_user' doesn't exist |
| piwik.piwik_user | check | status | Operation failed |
+------------------+-------+----------+----------------------------------------+
This is also the output phpmyadmin shows if I click on any of the tables.
Is there a chance to repair that database or is it lost for good? This is running on my backup server so unfortunately there are no backups, this will change in the future.
Thanks
Context:
I'm trying to set up mysql replication with one master and one slave. I'm using the new GTID method in mysql 5.6, along with the mysql utilities for automatic failover.
I'm able to set up replication with a master node M and a slave node S.
I'm trying to run through a simple script to show myself that everything is set up correctly. The script is as follows:
Set up replication with node M as master and node S as slave
Ensure that changes made to tables in M are reflected in S
Shutdown M and ensure that S takes over as new master
Make a few table changes to S
Startup M and add it back to cluster as a slave
Ensure that M catches up to S with its updates
Make M the master node again
My script seems successful until the last step.
Largely, I'm following the steps here: http://www.clusterdb.com/mysql/replication-and-auto-failover-made-easy-with-mysql-utilities
But when I run "mysqlrpladmin --master=root#S_NODE --new-master=root#M_NODE --demote-master --discover-slaves-login=root switchover", I'm getting the following error:
# Discovering slaves for master at S_NODE
# Discovering slave at M_NODE
# Found slave: M_NODE
# Checking privileges.
# Performing switchover from master at S_NODE to slave at M_NODE
# Checking candidate slave prerequisites.
Candidate slave is missing replication user.
ERROR: Candidate slave is missing replication user.
# Errors found. Switchover aborted.
#
# Replication Topology Health:
+-----------------+-------+---------+--------+------------+---------+
| host | port | role | state | gtid_mode | health |
+-----------------+-------+---------+--------+------------+---------+
| S_NODE | 3306 | MASTER | UP | ON | OK |
| M_NODE | 3306 | SLAVE | UP | ON | OK |
+-----------------+-------+---------+--------+------------+---------+
I can't seem to find any google or stackoverflow pages related to that error message "Candidate slave is missing replication user."
I believe I have a replication user on both node S and M, since both acted as a slave at one point in my test script.
Any thoughts?
I ran into this same problem while testing failover on some new clusters and noticed that after adding the --force flag as you suggest, the switchover completed successfully and created the user 'replication_user'#'S_NODE' without any prompting on my part.
It then occurred to me that the replication user I had set up was 'repl'#'specific_subnet_%' - which did not match what the script was looking for ('repl'#'S_NODE'), so the pre-check failed without --force.
I had another cluster I was testing, so to validate I did the following:
ran the mysqlrpladmin ... switchover command without --force (it failed as expected)
created both 'replication_user'#'S_NODE' and 'replication_user'#'M_NODE' with the same grant (REPLICATION SLAVE) and password as my existing replication_user. You need both S_NODE and M_NODE hosts if you want to be able to fail back and forth between them using this script.
tried mysqlrpladmin ... switchover again without --force
It worked like a charm this time!
Moral of the story: the script expects the 'replication_user'#'new_master' user to exist. The --force flag will create that user for you, and should not be necessary after the first time you use it.
For some reason, adding "--force" flag fixed the issue. More details in the comments above.
If anyone knows why this might be though, please comment.
We've got a client application that connects to our online MySQL database (5.1.44-community-log) thru a ODBC connector (the server is a managed* dedicated webserver). This works very nice. However I can't get it to work using SSL. This is what I've done so far:
1. MySQL server
I've got the server manager* set up MySQL with SSL, this is 'proven by':
mysql> SHOW VARIABLES LIKE '%ssl%';
which results is this response:
+---------------+---------------------------------+
| Variable_name | Value |
+---------------+---------------------------------+
| have_openssl | YES |
| have_ssl | YES |
| ssl_ca | /***/mysql-cert/ca-cert.pem |
| ssl_capath | |
| ssl_cert | /***/mysql-cert/server-cert.pem |
| ssl_cipher | |
| ssl_key | /***/mysql-cert/server-key.pem |
+---------------+---------------------------------+
Question: is the server configured right? I'm guessing it is...
2. Certificates
I've purchased real certificates (via my server manager). These are in the directory shown above. I've also downloaded the client-cert.pem, client-key.pem and ca-cert.pem from that directory.
3. MySQL user with REQUIRE [SSL|X509]
I've created a new user and then granted it access from any location (for testing) with SSL:
GRANT USAGE ON *.* TO 'somevaliduser'#'%' IDENTIFIED BY PASSWORD 'somevalidpass' REQUIRE X509
4. ODBC Client
I've (just downloaded and) installed : mysql-connector-odbc-5.1.8-winx64.msi (64-bit) as my machine is a 64-bit Windows 7 machine (so that's not what's wrong).
And I've created a User DSN configuring it like this (no options set on tabs), which shows it connecting to the server (however not using - nor requesting to do so - SSL) successfully (using some valid user which doesn't requires SSL):
So the connection is able to establish, now try using SSL.
This is configured like this, which is like I've read about on MySQL.com. So I'm not 100% sure the options set are right.
As you can see it results in a error HY000. Turning on tracing (within the ODBC configuration) also shows this error.
Can anyone give me a hint on how to make this work? Even if you know about just a part of the solution?
I solved the problem. Because I tried several things at a time I don't know what did the trick:
I've had the server manager re-create the certificates: I bought some but I found out that those couldn't be used to SSL-encrypt the connection. So for now I'm using OpenSSL certificates. I've had them re-create the certificates with 4) Create your client .... server. They must be unique. (as mentioned here) in mind.
I guess the checkbox 'Verify SSL Certificate' only applies when you buy a certificate and a thrid party service should check the validity of the certificate. Uncheck that box!
Only fill out the fields:
'SSL Key' (c:\path_to\client-key.pem)
'SSL Certificate' (c:\path_to\client-cert.pem)
'SSL Certificate Authority' (c:\path_to\ca-cert.pem)
Please note:
The port is still the same (for me).
The logs - as Michal Niklas proposed - didn't show any usefull information.
I've toggled on 'Use compression' which is said to improve performance.
I am using Ubuntu 12.04 LTS with MySQL
Ver 5.5.22-0ubuntu1 for debian-linux-gnu on x86_64 ((Ubuntu)) and OpenSSL OpenSSL 1.0.1 14 Mar 2012
I created the certificates following the tutorial on
http://www.thomas-krenn.com/de/wiki/MySQL_Verbindungen_mit_SSL_verschl%C3%BCsseln
(The tutorial is in German, but this is not important here).
When trying to connect with
mysql -u root -p --ssl-ca=/etc/mysql/ca-cert.pem --ssl-cert=/etc/mysql/client-cert.pem --ssl-key=/etc/mysql/client-key.pem --protocol=tcp
I always got an error message SSL connection error: protocol version mismatch
This lead me to the site
http://bugs.mysql.com/bug.php?id=64870
which confirms (for me) that there is a bug.
To make a long story short. In the end I created all certificates using my MacOS X Lion, copied the certificates to the server and client and it worked immediately!
When I got the Linux side working, Windows worked immediatly, too!
As mentioned above, you just have to set client-key, client-cert and ca-cert!
How to find mysql DB is slave with out using "show slave status" by query?
Here are 3 options you have to detect if Replication is running
OPTION #1 : Check Status Variable 'Slave_running'
Using MySQL 5.1/5.5
select variable_value from information_schema.global_status
where variable_name = 'Slave_running';
Using MySQL 5.0 and back
SHOW VARIABLES LIKE 'Slave_running';
OPTION #2 : Check the Process List
Using MySQL 5.1+/5.5
select COUNT(1) SlaveThreads
from information_schema.processlist
where user = 'system user';
If SlaveThreads = 2, Replication is Running
If SlaveThreads = 1, Replication is Broken
If SlaveThreads = 0, Replication is Stopped or Disabled
Using MySQL 5.0 and back
SHOW PROCESSLIST;
Look for 2 DB Conenctions thaty have 'system user' in the user column.
OPTION #3 : Check for presence of master.info
If replication is setup on a DB Server, look for master.info. By default, master.info is usually in /var/lib/mysql or wherever datadir is defined.
Simply run 'cat master.info' multiple times (For Windows community, type master.info). If the log position is moving, replication is on. If the log position is not moving, it could mean that replication is either broken (SQL Error in SQL Thread), stopped (due to STOP SLAVE;), or disabled (by running CHANGE MASTER TO MASTER_HOST='';).
According to MySQL doc - Checking Replication Status:
Slave_IO_Running: Whether the I/O
thread for reading the master's binary
log is running. Normally, you want
this to be Yes unless you have not yet
started replication or have explicitly
stopped it with STOP SLAVE.
Slave_SQL_Running: Whether the SQL
thread for executing events in the
relay log is running. As with the I/O
thread, this should normally be Yes.
Prior to MySQL 5.7, you can check the 'slave_running' variable by executing the following query:
SHOW GLOBAL STATUS LIKE 'slave_running';
Since MySQL 5.7, the slave_running has been removed and the above query returns an empty set
You can enable "show_compatibility_56" to get the value but “show_compatibility_56” is deprecated and will be removed soon. The reason for this is because MySQL is moving away from the information_schema GLOBAL_STATUS and SESSION_STATUS tables in preference for performance_schema.
The correct way to get the status of the slave running in MySQL 5.7 outside of SHOW SLAVE STATUS is to use the new replication-based performance_schema tables.
You can execute the following query to get the status of the replication service:
SELECT SERVICE_STATE FROM performance_schema.replication_connection_status;