How to use nifi CaptureChangeMySQL? - mysql

Introduction
I have a mysql database that receives regular updates. I want to regularly check this mysql db for changes in Nifi.
For this, the CaptureChangeMySQL processor seems perfect.
However, I am not able to make it work.
I followed this tutorial, but the processor does not catch anything when inserting/deleting rows from database.
Setup
One MySQL on local, accessible at localhost:3306
One nifi on the same machine.
MySQL Config
It has one database named test with a device table in it, containing a bit more than 20k rows.
My my.cnf file, in /etc/mysql/ is the following:
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
[mysqld]
server_id = 1
log_bin = delta
binlog_format=row
binlog_do_db = source
Nifi CaptureChangeMySQL config
Nifi CDC MapCache config (Distributed Map Cache CLient Service)
Nifi Distributed Map Cache Server Config
With this configuration, my CaptureChange processor just does nothing (at least nothing visible). What am I doing wrong that prevents me to use it ?

In your my.cnf file, you've set binlog_do_db = source.
binlog-do-db makes the master write only statements for the specified DB into its binary log. In your case, it should be set to test.
Refer MySQL 5.7 binlog-do-db=db_name

Related

I am using LOAD DATA LOCAL INFILE <URL> query In sql but ERROR: #2000 - LOAD DATA LOCAL INFILE is forbidden, check mysqli.allow_local_infile [duplicate]

Is there a MySQL command to locate the my.cnf configuration file, similar to how PHP's phpinfo() locates its php.ini?
There is no internal MySQL command to trace this, it's a little too abstract. The file might be in 5 (or more?) locations, and they would all be valid because they load cascading.
/etc/my.cnf
/etc/mysql/my.cnf
$MYSQL_HOME/my.cnf
[datadir]/my.cnf
~/.my.cnf
Those are the default locations MySQL looks at. If it finds more than one, it will load each of them & values override each other (in the listed order, I think). Also, the --defaults-file parameter can override the whole thing, so... basically, it's a huge pain in the butt.
But thanks to it being so confusing, there's a good chance it's just in /etc/my.cnf.
(If you just want to see the values: SHOW VARIABLES, but you'll need the permissions to do so.)
Run mysql --help and you will see:
Default options are read from the following files in the given order: /etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf
You can actually "request" MySQL for a list of all locations where it searches for my.cnf (or my.ini on Windows). It is not an SQL query though. Rather, execute:
$ mysqladmin --help
or, prior 5.7:
$ mysqld --help --verbose
In the very first lines you will find a message with a list of all my.cnf locations it looks for. On my machine it is:
Default options are read from the following files in the given order:
/etc/my.cnf
/etc/mysql/my.cnf
/usr/etc/my.cnf
~/.my.cnf
Or, on Windows:
Default options are read from the following files in the given order:
C:\Windows\my.ini
C:\Windows\my.cnf
C:\my.ini
C:\my.cnf
C:\Program Files\MySQL\MySQL Server 5.5\my.ini
C:\Program Files\MySQL\MySQL Server 5.5\my.cnf
Note however, that it might be the case that there is no my.cnf file at any of these locations. So, you can create the file on your own - use one of the sample config files provided with MySQL distribution (on Linux - see /usr/share/mysql/*.cnf files and use whichever is appropriate for you - copy it to /etc/my.cnf and then modify as needed).
Also, note that there is also a command line option --defaults-file which may define custom path to my.cnf or my.ini file. For example, this is the case for MySQL 5.5 on Windows - it points to a my.ini file in the data directory, which is not normally listed with mysqld --help --verbose. On Windows - see service properties to find out if this is the case for you.
Finally, check the https://dev.mysql.com/doc/refman/8.0/en/option-files.html - it is described there in more details.
You could always run find in a terminal.
find / -name my.cnf
mysql --help | grep /my.cnf | xargs ls
will tell you where my.cnf is located on Mac/Linux
ls: cannot access '/etc/my.cnf': No such file or directory
ls: cannot access '~/.my.cnf': No such file or directory
/etc/mysql/my.cnf
In this case, it is in /etc/mysql/my.cnf
ls: /etc/my.cnf: No such file or directory
ls: /etc/mysql/my.cnf: No such file or directory
ls: ~/.my.cnf: No such file or directory
/usr/local/etc/my.cnf
In this case, it is in /usr/local/etc/my.cnf
You can use :
locate my.cnf
whereis my.cnf
find . -name my.cnf
This might work:
strace mysql ";" 2>&1 | grep cnf
on my machine this outputs:
stat64("/etc/my.cnf", 0xbf9faafc) = -1 ENOENT (No such file or directory)
stat64("/etc/mysql/my.cnf", {st_mode=S_IFREG|0644, st_size=4271, ...}) = 0
open("/etc/mysql/my.cnf", O_RDONLY|O_LARGEFILE) = 3
read(3, "# /etc/mysql/my.cnf: The global "..., 4096) = 4096
stat64("/home/xxxxx/.my.cnf", 0xbf9faafc) = -1 ENOENT (No such file or directory)
So it looks like /etc/mysql/my.cnf is the one since it stat64() and read() were successful.
By default, mysql search my.cnf first at /etc folder. If there is no /etc/my.cnf file inside this folder, I advise you to create new one in this folder as indicated by the documentation (https://dev.mysql.com/doc/refman/5.6/en/option-files.html).
You can also search for existing my.cnf furnished by your mysql installation. You can launch the following command
sudo find / -name "*.cnf"
You can use the following configuration file with myisam table and without innodb mysql support (from port installation of mysql on mac os x maverick). Please verify each command in this configuration file.
# Example MySQL config file for large systems.
#
# This is for a large system with memory = 512M where the system runs mainly
# MySQL.
#
# MySQL programs look for option files in a set of
# locations which depend on the deployment platform.
# You can copy this option file to one of those
# locations. For information about these locations, see:
# http://dev.mysql.com/doc/mysql/en/option-files.html
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.
# The following options will be passed to all MySQL clients
[client]
#password = your_password
port = 3306
socket = /opt/local/var/run/mysql5/mysqld.sock
# Here follows entries for some specific programs
# The MySQL server
[mysqld]
port = 3306
socket = /opt/local/var/run/mysql5/mysqld.sock
skip-locking
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
#skip-networking
# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin
# binary logging format - mixed recommended
binlog_format=mixed
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
# MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
# where you replace <host>, <user>, <password> by quoted strings and
# <port> by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host = <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user = <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password = <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port = <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin
# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /opt/local/var/db/mysql5
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /opt/local/var/db/mysql5
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 256M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 64M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
For Ubuntu 16: /etc/mysql/mysql.conf.d/mysqld.cnf
As noted by konyak you can get the list of places mysql will look for your my.cnf file by running mysqladmin --help. Since this is pretty verbose you can get to the part you care about quickly with:
$ mysqladmin --help | grep -A1 'Default options'
This will give you output similar to:
Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf /usr/local/etc/my.cnf ~/.my.cnf
Depending on how you installed mysql it is possible that none of these files are present yet. You can cat them in order to see how your config is being built and create your own my.cnf if needed at your preferred location.
I don't know how you've setup MySQL on your Linux environment but have you checked?
/etc/my.cnf
Try running mysqld --help --verbose | grep my.cnf | tr " " "\n"
Output will be something like
/etc/my.cnf
/etc/mysql/my.cnf
/usr/local/etc/my.cnf
~/.my.cnf
You can also run this command.
mysql --help | grep cnf
If you're on a Mac with Homebrew, use
brew info mysql
You'll see something like
$ brew info mysql
mysql: stable 5.6.13 (bottled)
http://dev.mysql.com/doc/refman/5.6/en/
Conflicts with: mariadb, mysql-cluster, percona-server
/usr/local/Cellar/mysql/5.6.13 (9381 files, 354M) *
That last line is the INSTALLERDIR per the MySQL docs
Be aware that although mariadDB loads configuration details from the various my.cnf files as listed in the other answers here, it can also load them from other files with different names.
That means that if you make a change in one of the my.cnf files, it may get overwritten by another file of a different name. To make the change stick, you need to change it in the right (last loaded) config file - or, maybe, change it in all of them.
So how do you find all the config files that might be loaded? Instead of looking for my.cnf files, try running:
grep -r datadir /etc/mysql/
This will find all the places in which datadir is mentioned. In my case, it produces this answer:
/etc/mysql/mariadb.conf.d/50-server.cnf:datadir = /var/lib/mysql
When I edit that file (/etc/mysql/mariadb.conf.d/50-server.cnf) to change the value for datadir, it works, whereas changing it in my.cnf does not. So whatever option you are wanting to change, try looking for it this way.
If you are on Debian/Ubuntu system and already equipped with modern versions of the database (everything from 5.7 up, also true for mysql 8) the best way to locate the actual .cnf file I have found is:
sudo update-alternatives --config my.cnf
You should see a output like this:
There are 3 choices for the alternative my.cnf (providing /etc/mysql/my.cnf).
Selection Path Priority Status
------------------------------------------------------------
0 /etc/mysql/mariadb.cnf 500 auto mode
1 /etc/mysql/mariadb.cnf 500 manual mode
2 /etc/mysql/my.cnf.fallback 100 manual mode
* 3 /etc/mysql/mysql.cnf 300 manual mode
There are two lines in /etc/mysql/mysql.cnf that it makes sense to pay attention to:
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
MySQL will go watching for all .cnf files in /etc/mysql/conf.d/, then all files in /etc/mysql/mysql.conf.d/
Happy tuning!
Found mine using
mysqld --help --verbose | grep my.cnf
Answered for only MySQL Workbench users,
Another option is to use the whereis command.
E.g. whereis my.cnf
I installed xampp bundle with apache, php and mysql in ubuntu. There my.cnf file is located in /opt/lampp/etc/ folder. Hope it'll help somebody.
All great suggestions, in my case I didn't find it in any of those locations, but in /usr/share/mysql, I have a RHEL VM and I installed mysql5.5
You will have to look through the various locations depending on your version of MySQL.
mysqld --help -verbose | grep my.cnf
For Homebrew:
/usr/local/Cellar/mysql/8.0.11/bin/mysqld (mysqld 8.0.11)
Default possible locations:
/etc/my.cnf
/etc/mysql/my.cnf
~/.my.cnf
Found mine here:
/usr/local/etc/my.cnf
On Ubuntu (direct edit) :
$ sudo nano /etc/mysql.conf.d/mysqld.cnf
In case you are in a VPS and are trying to edit a my.cnf on an already running server you could try:
ps aux | grep mysql
You will be show the parameters the mysql command is being run and where the --defaults-file points to
Note that your server might be running more than one MySQL/MariaDB server's. If you see a line without --defaults-file parameter, that instance might be retrieving the configuration from the .cnf's that are mentioned on mysqladmin --help as others have pointed out.
You can find my.cnf or any other file with find command:
find / -name my.cnf (or any other file name)
find is a command
/ (slash) is a path
my.cnf is a file name
If you are using MAMP, access Templates > MySQL (my.cnf) > [version]
If you are running MAMP windowless you may need to customize the toolbar using the Customize button.
for me it was that i had "ENGINE=MyISAM" kind of tables , once i changed it to "ENGINE=InnoDB" it worked:)
in PhpMyAdmin on Azure App Service :)
It depend on your access right but for me
this work on phpmyadmin sql console
SHOW VARIABLES;
then after to change some variables
you can do
SET GLOBAL max_connections = 1000;
or
SET ##GLOBAL.max_connections = 1000;
give a try
MySQL configuration file:
/etc/my.cnf
try
mysql --verbose --help | grep -A 1 "Default options"
For MariaDB 10.5 on Ubuntu 20.04.4 LTS (Focal Fossa):
# The MariaDB configuration file
#
# The MariaDB/MySQL tools read configuration files in the following order:
# 0. "/etc/mysql/my.cnf" symlinks to this file, reason why all the rest is read.
# 1. "/etc/mysql/mariadb.cnf" (this file) to set global defaults,
# 2. "/etc/mysql/conf.d/*.cnf" to set global options.
# 3. "/etc/mysql/mariadb.conf.d/*.cnf" to set MariaDB-only options.
# 4. "~/.my.cnf" to set user-specific options.
#
# If the same option is defined multiple times, the last one will apply.

How to change port in mysql in cnf file

I keep trying to change the port number in mysql with a current db that I am using. FOr some reason each time I use this port my spring application throws an error saying the port is already in use even though I use the kill -9 PID command to kill whatever processes is listening on that port. Anyways I used the sudo nano /etc/mysql/my.cnf and created a variable port=3307 like I was told online when searching for a solution. I then restarted mysql using sudo service mysql restart but whenever I enter mysql and enter SHOW VARIABLES WHERE Variable_name = 'port'; it keeps showing up as 3306. The following is my cnf file:
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
port=3307
Please help I have been stuck on this for hours and can't seem to solve it. Also if there is anyway that I could just use port 3306 without having to kill it each time I start up that would be great too. Thanks!
port needs to be under a [mysqld] or equivalent section.
Don't kill -9 on databases. They will terminate eventually. Use service controls like systemctl/service.

Unable to controlling remotely mysql server

I need to access a mysql database remotely (from another device connected to the same network). Searching on internet i've knowed Searching that I have to enable remote control, but I couldn't. How can I do?
When I connect, after asking me for the password, it gives me this error:
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.1.206' (111).
I tried to follow some guides until you have to comment bind-address in my.cnf, because "my.cnf" file is like this:
# The MariaDB configuration file
#
# The MariaDB/MySQL tools read configuration files in the following order:
# 1. "/etc/mysql/mariadb.cnf" (this file) to set global defaults,
# 2. "/etc/mysql/conf.d/*.cnf" to set global options.
# 3. "/etc/mysql/mariadb.conf.d/*.cnf" to set MariaDB-only options.
# 4. "~/.my.cnf" to set user-specific options.
#
# If the same option is defined multiple times, the last one will apply.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]
# Import all .cnf files from configuration directory
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/
so i can't enable the remote control. Any suggestions?

changing my.cnf to setup MySQL replication

I am following this tutorial to set up MySQL replication (I am using Ubuntu).
According to the above tutorial, this is the config that I want in my.cnf file:
[mysqld]
log-bin = mysql-bin
server-id = 1
relay-log = relay-log-slave
gtid-mode =ON
enforce-gtid-consistency
binlog_format = MIXED
log_slave_updates
I have found my.cnf under: /etc/mysql:
When I open my.cnf this is the content (not sure why the file icon appears like a short-cut icon?):
#
# The MySQL database server configuration file.
# /****************************/
# You can copy this to one of: /* <-- what does this mean? */
# - "/etc/mysql/my.cnf" to set global options, /****************************/
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
I was expecting to find settings like: log-bin, server-id, gtid-mode in my.cnf. I thought I should just change the setting values. I am not sure if this is the correct file, because non of those setting values exists here...
Also the comment on top of the file tells me that I can copy it to /etc/mysql/my.cnf but this is where I opened the file from!! I am confused by this comment...
Is this the correct file? Am I supposed to add all the settings myself?
Update
Looks like all the settings exists in /etc/mysql/mysql.conf.d/mysqlid.cnf, am I supposed to change this file?
From MySQL 5.7 default configuration file path is /etc/mysql/mysql.conf.d/mysqld.cnf.
Before 5.7 file path will be /etc/mysql/my.cnf or /etc/my.cnf.
If the parameter does not exist, then add in the [mysqld] section of the configuration file.

Empty Slow Query Log

Good afternoon,
we are trying to have access to our slow query log file at "/var/log/mysql/localhost-slow.log" but it looks like the file is empty. The "my.cnf" file says "!includedir /etc/my.cnf.d", and inside that directory we have the following files:
mysql-clients.cnf;
server.cnf;
tokudb.cnf;
Looking at the "server.cnf" file it looks like it's set properly to write slow queries logs:
#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#
# this is read by the standalone daemon and embedded servers
[server]
# this is only for the mysqld standalone daemon
[mysqld]
slow-query-log = 1
slow-query-log-file = /var/log/mysql/localhost-slow.log
long_query_time = 1
log-queries-not-using-indexes
# this is only for embedded server
[embedded]
# This group is only read by MariaDB-5.5 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mysqld-5.5]
# These two groups are only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]
[mariadb-5.5]
We've tried to SET GLOBAL general_log = 'ON'; but it didn't help. Any idea on what could be the issue?
Verify that it is really reading that file:
mysqld --verbose --help | head -33
You will probably see
Default options are read from the following files in the given order:
followed by a list of file paths.