Change audit log to JSON format in Percona Mysql - mysql

How to modify the system variable audit_log_format to change to JSON format. The variable is read-only and cannot be changed when the server is running.
mysql> SHOW variables LIKE 'audit%';
+-----------------------------+---------------+
| Variable_name | Value |
+-----------------------------+---------------+
| audit_log_buffer_size | 1048576 |
| audit_log_exclude_accounts | |
| audit_log_exclude_commands | |
| audit_log_exclude_databases | |
| audit_log_file | audit.log |
| audit_log_flush | OFF |
| audit_log_format | OLD |
| audit_log_handler | FILE |
| audit_log_include_accounts | |
| audit_log_include_commands | |
| audit_log_include_databases | |
| audit_log_policy | ALL |
| audit_log_rotate_on_size | 0 |
| audit_log_rotations | 0 |
| audit_log_strategy | ASYNCHRONOUS |
| audit_log_syslog_facility | LOG_USER |
| audit_log_syslog_ident | percona-audit |
| audit_log_syslog_priority | LOG_INFO |
+-----------------------------+---------------+
18 rows in set (0.00 sec)
I edit my my.cnf in this way, but it doesnt work
bash-4.4$ cat ./etc/my.cnf
#
# The Percona Server 5.7 configuration file.
#
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
# Please make any edits and changes to the appropriate sectional files
# included below.
#
!includedir /etc/my.cnf.d/
!includedir /etc/percona-server.conf.d/
[mysqld]
## Audit Logging ##
audit_log_format=JSON

The audit plugin is need to be loaded at first.
[mysqld]
plugin-load="audit_log=audit_log.so"
audit_log_policy=ALL
audit_log_format=JSON
audit_log_rotate_on_size=1024M
audit_log_rotations=10

Related

xtrabackup does not write gtid information in xtrabackup_binlog_info

Software versions:
xtrabackup 8.0.12
percona-xtradb-cluster-server 8.0.18-9
I am running xtrabackup with this options:
--defaults-file=/etc/mysql/my.cnf --backup --user=backup
--password=**** --parallel=4 --no-timestamp --target-dir=/my-backup-dir
Some of server options:
binlog_format | ROW
gtid_mode | ON_PERMISSIVE
enforce_gtid_consistency | ON
File xtrabackup_binlog_info has only binlog file name and position:
mysql-bin.000159 251
No GTID, so I can not create GTID-based replication restoring a slave from this backup.
What should I do to make xtrabackup include this information?
UPDATE:
Check if GTIDs are enabled:
show global variables like '%gtid%';
| Variable_name | Value |
| binlog_gtid_simple_recovery | ON |
| enforce_gtid_consistency | ON |
| gtid_executed | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3399594,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
| gtid_executed_compression_period | 1000 |
| gtid_mode | ON_PERMISSIVE |
| gtid_owned | |
| gtid_purged | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-2661056,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
| session_track_gtids | OFF |
show master status;
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
| mysql-bin.000166 | 15285372 | | | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3358798,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
(pavel.selivanov#localhost) [kassa_prod]> show binlog events in 'mysql-bin.000166' limit 7;
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info
| mysql-bin.000166 | 4 | Format_desc | 4315 | 124 | Server ver: 8.0.18-9, Binlog ver: 4 |
| mysql-bin.000166 | 124 | Previous_gtids | 4315 | 251 | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3347159,
de211648-2642-ee18-628d-dc48283b005c:3697598:3877279-10141440 |
| mysql-bin.000166 | 251 | Gtid | 4315 | 330 | SET ##SESSION.GTID_NEXT= 'c0e3de06-a2a6-11ea-913c-c7b046cf5782:3347160' |
| mysql-bin.000166 | 330 | Query | 4315 | 411 | BEGIN |
| mysql-bin.000166 | 411 | Table_map | 4315 | 499 | table_id: 150 (db.table) |
| mysql-bin.000166 | 499 | Update_rows | 4315 | 3475 | table_id: 150 flags: STMT_END_F |
| mysql-bin.000166 | 3475 | Xid | 4315 | 3506 | COMMIT /* xid=9611331 */
From the manual:
ON_PERMISSIVE: New transactions are GTID transactions. Replicated transactions can be either anonymous or GTID transactions.
Check with show global variables like 'gt%'; or in your binary logs, if you actually have GTID transactions.
You actually don't have anything special to do, to have xtrabackup include GTIDs in the xtrabackup_binlog_info file: How to create a new (or repair a broken) GTID based slave

How do I check the value of 'binlog_do_db' in MySQL?

I am trying to debug why my DML statements aren't ending up in the binlogs. The master and slave appear to be talking to eachother and the slave status indicates that it caught up properly to the most recent log file.
I did a show variables LIKE '%binlog%'; with this output, but binlog_do_db is not there.
+-----------------------------------------+----------------------+
| Variable_name | Value |
+-----------------------------------------+----------------------+
| binlog_cache_size | 32768 |
| binlog_checksum | CRC32 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_error_action | ABORT_SERVER |
| binlog_format | ROW |
| binlog_group_commit_sync_delay | 0 |
| binlog_group_commit_sync_no_delay_count | 0 |
| binlog_gtid_simple_recovery | ON |
| binlog_max_flush_queue_time | 0 |
| binlog_order_commits | ON |
| binlog_row_image | FULL |
| binlog_rows_query_log_events | OFF |
| binlog_stmt_cache_size | 32768 |
| innodb_api_enable_binlog | OFF |
| innodb_locks_unsafe_for_binlog | OFF |
| log_statements_unsafe_for_binlog | ON |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 104857600 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| sync_binlog | 1 |
+-----------------------------------------+----------------------+
After much effort and grief, I found that in the previous administrator must have been trying to get replication to work and put these lines into the /etc/mysql/mysql.conf.d/mysqld.cnf file
# Setup Incremental Backups
log-bin = /srv/ddisk/backups/mysql/mysql_inc
expire_logs_days = 3
max_binlog_size = 100M
server-id = 1
binlog_do_db = include_database_name
The last line is probably the culprit (though I will likely clean out that whole config block since I have them in the my.cnf config file already. In any case, I am now trying to verify that this configuration is actually being read by MySQL, but I'm unable to figure out what query to run to find this variable.
Would someone happen to know how to find binlog_do_db?
Thanks ahead of time
You can check using mysql prompt on the master node below:
mysql> SHOW MASTER STATUS;
It will show Binary log file, position, Binlog_Do_DB, Binlog_Ignore_DB parameter and so on.

How to filter mysql audit log by user account

My issue is even i disable the root user from audit logging but still logging for these user. Anyone please help. Here is i did step by step.
[Setp -1] Check the audit log variable.
mysql> SHOW VARIABLES LIKE 'audit_log%';
+-----------------------------+--------------+
| Variable_name | Value |
+-----------------------------+--------------+
| audit_log_buffer_size | 1048576 |
| audit_log_connection_policy | ALL |
| audit_log_current_session | ON |
| audit_log_exclude_accounts | |
| audit_log_file | audit.log |
| audit_log_flush | OFF |
| audit_log_format | OLD |
| audit_log_include_accounts | |
| audit_log_policy | ALL |
| audit_log_rotate_on_size | 0 |
| audit_log_statement_policy | ALL |
| audit_log_strategy | ASYNCHRONOUS |
+-----------------------------+--------------+
12 rows in set (0.00 sec)
[Setp-2]
The following statement is disable audit logging for root account.
-- audit_log_include_accounts to NULL
SET GLOBAL audit_log_include_accounts = NULL;
SET GLOBAL audit_log_exclude_accounts = root#%;
Note: I used the root#% instead root#localhost because of this database server can access from another ip address.
[Setp-3] I call the select statement SELECT * FROM SSVR_AUDIT_LOG from remote PC.
[Step-4] I checked the audit log in DB server.
<AUDIT_RECORD TIMESTAMP="2016-04-22T03:49:11 UTC" RECORD_ID="593_2016-04-22T01:28:17" NAME="Query" CONNECTION_ID="6" STATUS="0" STATUS_CODE="0" USER="root[root] # [162.16.22.48]" OS_LOGIN="" HOST="" IP="162.16.22.48" COMMAND_CLASS="show_create_table" SQLTEXT="SHOW CREATE TABLE `SSVR_AUDIT_LOG`"/>
<AUDIT_RECORD TIMESTAMP="2016-04-22T03:49:12 UTC" RECORD_ID="594_2016-04-22T01:28:17" NAME="Query" CONNECTION_ID="7" STATUS="0" STATUS_CODE="0" USER="root[root] # [162.16.22.48]" OS_LOGIN="" HOST="" IP="162.16.22.48" COMMAND_CLASS="select" SQLTEXT="SELECT * FROM `SSVR_AUDIT_LOG` LIMIT 0, 1000"/>
<AUDIT_RECORD TIMESTAMP="2016-04-22T03:49:12 UTC" RECORD_ID="595_2016-04-22T01:28:17" NAME="Query" CONNECTION_ID="7" STATUS="0" STATUS_CODE="0" USER="root[root] # [162.16.22.48]" OS_LOGIN="" HOST="" IP="162.16.22.48" COMMAND_CLASS="show_fields" SQLTEXT="SHOW COLUMNS FROM `tldssvr`.`SSVR_AUDIT_LOG`"/>
<AUDIT_RECORD TIMESTAMP="2016-04-22T03:49:13 UTC" RECORD_ID="596_2016-04-22T01:28:17" NAME="Quit" CONNECTION_ID="7" STATUS="0" STATUS_CODE="0" USER="root" OS_LOGIN="" HOST="" IP="162.16.22.48" COMMAND_CLASS="connect"/>
Here is my reference link enter link description here
I got the answer for my question. Here is correct answer. When you facing like that issue, you can follow below the steps.
Audit Log Filtering by Account
List all ‘audit log’ configuration items
> mysql -u root -p
> SHOW VARIABLES LIKE ‘audit_log%’;
+-----------------------------+--------------+
| Variable_name | Value |
+-----------------------------+--------------+
| audit_log_buffer_size | 1048576 |
| audit_log_connection_policy | ALL |
| audit_log_current_session | OFF |
| audit_log_exclude_accounts | |
| audit_log_file | audit.log |
| audit_log_flush | OFF |
| audit_log_format | OLD |
| audit_log_include_accounts | |
| audit_log_policy | ALL |
| audit_log_rotate_on_size | 0 |
| audit_log_statement_policy | ALL |
| audit_log_strategy | ASYNCHRONOUS |
+-----------------------------+--------------+
To add the remote application server host name and ip address in database server.
> cat /etc/hosts
> 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
162.16.22.48 App_PC
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
To disable audit logging only for the application database user (root) local host and remote host accounts.
> mysql –u root –p
>SET GLOBAL audit_log_include_accounts = NULL;
>SET GLOBAL audit_log_exclude_accounts = 'root#localhost,root#App_PC';
List all ‘audit log’ configuration items and check the audit_log_exclude_account value.
> SHOW VARIABLES LIKE 'audit_log%';
> +-----------------------------+----------------------------+
| Variable_name | Value |
+-----------------------------+----------------------------+
| audit_log_buffer_size | 1048576 |
| audit_log_connection_policy | ALL |
| audit_log_current_session | OFF |
| audit_log_exclude_accounts | root#localhost,root#App_PC |
| audit_log_file | audit.log |
| audit_log_flush | OFF |
| audit_log_format | OLD |
| audit_log_include_accounts | |
| audit_log_policy | ALL |
| audit_log_rotate_on_size | 0 |
| audit_log_statement_policy | ALL |
| audit_log_strategy | ASYNCHRONOUS |
+-----------------------------+----------------------------+

How can I start and check my MySQL log?

I want to check the log in MySQL to see the queries that are being run by my application. How can I do this? I am using XAMPP and the directory to MySQL is C:\xampp\mysql.
This is what I get when I do show variables like '%log%';
mysql> show variables like '%log%';
+---------------------------------+------------------------------------+
| Variable_name | Value |
+---------------------------------+------------------------------------+
| back_log | 50 |
| binlog_cache_size | 32768 |
| binlog_format | STATEMENT |
| expire_logs_days | 0 |
| general_log | OFF |
| general_log_file | C:/xampp/mysql/data/mysql.log |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_buffer_size | 8388608 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | C:\xampp\mysql\data\ |
| innodb_mirrored_log_groups | 1 |
| log | OFF |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| log_bin_trust_routine_creators | OFF |
| log_error | C:\xampp\mysql\data\mysql.err |
| log_output | FILE |
| log_queries_not_using_indexes | OFF |
| log_slave_updates | OFF |
| log_slow_queries | OFF |
| log_warnings | 1 |
| max_binlog_cache_size | 4294963200 |
| max_binlog_size | 1073741824 |
| max_relay_log_size | 0 |
| relay_log | |
| relay_log_index | |
| relay_log_info_file | relay-log.info |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| slow_query_log | OFF |
| slow_query_log_file | C:/xampp/mysql/data/mysql-slow.log |
| sql_log_bin | ON |
| sql_log_off | OFF |
| sql_log_update | ON |
| sync_binlog | 0 |
+---------------------------------+------------------------------------+
37 rows in set (0.00 sec)
Enable general query log by the following query in mysql command line
SET GLOBAL general_log = 'ON';
Now open C:/xampp/mysql/data/mysql.log and check query log
If it fails, open your my.cnf file. For windows its my.ini file and enable it there. Just make sure its in the [mysqld] section
[mysqld]
general_log = 1
Note: In xampp my.ini file can be either found in xampp\mysql or in c:\windows directory
Set up the General Query Log.
Its given on OFFICIAL MYSQL website.
SET GLOBAL general_log = 'ON';
You can also use custom path:
[mysqld]
# Set Slow Query Log
long_query_time = 1
slow_query_log = 1
slow_query_log_file = "C:/slowquery.log"
#Set General Log
log = "C:/genquery.log"
Seems like the general query log is the file that you need. A good introduction to this is at http://dev.mysql.com/doc/refman/5.1/en/query-log.html
For me, general_log didn't worked. But adding this to my.ini worked
[mysqld]
log-output=FILE
slow_query_log = 1
slow_query_log_file = "d:/temp/developer.log"

MySQL logging confusion

I need to turn this on/enable it. I found some info on the net but am very confused now, I don't know where to start.
This is what I have so far. I logged onto MySQL as root and did:
show variables like '%log%';
and got:
Variable_name | Value |
+---------------------------------+----------------------------------+
| back_log | 50 |
| binlog_cache_size | 32768 |
| binlog_format | STATEMENT |
| expire_logs_days | 10 |
| general_log | ON |
| general_log_file | /var/lib/mysql/helloise.log |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | ./ |
| innodb_mirrored_log_groups | 1 |
| log | ON |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| log_bin_trust_routine_creators | OFF |
| log_error | /var/log/mysql/error.log |
| log_output | FILE |
| log_queries_not_using_indexes | OFF |
| log_slave_updates | OFF |
| log_slow_queries | OFF |
| log_warnings | 1 |
| max_binlog_cache_size | 4294963200 |
| max_binlog_size | 104857600 |
| max_relay_log_size | 0 |
| relay_log | |
| relay_log_index | |
| relay_log_info_file | relay-log.info |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| slow_query_log | OFF |
| slow_query_log_file | /var/lib/mysql/helloise-slow.log |
| sql_log_bin | ON |
| sql_log_off | OFF |
| sql_log_update | ON |
| sync_binlog | 0
I did: SET GLOBAL general_log = 1 to set general_log = ON. Will all queries be logged in /var/lib/mysql/helloise.log?
And when I try to access the mysql directory via helloises#helloise:/var/lib$ cd mysql/ I get permission denied
and then I also have a log directory:
helloises#helloise:/var/log$
with a whole lot of log files ex: mysql.log
Which log file will contain my queries so I can track what my db is doing and see how long queries take?
Firs of all you got a permission denied because you hav not the privileges to read it. Try a
ls -l
on that file and see what permission are defined. Then yuo can use chmod to change permissions. Or simply try to read that file as a root user.
The general log is a logging feature for query log, reading, from guide:
The general query log is a general record of what mysqld is doing. The
server writes information to this log when clients connect or
disconnect, and it logs each SQL statement received from clients. The
general query log can be very useful when you suspect an error in a
client and want to know exactly what the client sent to mysqld.
The error log:
The error log contains information indicating when mysqld was started
and stopped and also any critical errors that occur while the server
is running. If mysqld notices a table that needs to be automatically
checked or repaired, it writes a message to the error log.
For understand the differences i suggest you to read that site:
http://dev.mysql.com/doc/refman/5.1/en/server-logs.html
The file that contains the queries is the one in /var/lib/mysql/helloise.log
For the slow query logs you can check:
http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html
it logs all the query that take more time than long_query_time seconds (that variable must be defined by yourself). I don't found a way to show the execution time of a query, but check that page http://dev.mysql.com/doc/refman/5.1/en/log-destinations.html it could help you on how to configure general/slowquery logging.
Hope this help.