No Historical Data in IDO DB with Icinga2 Web2 - mysql

Question

Context
I'm working on provisioning Icinga2 Web2 with IDO PostgreSQL and IDO MySQL using Ansible on CentOS 7.
I base my work on the great https://github.com/Icinga/icinga2-ansible roles. I have forked the project (mickael-ange/icinga2-ansible) to added more automation and support such as IDO PostgreSQL. I'm using PostgreSQL server 9.4.
Well, I ends up with a version of icinga2-ansible-web2-ui role which installs Icinga2 Web2 with IDO PostgreSQL or IDO MySQL automatically without wizard.
My Problem
The server seems to work properly at the exception that I cannot figure out how to get data in Icinga2 Web2 interface such as:
Comments and Downtimes from Overview menu
Event Grid, Event Overview, Notifications, Timeline from History menu
I can create Comments and Schedule Downtimes, but they don't get listed in the UI. Downtimes are processed correctly.
I can send Notifications but they don't get listed in the UI as well. Notifications are sent out when SELinux is in permissive mode.
My questions is: is there any IDO or Icinga2 Web2 configuration to get the database populate with Comments, Downtimes, etc..?
My /etc/icinga2/features-enabled/ido-pgsql.conf
library "db_ido_pgsql"
object IdoPgsqlConnection "ido-pgsql" {
user = "icinga"
password = "icinga"
host = "localhost"
database = "icinga"
table_prefix = "icinga_"
instance_name = "icinga2"
instance_description = "icinga2 instance"
cleanup = {
downtimehistory_age = 48h
logentries_age = 31d
}
categories = DbCatConfig | DbCatState
}
Notes
Note 1
I have already searched in most of the documentation of Icinga2 project and forums I could find but no cigar. I'm surprise none gets this problem before so it is probably a problem on my side.
Note 2
After chatting on Icinga IRC channel and doing more tests I figure out that my problem only happens with PostgreSQL backend. It is working as expected with MySQL backend.
Debugging Sessions
Debugging session 1
I have gathered more info/logs about my environments.
PostgreSQL Permissions
I checked my PostgreSQL permissions for icinga user:
TABLE permissions are set to SELECT,INSERT,UPDATE,DELETE
SEQUENCE permissions are set to USAGE
I even tried configuring postgres as user to access to the backend (icinga_ido and icingaweb_ido resources) but I'm still not getting any Comments listed in the UI (after having created a comment).
Here are the PosgreSQL permissions listed by \dp command:
icinga=> \dp
Access privileges
Schema | Name | Type | Access privileges | Column access privileges
--------+-----------------------------------------------------------------+----------+---------------------------+--------------------------
public | icinga_acknowledgements | table | postgres=arwdDxt/postgres+|
| | | icinga=arwd/postgres |
public | icinga_acknowledgements_acknowledgement_id_seq | sequence | postgres=rwU/postgres +|
| | | icinga=U/postgres |
public | icinga_commands | table | postgres=arwdDxt/postgres+|
| | | icinga=arwd/postgres |
public | icinga_commands_command_id_seq | sequence | postgres=rwU/postgres +|
| | | icinga=U/postgres |
public | icinga_commenthistory | table | postgres=arwdDxt/postgres+|
| | | icinga=arwd/postgres |
public | icinga_commenthistory_commenthistory_id_seq | sequence | postgres=rwU/postgres +|
| | | icinga=U/postgres |
<truncated>
Furthermore, I have no error in PostgreSQL logs (e.g. /var/lib/pgsql/9.4/data/pg_log/postgresql-Thu.log) when creatinh Comment from Icinga UI.
However, I figured out there are errors when trying to browse SEQUENCE from pgAdmin UI using icinga user (no error with postgres user):
< 2016-01-21 03:55:52.436 GMT >STATEMENT: SELECT last_value, min_value, max_value, cache_value, is_cycled, increment_by, is_called
FROM icinga_acknowledgements_acknowledgement_id_seq
But I'm not sure if it makes sense or not as icinga user has only USAGE permission on SEQUENCE.
Comparing PostgreSQL and MySQL Backends' Debug Logs
So I ends up with comparing Icinga debug logs on both sides (MySQL IDO and PostgreSQL IDO) when I creates a comment. I can see that only with MySQL backend there are the SQL statements I was looking for, whom insert data in icinga_externalcommands, icinga_comments, and icinga_commenthistory tables.
I don't have enough reputation to provide more than 2 links (e.g. pastebin). So here are the logs.
Adding a Comment with PostgreSQL backend:
[2016-01-21 04:14:00 +0000] information/ExternalCommandListener: Executing external command: [1453349640] ADD_HOST_COMMENT;icinga2-web2-postgres;1;icingaadmin;dededewdwwewdew
[2016-01-21 04:14:00 +0000] debug/DbEvents: add external command history
[2016-01-21 04:14:00 +0000] notice/ExternalCommandProcessor: Creating comment for host icinga2-web2-postgres
[2016-01-21 04:14:00 +0000] information/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/icinga2-web2-postgres-1453347670-1/conf.d/comments/icinga2-web2-postgres!icinga2-web2-postgres-1453349640-1.conf
[2016-01-21 04:14:00 +0000] information/ConfigItem: Committing config items
[2016-01-21 04:14:00 +0000] warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
[2016-01-21 04:14:00 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 57:1-57:65) for type 'Service' does not match anywhere!
[2016-01-21 04:14:00 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 65:1-65:53) for type 'Service' does not match anywhere!
[2016-01-21 04:14:00 +0000] information/ConfigItem: Instantiated 1 Comment.
[2016-01-21 04:14:00 +0000] information/ConfigItem: Triggering Start signal for config items
[2016-01-21 04:14:00 +0000] information/ConfigItem: Activated all objects.
[2016-01-21 04:14:00 +0000] notice/Comment: Added comment 'icinga2-web2-postgres!icinga2-web2-postgres-1453349640-1'.
[2016-01-21 04:14:01 +0000] debug/IdoPgsqlConnection: Query: COMMIT
[2016-01-21 04:14:01 +0000] debug/IdoPgsqlConnection: Query: BEGIN
Adding a Comment with MySQL backend:
[2016-01-21 04:12:49 +0000] information/ExternalCommandListener: Executing external command: [1453349569] ADD_HOST_COMMENT;icinga2-web2-mysql;1;icingaadmin;yggygyyj
[2016-01-21 04:12:49 +0000] debug/DbEvents: add external command history
[2016-01-21 04:12:49 +0000] notice/ExternalCommandProcessor: Creating comment for host icinga2-web2-mysql
[2016-01-21 04:12:49 +0000] information/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/icinga2-web2-mysql-1453347256-1/conf.d/comments/icinga2-web2-mysql!icinga2-web2-mysql-1453349569-0.conf
[2016-01-21 04:12:49 +0000] information/ConfigItem: Committing config items
[2016-01-21 04:12:49 +0000] warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
[2016-01-21 04:12:49 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 57:1-57:65) for type 'Service' does not match anywhere!
[2016-01-21 04:12:49 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 65:1-65:53) for type 'Service' does not match anywhere!
[2016-01-21 04:12:49 +0000] information/ConfigItem: Instantiated 1 Comment.
[2016-01-21 04:12:49 +0000] information/ConfigItem: Triggering Start signal for config items
[2016-01-21 04:12:49 +0000] information/ConfigItem: Activated all objects.
[2016-01-21 04:12:49 +0000] notice/Comment: Added comment 'icinga2-web2-mysql!icinga2-web2-mysql-1453349569-0'.
[2016-01-21 04:12:50 +0000] debug/IdoMysqlConnection: Query: INSERT INTO icinga_externalcommands (command_args, command_name, command_type, endpoint_object_id, entry_time, instance_id) VALUES ('icinga2-web2-mysql;1;icingaadmin;yggygyyj', 'ADD_HOST_COMMENT', '1', 1, FROM_UNIXTIME(1453349569), 1)
[2016-01-21 04:12:50 +0000] debug/IdoMysqlConnection: Query: INSERT INTO icinga_comments (author_name, comment_data, comment_source, comment_time, comment_type, endpoint_object_id, entry_time, entry_time_usec, entry_type, expires, instance_id, internal_comment_id, is_persistent, name, object_id) VALUES ('icingaadmin', 'yggygyyj', '1', FROM_UNIXTIME(1453349569), '2', 1, FROM_UNIXTIME(1453349569), '947563', '1', '0', 1, '1', '1', 'icinga2-web2-mysql!icinga2-web2-mysql-1453349569-0', 68)
[2016-01-21 04:12:50 +0000] debug/IdoMysqlConnection: Query: INSERT INTO icinga_commenthistory (author_name, comment_data, comment_source, comment_time, comment_type, endpoint_object_id, entry_time, entry_time_usec, entry_type, expires, instance_id, internal_comment_id, is_persistent, name, object_id) VALUES ('icingaadmin', 'yggygyyj', '1', FROM_UNIXTIME(1453349569), '2', 1, FROM_UNIXTIME(1453349569), '947563', '1', '0', 1, '1', '1', 'icinga2-web2-mysql!icinga2-web2-mysql-1453349569-0', 68)
[2016-01-21 04:12:50 +0000] debug/IdoMysqlConnection: Query: COMMIT
[2016-01-21 04:12:50 +0000] debug/IdoMysqlConnection: Query: BEGIN
Restarting Icinga2 with PostgreSQL and MySQL Backends
When restarting Icinga2 with both PostgreSQL and MySQL Backends, I have pretty much the same logs which sounds good!
# Icinga2 restart with PostgreSQL
[2016-01-21 05:02:11 +0000] information/Application: Received request to shut down.
[2016-01-21 05:02:11 +0000] information/Application: Shutting down...
[2016-01-21 05:02:11 +0000] information/CheckerComponent: Checker stopped.
[2016-01-21 05:02:11 +0000] information/DbConnection: Resuming IDO connection: ido-pgsql
[2016-01-21 05:02:11 +0000] information/ConfigItem: Activated all objects.
[2016-01-21 05:02:11 +0000] information/ConfigCompiler: Compiling config file: /var/lib/icinga2/modified-attributes.conf
[2016-01-21 05:02:11 +0000] information/IdoPgsqlConnection: pgSQL IDO instance id: 1 (schema version: '1.14.0')
# Icinga2 restart with MySQL
[2016-01-21 05:03:20 +0000] information/Application: Received request to shut down.
[2016-01-21 05:03:20 +0000] information/Application: Shutting down...
[2016-01-21 05:03:20 +0000] information/CheckerComponent: Checker stopped.
[2016-01-21 05:03:20 +0000] information/DbConnection: Resuming IDO connection: ido-mysql
[2016-01-21 05:03:20 +0000] information/ConfigItem: Activated all objects.
[2016-01-21 05:03:20 +0000] information/ConfigCompiler: Compiling config file: /var/lib/icinga2/modified-attributes.conf
[2016-01-21 05:03:20 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.0')
Packages Version
icinga2-2.4.1-1.el7.centos.x86_64
icinga2-ido-pgsql-2.4.1-1.el7.centos.x86_64
postgresql94-9.4.5-1PGDG.rhel7.x86_64
postgresql94-server-9.4.5-1PGDG.rhel7.x86_64
How to Reproduce this Issue
For those who has the energy to reproduce the problem you can clone mickael-ange/icinga2-ansible Github repo (branch: Issue-No-Historical-Data-in-Icinga2-Web2-and-DB) and follow instructions at icinga2-we2-vagrant.md. You need to install the required tools if it is not already the case. Then install role dependencies with librarian-ansible. Finally, I wrote a section (#ido-no-historical-data-in-icinga2-web2-and-db) to reproduce the problem.
Thank you in advance to take the time read until here!
Mickael

Answer 1

Finally, I figured out why historical data were not inserted into the DB, so I'm here responding to myself.
As explained in the documentation:
categories Optional. The types of information that should be written to the database.
So when I removed categories from my ido-pgsql.conf configuration. I restart Icinga2 then the data are written in the DB as expected. Comments, Downtimes, well everything I wanted is listed the UI now.
I also removed the cleanup section as it is probably not what I want too.
The final ido-pgsql.conf configuration:
library "db_ido_pgsql"
object IdoPgsqlConnection "ido-pgsql" {
user = "icinga"
password = "icinga"
host = "localhost"
database = "icinga"
table_prefix = "icinga_"
instance_name = "icinga2"
instance_description = "icinga2 instance"
}