Is there a command line tool that will auto pretty-print json? - json

I know that tools like 'jq' can let you add a pipe to the command and format it, but you can't use it for situations like displaying real time json logs from a local server. Is there a tool that can achieve that?

I assume you use a default linux distribution with systemd installed, and all services will communicate with systemd to make protocol entries.
Back in the days linux created log files only, but they got to big. Then they added logrotate to keep it a bit more clean.
A up to date linux distribution uses systemd to store protocol entries, which is much better.
To read a protocol you need to find the service you are looking for first.
To list all services connected with systemd call ...
systemctl list-units | grep service
You could use it without grep if you really cant use pipes.
Anyway, pick the service you like and call its protocol via journalctl. Here are some examples ...
journalctl -u ssh.service -p warning -n 30 # (sudo) last 30 warnings and errors SSH created
journalctl -u apache2.service -p err --no-pager # (sudo) all Apache entries without using the pager (less)
journalctl -k -p err --since today # (sudo) kernel error messages of today
Add the option -o json or -o json-pretty to it and you will have the output you wanted. You can also add option -F or --follow to show most recent journal entries, and continuously print new entries as they are appended to the journal. This way you could make cool backgrounds with conky for example.
You can also get the status of your service via systemctl ...
systemctl status apache2.service # (sudo) status of Apache
This way you can pretty print ...
real time json logs from a local server
like you wanted, and without using jq ...
admin#suse:~$ journalctl -u apache2.service -p warning -n 1 -o json-pretty --no-pager
{
"__CURSOR" : "s=65cbb17b253f44c0b800cee690cfb9ee;i=187fd;b=d3effa777ddf4ba9b4f82f126be4439e;m=51cb2bb2d53;t=54b7c4c9ffbeb;x=9fd3b1bc1e50a798",
"__REALTIME_TIMESTAMP" : "1490372117134315",
"__MONOTONIC_TIMESTAMP" : "5620815834451",
"_BOOT_ID" : "d3effa777ddf4ba9b4f82f126be4439e",
"_TRANSPORT" : "syslog",
"_SYSTEMD_SLICE" : "system.slice",
"_MACHINE_ID" : "dc3b59f36cc34cfa8873c5b530577a2a",
"_HOSTNAME" : "suse",
"SYSLOG_FACILITY" : "1",
"_UID" : "33",
"_GID" : "33",
"PRIORITY" : "4",
"SYSLOG_IDENTIFIER" : "/usr/lib/cgi-bin/captainlog/search.sh",
"MESSAGE" : "Request for the specified frontend is not possible. REMOTE_ADDR: xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxx:xxxx",
"_COMM" : "logger",
"_CAP_EFFECTIVE" : "0",
"_SYSTEMD_CGROUP" : "/system.slice/apache2.service",
"_SYSTEMD_UNIT" : "apache2.service",
"SYSLOG_PID" : "21314",
"_PID" : "21314",
"_SOURCE_REALTIME_TIMESTAMP" : "1490372117133991"
}
Here is a good guide I found on the internet.

Related

Zabbix sender. Discovery rules. Host prototypes

I would like to automate host creation on zabbix server without using agent on hosts. Tried to use Discovery rules and sending JSON data with zabbix_sender. But without luck. Server does not accept data.
Environment:
Zabbix server 3.4 installed on Centos 7.Hosts with Windows or Ubuntu.
On server I created host with name zab_trap
In that host I created Discovery rule with key zab_trap.discovery and type Zabbix_trapper. Then in Discovery rule I created Host prototype with name {#RH.NAME}.
Command line with JSON "data":
zabbix_sender.exe -z zab_server -s zab_trap -k zab_trap.discovery -o "{"data":[{"{#RH.NAME}":"HOST1"}]}"
I expected that "HOST1" will be created. But after execution I got:
"info from server: "processed: 0; failed: 1; total: 1; seconds spent: 0.000188"
sent: 1; skipped: 0; total: 1"
And there is no error in zabbix_server.log (with debug level 5)
I see this:
trapper got '{"request":"sender data","data":[{"host":"zab_trap","key":"zab_trap.discovery","value":"'{data:[{{#RH.NAME}:HOST1}]}'"}]}'
I think that maybe there is something wrong with JSON syntax.
Please help.
It seems I have found solution. Problem is hidden in a way to send JSON. As I understood it does not work properly or there is problem with syntax(quotes) if write JSON directly in command line. But it works if zabbix_sender send file with JSON.
Command line:
zabbix_sender -z zab_server -s zab_trap -i test.json
File test.json contain line:
- zab_trap.discovery {"data":[{"{#RH.NAME}":"HOST1"}]}
Host created.
If you want to use the command line, without file json, you need to clean the string with:
zabbix_sender.exe -z zab_server -s zab_trap -k zab_trap.discovery -o "$(echo '{"data":[{"{#RH.NAME}":"HOST1"}]}' | tr -cd '[:print:]')"

Orion Context Broker functional test failure

I have successfully forked and built the Context Broker source code on a CentOS 6.9 VM and now I am trying to run the functional tests as the official documentation suggests. First, I installed the accumulator-server.py script:
$ make install_scripts INSTALL_DIR=~
Verified that it is installed:
$ accumulator-server.py -u
Usage: accumulator-server.py --host <host> --port <port> --url <server url> --pretty-print -v -u
Parameters:
--host <host>: host to use database to use (default is '0.0.0.0')
--port <port>: port to use (default is 1028)
--url <server url>: server URL to use (default is /accumulate)
--pretty-print: pretty print mode
--https: start in https
--key: key file (only used if https is enabled)
--cert: cert file (only used if https is enabled)
-v: verbose mode
-u: print this usage message
And then run the functional tests:
$ make functional_test INSTALL_DIR=~
But the test fails and exits with the message below:
024/927: 0000_ipv6_support/ipv4_ipv6_both.test ........................................................................ (FAIL 11 - SHELL-INIT exited with code 1) testHarness.sh/IPv6 IPv4 Both : (0000_ipv6_support/ipv4_ipv6_both.test)
make: *** [functional_test] Error 11
$
I checked the file ../0000_ipv6_support/ipv4_ipv6_both.shellInit.stdout for any hint on what may be going wrong but error log does not lead me anywhere:
{ "dropped" : "ftest", "ok" : 1 }
accumulator running as PID 6404
Unable to start listening application after waiting 30
Does anyone have any idea about what may be going wrong here?
I checked the script which prints the error line Unable to start listening application after waiting 30 and noticed that stderr for accumulator-server.py is logged into the /tmp folder.
The accumulator_9977_stderr file had this log: 0000_ipv6_support/ipv4_ipv6_both.shellInit: line 27: accumulator-server.py: command not found
Once I saw this log I understood the mistake I made. I was running the
functional tests with sudo and the secure_path was being used instead of my PATH variable.
So at the end, running the functional tests with the command below solved the issue for me.
$ sudo "PATH=$PATH" make functional_test INSTALL_DIR=~
This can also be solved by editing the /etc/sudoers file by:
$ sudo visudo
and modifying the secure_path value.

userparameters and ZBX_NOTSUPPORTED

I want to ping an external ip from all of my servers that run zabbix agent.
I searched and find some articles about zabbix user parameters.
In /etc/zabbix/zabbix_agentd.conf.d/ I created a file named userparameter_ping.conf with following content:
UserParameter=checkip[*],ping -c4 8.8.8.8 && echo 0 || echo 1
I created an item named checkip in zabbix server with a graph but got no data. After some another digging I found zabbix_get and tested my userparameter but I got the error : ZBX_NOTSUPPORTED
# zabbix_get -s 172.20.4.43 -p 10050 -k checkip
my zabbix version :
Zabbix Agent (daemon) v2.4.5 (revision 53282) (21 April 2015)
Does anybody know what I can do to address this?
After some change and talks with folks in mailing list finally it worked but how :
first i created a file in :
/etc/zabbix/zabbix_agentd.conf.d/
and add this line :
UserParameter=checkip[*],ping -W1 -c2 $1 >/dev/null 2>&1 && echo 0 || echo 1
and run this command :
./sbin/zabbix_agentd -t checkip["8.8.8.8"]
checkip[8.8.8.8] [t|0]
so everything done but Timeout option is very important for us :
add time out in /etc/zabbix/zabbix_agentd.conf
Timeout=30
Timeout default is 3s so if we run
time ping -W1 -c2 8.8.8.8
see maybe it takes more than 3s so you got error :
ZBX_NOTSUPPORTED
It can be anything. For example timeout - default timeout is 3 sec and ping -c4 requires at least 3 seconds, permission/path to ping, not restarted agent, ...
Increase debug level, restart agent and check zabbix logs. Also you can test zabbix_agentd directly:
zabbix_agentd -t checkip[]
[m|ZBX_NOTSUPPORTED] [Timeout while executing a shell script.] => Timeout problem. Edit zabbix_agentd.conf and increase Timeout settings. Default 3 seconds are not the best for your ping, which needs 3+ seconds.
If you need more than 30s for the execution, you can use the nohup (command..) & combo to curb the timeout restriction.
That way, if you generate some file with the results, in the next pass, you can read the file and get back the results without any need to wait at all.
For those who may be experiencing other issues with the same error message.
It is important to run zabbix_agentd with the -c parameter:
./sbin/zabbix_agentd -c zabbix_agentd.conf --test checkip["8.8.8.8"]
Otherwise zabbix might not pick up on the command and will thus yield ZBX_NOTSUPPORTED.
It also helps to isolate the command into a script file, as Zabbix will butcher in-line commands in UserParameter= much more than you'd expect.
I defined two user parameters like this for sync checking between to samba DCs.
/etc/zabbix/zabbix_agentd.d/userparameter_samba.conf:
UserParameter=syncma, sudo samba-tool drs replicate smb1 smb2 cn=schema,cn=configuration,dc=domain,dc=com
UserParameter=syncam, sudo samba-tool drs replicate smb2 smb1 cn=schema,cn=configuration,dc=domain,dc=com
and also provided sudoer access for Zabbix user to execute the command. /etc/sudoers.d/zabbix:
Defaults:zabbix !syslog
Defaults:zabbix !requiretty
zabbix ALL=(ALL) NOPASSWD: /usr/bin/samba-tool
zabbix ALL=(ALL) NOPASSWD: /usr/bin/systemctl
And "EnableRemoteCommands" is enabled on my zabbix_aganetd.conf, sometimes when I run
zabbix_get -s CLIENT_IP -p10050 -k syncma or
zabbix_get -s CLIENT_IP -p10050 -k syncam
I get the error ZBX_NOTSUPPORTED: Timeout while executing a shell script.
but after executing /sbin/zabbix_agentd -t syncam on the client, Zabbix server just responses normally.
Replicate from smb2 to smb1 was successful.
and when it has a problem I get below error on my zabbix.log
failed to kill [ sudo samba-tool drs replicate smb1 smb2 cn=schema,cn=configuration,dc=domain,dc=com]: [1] Operation not permitted
It seems like it is a permission error! but It just resolved after executing /sbin/zabbix_agentd -t syncam but I am not sure the error is gone permanently or will happen at the next Zabbix item check interval.

Script for CentOS to restart MySQL service whenever it is stopped

I want to write a script for restart MySQL service whenever it is in stopped status.
I don't know how to write the IF statement on this script.
#vastlysuperiorman
if [ ! $(ps aux | grep mysql) ]
then
service mysql restart
fi
First, don't forget to protect your operand. See this reminder : bash : Illegal number
Then, you could simplify by :
[[ $(ps -ef | grep [m]ysql) ]] || service mysql restart
(I prefere to use ps -ef because it's more POSIX compliant even if I always use [[ instead of test or [ in Bash).
Finaly, it's better to check the status with service mysql status. You could try something like instead :
[[ $(service mysql status) =~ running ]] || service mysql restart
#user33398
You could try to use this test in a loop or with watch command for example. Or you could use more smart tools like cron or jenkins or nagios to do that.
No sense re-inventing the wheel. There are great software solutions out there that can do this for you. Monit is a good open source example.
If you do opt to write a script, a simple solution could be to do something like the following:
if [ ! $(ps aux | grep mysql) ]
then
service mysql restart
fi

dtrace: failed to compile script probe description pid29441::*Slave_open_temp_tables*:return does not match any probes?

Due to this problem, I'm going to use dtrace to find out what the slave SQL thread is doing with temporary tables follow this guide.
Here is my script:
#!/usr/sbin/dtrace -s
#pragma D option quiet
dtrace:::BEGIN
{
printf("Tracing... Hit Ctrl-C to end.\n");
}
pid$target::*mysql_parse*:entry
{
self->query = copyinstr(arg1);
}
pid$target::*Slave_open_temp_tables*:return
{
#query[self->query] = count();
}
and this is what I got when running:
# ./Slave_open_temp_tables.d -p `pgrep -x mysqld`
proc-stub:rd_event_enable
proc-stub:rd_errstr err=26
dtrace: failed to compile script ./Slave_open_temp_tables.d: line 14: probe description pid29441::*Slave_open_temp_tables*:return does not match any probes
User defined signal 1
I also have tried with create_myisam_tmp_table but got the same result.
Where did I do wrong?
I don't have MySQL installed to look for you, but the -l option to dtrace will tell you what probes are available, from which you should be able to figure out what you're doing wrong.
$ dtrace -ln 'pid$target::*Slave_open_temp_tables*:return' -p `pgrep -x mysqld`
Or try using a less-specific name such as *open_temp_tables* or *tables* or even * to get a broader set of results... it could be that the function no longer exists in your target process due to changes in the source code of MySQL.
Also, I assume you've checked the output of pgrep -x mysqld to make sure it's a valid input pid?