To avoid an xy problem, I am trying to log/graph/alert on iperf results to several different iperf servers on our network. We have been having problems with inconsistent and slow traffic over our metro ethernet provider. If someone has a better way of going about this, I'm all ears. I am where I am now from random other guides and instructions so this is the best I have come up with.
I am trying to run iperf with telegraf in a bash script and within that script run jq to parse the data to just the average results.
Here is the bash script:
#!/bin/bash
# A simple Bash script
filename='district_list_test.txt'
#echo "Start"
while read DISTRICT
do
#echo $DISTRICT
Address=$(/usr/bin/jq -r '.District_List[]."'"${DISTRICT}"'"' District_list1.json)
#echo $Address
/usr/bin/iperf3 --client ${Address} -t 5 --connect-timeout 250 --json --bidir >> $DISTRICT
download=$(/usr/bin/jq '.end.streams[0] | .sender.bits_per_second' ${DISTRICT})
upload=$(/usr/bin/jq '.end.streams[1] | .sender.bits_per_second' ${DISTRICT})
time=$(/usr/bin/jq '.start.timestamp.timesecs' ${DISTRICT})
echo "{\"District\":$DISTRICT,\"TimeStamp\":$time,\"download\":$download,\"upload\":$upload}" #>> /home/core/test_iperf_output.txt
rm $DISTRICT
done <"$filename"
district_list_test.txt will be a longer list that matches the keys in District_list1.json but for testing its just one device.
district_list_test.txt:
iperf_server
District_list1.json
{
"District_List": [
{
"iperf_server": "10.115.5.5"
}
]
}
The script runs great when I run it manually and it runs in telegraf because I can see the iperfs on the server and for testing I added a file output which it makes in the telegraf config.
telegraf.conf:
[[inputs.exec]]
commands=["bash /home/core/iperf_test4.sh"]
interval = "30s"
timeout = "15s"
data_format = "json"
json_query = "end"
name_override = "iPerf_testing"
[[outputs.file]]
files = ["/home/core/text.txt"]
data_format = "json"
[[outputs.influxdb_v2]]
urls = ["http://localhost:8086"]
data_format = "json"
token = "$INFLUX_TOKEN"
organization = "Test"
bucket = "telegraf"
user_agent = "core"
the file output file, text.txt is blank and I do not see any data in influxdb.
Running the debug for telegraf, all i see is this:
2022-05-04T15:30:41Z I! Starting Telegraf 1.22.0
2022-05-04T15:30:41Z I! Loaded inputs: exec
2022-05-04T15:30:41Z I! Loaded aggregators:
2022-05-04T15:30:41Z I! Loaded processors:
2022-05-04T15:30:41Z I! Loaded outputs: file influxdb_v2
2022-05-04T15:30:41Z I! Tags enabled: host=Iperf-Speedtest
2022-05-04T15:30:41Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"Iperf-Speedtest", Flush Interval:1m0s
2022-05-04T15:30:41Z D! [agent] Initializing plugins
2022-05-04T15:30:41Z D! [agent] Connecting outputs
2022-05-04T15:30:41Z D! [agent] Attempting connection to [outputs.file]
2022-05-04T15:30:41Z D! [agent] Successfully connected to outputs.file
2022-05-04T15:30:41Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-05-04T15:30:41Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-05-04T15:30:41Z D! [agent] Starting service inputs
2022-05-04T15:31:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-05-04T15:31:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-05-04T15:32:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-05-04T15:32:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-05-04T15:33:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-05-04T15:33:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-05-04T15:34:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-05-04T15:34:41Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
I can't figure out what I am missing or how to troubleshoot the issue at this point.
Thanks in advance for any thoughts or suggestions and sorry for the long post.
Related
Good day!
I have built a small IoT device that monitors the conditions inside a specific enclosure using an ESP32 and a couple of sensors. I want to monitor that data by publishing it to the ThingSpeak cloud, then writing it to InfluxDB with Telegraf and finally using the InfluxDB data source in Grafana to visualize it.
So far I have made everything work flawlessly, but with one small exception.
Which is: One of the plugins in my telegraf config fails with the error:
parsing metrics failed: Unable to convert field 'temperature' to type int: strconv.ParseInt: parsing "15.4": invalid syntax
The plugins are [inputs.http]] and [[inputs.http.json_v2]] and what I am doing with them is authenticating against my ThingSpeak API and parsing the json output of my fields. Then in my /etc/telegraf/telegraf.conf under [[inputs.http.json_v2.field]] I have added type = int as otherwise telegraf writes my metrics as Strings in InfluxDB and the only way to visualize them is using either a table or a single stat, because the rest of the flux queries fail with the error unsupported input type for mean aggregate: string. However, when I change to type = float in the config file I get a different error:
unprocessable entity: failure writing points to database: partial write: field type conflict: input field "temperature" on measurement "sensorData" is type float, already exists as type string dropped=1
I have a suspicion that I have misconfigured the parser plugin, however after hours of debugging I couldn't come up with a solution.
Some information that might be of use:
Telegraf version: Telegraf 1.24.2
Influxdb version: InfluxDB v2.4.0
Please see below for my telegraf.conf as well as the error messages.
Any help would be highly appreciated! (:
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 1000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
hostname = ""
omit_hostname = false
[[outputs.influxdb_v2]]
urls = ["http://localhost:8086"]
token = "XXXXXXXX"
organization = "XXXXXXXXX"
bucket = "sensor"
[[inputs.http]]
urls = [
"https://api.thingspeak.com/channels/XXXXX/feeds.json?api_key=XXXXXXXXXX&results=2"
]
name_override = "sensorData"
tagexclude = ["url", "host"]
data_format = "json_v2"
## HTTP method
method = "GET"
[[inputs.http.json_v2]]
[[inputs.http.json_v2.field]]
path = "feeds.1.field1"
rename = "temperature"
type = "int" #Error message 1
#type = "float" #Error message 2
Error when type = "float":
me#myserver:/etc/telegraf$ telegraf -config telegraf.conf --debug
2022-10-16T00:31:43Z I! Starting Telegraf 1.24.2
2022-10-16T00:31:43Z I! Available plugins: 222 inputs, 9 aggregators, 26 processors, 20
parsers, 57 outputs
2022-10-16T00:31:43Z I! Loaded inputs: http
2022-10-16T00:31:43Z I! Loaded aggregators:
2022-10-16T00:31:43Z I! Loaded processors:
2022-10-16T00:31:43Z I! Loaded outputs: influxdb_v2
2022-10-16T00:31:43Z I! Tags enabled: host=myserver
2022-10-16T00:31:43Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"myserver",
Flush Interval:10s
2022-10-16T00:31:43Z D! [agent] Initializing plugins
2022-10-16T00:31:43Z D! [agent] Connecting outputs
2022-10-16T00:31:43Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-10-16T00:31:43Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-10-16T00:31:43Z D! [agent] Starting service inputs
2022-10-16T00:31:53Z E! [outputs.influxdb_v2] Failed to write metric to sensor (will be
dropped: 422 Unprocessable Entity): unprocessable entity: failure writing points to
database: partial write: field type conflict: input field "temperature" on measurement
"sensorData" is type float, already exists as type string dropped=1
2022-10-16T00:31:53Z D! [outputs.influxdb_v2] Wrote batch of 1 metrics in 8.9558ms
2022-10-16T00:31:53Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
Error when type = "int"
me#myserver:/etc/telegraf$ telegraf -config telegraf.conf --debug
2022-10-16T00:37:05Z I! Starting Telegraf 1.24.2
2022-10-16T00:37:05Z I! Available plugins: 222 inputs, 9 aggregators, 26 processors, 20
parsers, 57 outputs
2022-10-16T00:37:05Z I! Loaded inputs: http
2022-10-16T00:37:05Z I! Loaded aggregators:
2022-10-16T00:37:05Z I! Loaded processors:
2022-10-16T00:37:05Z I! Loaded outputs: influxdb_v2
2022-10-16T00:37:05Z I! Tags enabled: host=myserver
2022-10-16T00:37:05Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"myserver",
Flush Interval:10s
2022-10-16T00:37:05Z D! [agent] Initializing plugins
2022-10-16T00:37:05Z D! [agent] Connecting outputs
2022-10-16T00:37:05Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2022-10-16T00:37:05Z D! [agent] Successfully connected to outputs.influxdb_v2
2022-10-16T00:37:05Z D! [agent] Starting service inputs
2022-10-16T00:37:10Z E! [inputs.http] Error in plugin:
[url=https://api.thingspeak.com/channels/XXXXXX/feeds.json?
api_key=XXXXXXX&results=2]: parsing metrics failed: Unable to convert field
'temperature' to type int: strconv.ParseInt: parsing "15.3": invalid syntax
Fixed it by leaving type = float under [[inputs.http.json_v2.field]] in telegraf.conf and creating a NEW bucket with a new API key in Influx.
The issue was that the bucket sensor that I had previously defined in my telegraf.conf already had the field temperature created in my influx database from previous tries with its type set as last (aka: String) which could not be overwritten with the new type mean (aka: float).
As soon as I deleted all pre existing buckets everything started working as expected.
InfluxDB dashboard
Any help is appreciated, Please let me know were I am going wrong!
I am getting errors shown in the following image, I am running Loki and Grafana as 2 different AWS ECS-FARGATE tasks but my Liki container is failing and keep restarting itself:
My loki-config.yaml:
auth_enabled: true
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed
max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h
chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
max_transfer_retries: 0 # Chunk transfers disabled
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: index_
period: 24h
storage_config:
aws:
s3: s3://XXXXX:YYYY#eu-west-1/logs-loki-test
boltdb_shipper:
active_index_directory: /loki/boltdb-shipper-active
cache_location: /loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: s3
compactor:
working_directory: /loki/boltdb-shipper-compactor
shared_store: aws
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
ruler:
storage:
type: local
local:
directory: /loki/rules
rule_path: /loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true
In the compactor block, line shared_store replace aws with s3 and try out
We have a magento website
Our website some times it showing below error like
There has been an error processing your request
Exception printing is disabled by default for security reasons.
Error log record number: 855613014442
Based on our logs, it is showing that Mysql is going down as shown below
2019-06-24T04:44:49.542168Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.26' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2019-06-24T04:44:50.594943Z 0 [Note] InnoDB: Buffer pool(s) load completed at 190624 4:44:50
2019-06-24T04:45:11.103402Z 0 [Note] Giving 0 client threads a chance to die gracefully
2019-06-24T04:45:11.103429Z 0 [Note] Shutting down slave threads
2019-06-24T04:45:11.103438Z 0 [Note] Forcefully disconnecting 0 remaining clients
2019-06-24T04:45:11.103444Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2019-06-24T04:45:11.103484Z 0 [Note] Binlog end
We have increased innodb_buffer_pool_size but still i am facing same issue.
I have executed below commands in my server..check it these outputs
1)free -m
Output:
total used free shared buff/cache available
Mem: 7819 1430 4688 81 1701 6009
Swap: 0 0 0
2)dmesg | tail -30
Output:
[ 6.222373] [TTM] Initializing pool allocator
[ 6.241079] [TTM] Initializing DMA pool allocator
[ 6.255768] [drm] fb mappable at 0xF0000000
[ 6.259225] [drm] vram aper at 0xF0000000
[ 6.262574] [drm] size 33554432
[ 6.265475] [drm] fb depth is 24
[ 6.268473] [drm] pitch is 3072
[ 6.289079] fbcon: cirrusdrmfb (fb0) is primary device
[ 6.346169] Console: switching to colour frame buffer device 128x48
[ 6.347151] loop: module loaded
[ 6.357709] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device
[ 6.364646] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0
[ 6.722341] input: PC Speaker as /devices/platform/pcspkr/input/input4
[ 6.788110] EXT4-fs (loop0): mounting ext3 file system using the ext4 subsystem
[ 6.802845] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[ 6.841332] cryptd: max_cpu_qlen set to 1000
[ 6.871200] AVX2 version of gcm_enc/dec engaged.
[ 6.873349] AES CTR mode by8 optimization enabled
[ 6.936609] EXT4-fs (loop0): mounting ext3 file system using the ext4 subsystem
[ 6.949717] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[ 6.964446] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[ 6.984659] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni)
[ 7.084148] intel_rapl: Found RAPL domain package
[ 7.086591] intel_rapl: Found RAPL domain dram
[ 7.088788] intel_rapl: DRAM domain energy unit 15300pj
[ 7.102115] EDAC sbridge: Seeking for: PCI ID 8086:6fa0
[ 7.102119] EDAC sbridge: Ver: 1.1.2
[ 7.175339] ppdev: user-space parallel port driver
[ 10.728980] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 10.772307] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
3)ps auxw | grep mysql
Output:
mysql 5056 2.9 10.8 7009056 871240 ? Sl 12:29 0:12 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
root 5538 0.0 0.0 112708 976 pts/0 S+ 12:36 0:00 grep --color=auto mysql
Can anyone has idea how to resolve this issue.
Thanks
I have a cuba application which I want to use sidekiq with.
This is how I setup the config.ru:
require './app'
require 'sidekiq'
require 'sidekiq/web'
environment = ENV['RACK_ENV'] || "development"
config_vars = YAML.load_file("./config.yml")[environment]
Sidekiq.configure_client do |config|
config.redis = { :url => config_vars["redis_uri"] }
end
Sidekiq.configure_server do |config|
config.redis = { url: config_vars["redis_uri"] }
config.average_scheduled_poll_interval = 5
end
# run Cuba
run Rack::URLMap.new('/' => Cuba, '/sidekiq' => Sidekiq::Web)
I started sidekiq using systemd. This is the systemd script which I adapted from the sidekiq.service on the sidekiq site.:
#
# systemd unit file for CentOS 7, Ubuntu 15.04
#
# Customize this file based on your bundler location, app directory, etc.
# Put this in /usr/lib/systemd/system (CentOS) or /lib/systemd/system (Ubuntu).
# Run:
# - systemctl enable sidekiq
# - systemctl {start,stop,restart} sidekiq
#
# This file corresponds to a single Sidekiq process. Add multiple copies
# to run multiple processes (sidekiq-1, sidekiq-2, etc).
#
# See Inspeqtor's Systemd wiki page for more detail about Systemd:
# https://github.com/mperham/inspeqtor/wiki/Systemd
#
[Unit]
Description=sidekiq
# start us only once the network and logging subsystems are available,
# consider adding redis-server.service if Redis is local and systemd-managed.
After=syslog.target network.target
# See these pages for lots of options:
# http://0pointer.de/public/systemd-man/systemd.service.html
# http://0pointer.de/public/systemd-man/systemd.exec.html
[Service]
Type=simple
Environment=RACK_ENV=development
WorkingDirectory=/media/temp/bandmanage/repos/fall_prediction_verification
# If you use rbenv:
#ExecStart=/bin/bash -lc 'pwd && bundle exec sidekiq -e production'
ExecStart=/home/froy001/.rvm/wrappers/fall_prediction/bundle exec "sidekiq -r app.rb -L log/sidekiq.log -e development"
# If you use the system's ruby:
#ExecStart=/usr/local/bin/bundle exec sidekiq -e production
User=root
Group=root
UMask=0002
# if we crash, restart
RestartSec=1
Restart=on-failure
# output goes to /var/log/syslog
StandardOutput=syslog
StandardError=syslog
# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq
[Install]
WantedBy=multi-user.target
The code calling the worker is :
raw_msg = JSON.parse(req.body.read, {:symbolize_names => true})
if raw_msg
ts = raw_msg[:ts]
waiting_period = (1000*60*3) # wait 3 min before checking
perform_at_time = Time.at((ts + waiting_period)/1000).utc
FallVerificationWorker.perform_at((0.5).minute.from_now, raw_msg)
my_res = { result: "success", status: 200}.to_json
res.status = 200
res.write my_res
else
my_res = { result: "not found", status: 404}.to_json
res.status = 404
res.write my_res
end
I am only using the default q.
My problem is that the job is not being processed at all.
After you run systemctl enable sidekiq so that it starts at boot and systemctl start sidekiq so that it starts immediately, then you should have some logs to review which will provide some detail about any failure to start:
sudo journalctl -u sidekiq
Review the logs, review the systemd docs and adjust your unit file as needed. You can find all the installed systemd documentation with apropos systemd. Some of the most useful man pages to review are systemd.service,systemd.exec and systemd.unit
I have configured filebeat as it is descripted at elastic.co
The Problem is that when I add a new log-file the data is not uploaded to logstash. What can be the problem?
I already tried different config ways but it didn't work at all.
################### Filebeat Configuration Example #########################
############################# Filebeat ######################################
filebeat:
# List of prospectors to fetch data.
prospectors:
-
paths:
- /Users/apps/*.log
input_type: log
###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features
############################# Output ##########################################
output:
elasticsearch:
hosts: ["localhost:9200"]
worker: 1
index: "filebeat"
template:
path: "filebeat.template.json"
### Logstash as output
logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
index: filebeat
############################# Shipper #########################################
############################# Logging #########################################
# There are three options for the log ouput: syslog, file, stderr.
# Under Windos systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:
files:
rotateeverybytes: 10485760 # = 10MB
Config in logstash.conf:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => true
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
document_id => "%{fingerprint}"
}
}
You are both sending Elasticsearch and logstash. You need to remove elasticsearch part if you want to send it to logstash. Taken from https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html:
If you want to use Logstash to perform additional processing on the
data collected by Filebeat, you need to configure Filebeat to use
Logstash.
To do this, you edit the Filebeat configuration file to disable the
Elasticsearch output by commenting it out and enable the Logstash
output by uncommenting the logstash section
Can you check the worker count in filebeat.yml as below?
### Logstash as output
logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
# Number of workers per Logstash host.
worker: 1
You should add worker count in the logstash part