I have a container running as non-privileged mode. I'd like to update file atime via python code for some reason but found I could not do that due to permission issue, even though I can write to that file.
I tried to add linux capabilities to the container, but even with SYS_AMDIN, it still does not work.
Anyone happens to know what capabilities to add or what I missed there?
thank you!
bash-5.1$ id
uid=1000(contest) gid=1000(contest) groups=1000(contest)
bash-5.1$ ls -l
total 250
-rwxrwxrwx 1 root contest 0 Oct 27 07:16 anotherfile
-rwxrwxrwx 1 root contest 254823 Oct 27 07:37 outfile
-rwxrwxrwx 1 root contest 0 Oct 24 03:52 test
-rwxrwxrwx 1 root contest 364 Oct 27 07:16 test.py
-rwxrwxrwx 1 root contest 18 Oct 24 05:25 testfile
bash-5.1$ python3 test.py
1666854988.190472
1666851388.190472
Traceback (most recent call last):
File "/mnt/azurefile/test.py", line 19, in <module>
os.utime(myfile, (atime - 3600.0, mtime))
PermissionError: [Errno 1] Operation not permitted
bash-5.1$ capsh --print
Current: =
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_sys_admin,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =
Current IAB: !cap_dac_read_search,!cap_linux_immutable,!cap_net_broadcast,!cap_net_admin,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_lease,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read
Securebits: 00/0x0/1'b0 (no-new-privs=0)
secure-noroot: no (unlocked)
secure-no-suid-fixup: no (unlocked)
secure-keep-caps: no (unlocked)
secure-no-ambient-raise: no (unlocked)
uid=1000(contest) euid=1000(contest)
gid=1000(contest)
groups=1000(contest)
Guessed mode: HYBRID (4)
my python code to update atime:
from datetime import datetime
import os
import time
myfile = "anotherfile"
current_time = time.time()
"""
Set the access time of a given filename to the given atime.
atime must be a datetime object.
"""
stat = os.stat(myfile)
mtime = stat.st_mtime
atime = stat.st_atime
print(mtime)
mtime = mtime - 3600.0
print(mtime)
os.utime(myfile, (atime - 3600.0, mtime))
pod yaml
---
kind: Pod
apiVersion: v1
metadata:
name: nginx-azurefile
spec:
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
nodeSelector:
"kubernetes.io/os": linux
containers:
- image: acheng.azurecr.io/capsh
name: nginx-azurefile
securityContext:
capabilities:
add: ["CHOWN","SYS_ADMIN","SYS_RESOURCES"]
command:
- "/bin/bash"
- "-c"
- set -euo pipefail; while true; do echo $(date) >> /mnt/azurefile/outfile; sleep 10; done
volumeMounts:
- name: persistent-storage
mountPath: "/mnt/azurefile"
imagePullSecrets:
- name: acr-secret
volumes:
- name: persistent-storage
persistentVolumeClaim:
claimName: pvc-azurefile
tried to add SYS_ADMIN capabilities but didn't work.
if container runs in privileged mode, the code is able to update file access time as expected
answering my own question here.
After searching around, I found kubernetes does not support capabilities for non-root users. the capabilities added in container spec is for root user only. won't take effect for non-root users.
see this github issue for details: https://github.com/kubernetes/kubernetes/issues/56374
a workaround is to add cap directly to the executable file using setcap command (from libcap).
and the capability needed is CAP_FOWNER
I just finished testing the entire thing on my virtual machine environment and now I am trying to launch it on the dedicated server. And now I ran into a completely new issue. First I confirmed that I have both context Broker and centos running (on 1026 and 5050 respectively):
[root#centos conf]# netstat -ntlpd
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1026 0.0.0.0:* LISTEN 1321/contextBroker
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1282/mongod
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1791/mysqld
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1260/sshd
tcp 0 0 :::1026 :::* LISTEN 1321/contextBroker
tcp 0 0 :::8081 :::* LISTEN 2481/java
tcp 0 0 :::22 :::* LISTEN 1260/sshd
tcp 0 0 :::5050 :::* LISTEN 2481/java
[root#centos conf]# service cygnus status
Cygnus 1 status...
cygnus-flume-ng (pid 2481) is running...
Then I confirmed that I have data on contextBroker because this command gave me an appropriate response:
(curl localhost:1026/v1/queryContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
]
}
EOF
Following the workaround to an issue with root user and logging I fixed the log4j.properties and changed the follwing:
flume.log.dir=/var/log/cygnus
I then started cygnus and got the following log:
Starting an ordered shutdown of Cygnus
Stopping sources
Stopping http-source (lyfecycle state=START)
All the channels are empty
Stopping channels
Stopping mysql-channel (lyfecycle state=START)
Stopping sinks
Stopping mysql-sink (lyfecycle state=START)
Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplic$
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.11.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
17 Dec 2015 13:35:37,684 INFO [main] (com.telefonica.iot.cygnus.nodes.CygnusApplication.main:235) - Starting a Jetty server listening on port 8081 (Management Interface)
17 Dec 2015 13:35:37,700 INFO [main] (org.mortbay.log.Slf4jLog.info:67) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
17 Dec 2015 13:35:37,700 INFO [main] (com.telefonica.iot.cygnus.nodes.CygnusApplication.main:238) - Starting Cygnus application
17 Dec 2015 13:35:37,700 INFO [Thread-1] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,713 INFO [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61) - Configuration provider starting
17 Dec 2015 13:35:37,715 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/cygnus/conf/agent_1.conf
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,755 INFO [Thread-1] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:8081
17 Dec 2015 13:35:37,764 WARN [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid:319) - Agent configuration for 'cygunsagent' does not contain any channels. Marking it as invalid.
17 Dec 2015 13:35:37,765 WARN [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:127) - Agent configuration invalid for agent 'cygunsagent'. It will be removed.
17 Dec 2015 13:35:37,766 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [cygnusagent]
17 Dec 2015 13:35:37,766 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150) - Creating channels
17 Dec 2015 13:35:37,778 INFO [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40) - Creating instance of channel mysql-channel type memory
17 Dec 2015 13:35:37,782 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel mysql-channel
17 Dec 2015 13:35:37,783 INFO [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39) - Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
17 Dec 2015 13:35:37,791 INFO [conf-file-poller-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.<init>:75) - Cygnus version (0.11.0.2a9c87fb7fd6156225e2eed7fbc9792f1d9c5dfe)
17 Dec 2015 13:35:37,807 INFO [conf-file-poller-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.configure:141) - Startup completed
17 Dec 2015 13:35:37,826 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: mysql-sink, type: com.telefonica.iot.cygnus.sinks.OrionMySQLSink
17 Dec 2015 13:35:37,839 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel mysql-channel connected to [http-source, mysql-sink]
17 Dec 2015 13:35:37,843 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTT$
17 Dec 2015 13:35:37,844 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel mysql-channel
17 Dec 2015 13:35:37,910 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: CHANNEL, name: mysql-channel, registered successfully.
17 Dec 2015 13:35:37,910 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: CHANNEL, name: mysql-channel started
17 Dec 2015 13:35:37,911 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink mysql-sink
17 Dec 2015 13:35:37,913 INFO [lifecycleSupervisor-1-1] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.start:153) - [mysql-sink] Startup completed
17 Dec 2015 13:35:37,915 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184) - Starting Source http-source
17 Dec 2015 13:35:37,916 INFO [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:92) - Grouping rules read:
17 Dec 2015 13:35:37,916 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HT$
17 Dec 2015 13:35:37,917 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:105) - Stopping Source http-source
17 Dec 2015 13:35:37,920 ERROR [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.parseGroupingRules:165) - Error while parsing the Json-based grouping rules file. Details=null
17 Dec 2015 13:35:37,921 WARN [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:98) - Grouping rules syntax has errors
17 Dec 2015 13:35:37,948 INFO [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,973 INFO [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,974 INFO [lifecycleSupervisor-1-2] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
17 Dec 2015 13:35:37,974 INFO [lifecycleSupervisor-1-2] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: SOURCE, name: http-source started
17 Dec 2015 13:35:37,974 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state$
17 Dec 2015 13:35:37,974 INFO [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67) - Stopped SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,975 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:139) - Component type: SOURCE, name: http-source stopped
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:145) - Shutdown Metric for type: SOURCE, name: http-source. source.start.time == 1450355737974
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:151) - Shutdown Metric for type: SOURCE, name: http-source. source.stop.time == 1450355737975
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append-batch.accepted == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append-batch.received == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append.accepted == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append.received == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.events.accepted == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.events.received == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.open-connection.count == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.source.http.HTTPSource.stop:172) - Http source http-source stopped. Metrics: SOURCE:http-source{src.events.accepted=0, src.open-connection.count=0, src.append.receiv$
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:115) - Stopping Sink mysql-sink
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor#76da521f counterGroup:{ name:nul$
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:125) - Stopping Channel mysql-channel
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: org.apache.flume.channel.MemoryChannel{name: mysql-channel}
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:139) - Component type: CHANNEL, name: mysql-channel stopped
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:145) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.start.time == 1450355737910
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:151) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.stop.time == 1450355737987
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.capacity == 1000
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.current.size == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.put.attempt == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.put.success == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.take.attempt == 1
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.take.success == 0
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTT$
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel mysql-channel
17 Dec 2015 13:35:37,989 INFO [lifecycleSupervisor-1-3] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: CHANNEL, name: mysql-channel started
17 Dec 2015 13:35:37,992 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink mysql-sink
17 Dec 2015 13:35:37,993 INFO [lifecycleSupervisor-1-8] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.start:153) - [mysql-sink] Startup completed
17 Dec 2015 13:35:37,993 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184) - Starting Source http-source
17 Dec 2015 13:35:37,993 INFO [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:92) - Grouping rules read:
17 Dec 2015 13:35:37,994 ERROR [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.parseGroupingRules:165) - Error while parsing the Json-based grouping rules file. Details=null
17 Dec 2015 13:35:37,994 WARN [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:98) - Grouping rules syntax has errors
17 Dec 2015 13:35:37,994 INFO [lifecycleSupervisor-1-4] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,996 INFO [lifecycleSupervisor-1-4] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,996 INFO [lifecycleSupervisor-1-4] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: SOURCE, name: http-source started
Then I tried to subscribe the previously mentioned data to cygnus:
(curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
],
"attributes": [
"pressure"
"temperature"
],
"reference": "http://localhost:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"pressure",
"temperature"
]
}
],
"throttling": "PT1S"
}
EOF
Even after I updated the information on context Broker thinking it would trigger an event:
(curl localhost:1026/v1/updateContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"contextElements": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1",
"attributes": [
{
"name": "temperature",
"type": "float",
"value": "333"
},
{
"name": "pressure",
"type": "integer",
"value": "555"
}
]
}
],
"updateAction": "APPEND"
}
EOF
But the cygnus log remainged exactly the same and its like nothing even got through to it. Which is odd considering my agent_1.conf:
# Copyright 2014 Telefónica Investigación y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es
#=============================================
# To be put in APACHE_FLUME_HOME/conf/agent.conf
#
# General configuration template explaining how to setup a sink of each of the available types (HDFS, CKAN, MySQL).
#=============================================
# The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
# ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
# Regarding sinks, you can use multiple types at the same time; the only requirement is to provide a channel for each
# one of them (this example shows how to configure 3 sink types at the same time). Even, you can define more than one
# sink of the same type and sharing the channel in order to improve the performance (this is like having
# multi-threading).
cygnusagent.sources = http-source
cygnusagent.sinks = mysql-sink
cygnusagent.channels = mysql-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = mysql-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = Trace_Data
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = Sensor
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
# ============================================
# OrionMySQLSink configuration
# channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel
# sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.OrionMySQLSink
# true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.mysql-sink.enable_grouping = false
# the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = 127.0.0.1
# the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306
# a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = root
# password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = klasika
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = column
# select the table type from table-by-destination and table-by-service-path
cygnusagent.sinks.mysql-sink.table_type = table-by-destination
# number of notifications to be included within a processing batch
cygnusagent.sinks.mysql-sink.batch_size = 1
# timeout for batch accumulation
cygunsagent.sinks.mysql-sink.batch_timeout = 30
#=============================================
# mysql-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory
# capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100
#============================================
It has 5050 and notify as the reference address. I double checked the cygnus_instance_1.conf as well and it is pointing at agent_1.conf
#####
#
# Configuration file for apache-flume
#
#####
# Copyright 2014 Telefonica Investigación y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es
# Who to run cygnus as. Note that you may need to use root if you want
# to run cygnus in a privileged port (<1024)
CYGNUS_USER=cygnus
# Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf
# Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
# Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
# naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnusagent
# Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log
# Administration port. Must be unique per instance
ADMIN_PORT=8081
# Polling interval (seconds) for the configuration reloading
POLLING_INTERVAL=30
This is the content of my config folder in /usr/cygnus/conf:
[root#centos conf]# ls
agent_1.conf cygnus_instance_1.conf flume-env.sh grouping_rules.conf krb5.conf krb5_login.conf log4j.properties.template
agent.conf.template cygnus_instance.conf.template flume-env.sh.template grouping_rules.conf.template krb5.conf.template log4j.properties README.md
I noticed that there is an exacted mirror of this in /etc/cygnus/conf but I didn't touch anything because the installation only intructs me to use the /usr/ folder.
Here is my Mysql create statement. In this table I am expecting to receive the context broker data but I get nothing of course since the log didnt register anything.
CREATE TABLE sensor_room1_room (
sensorID INT NOT NULL AUTO_INCREMENT,
recvTime mediumtext,
fiwareservicepath text,
entityId text,
entityType text,
pressure text,
pressure_md text,
temperature text,
temperature_md text,
PRIMARY KEY (sensorID));
Edit 1:
Here is the listener
[root#centos conf]# nc -l 5050
But when I tried subscribing or updating context, nothing was received on the listener side. I am not taking into account the client side of nc: nc 127.0.0.1 5050 because it successfully sends everything i type (even gibberish).
I also tried the test: /usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_1.conf -n cygnusagent -Dflume.root.logger=DEBUG,console. I tried both 5050 and 8081 ports to subscribe to and then update context but nothing is read on the console.
Since I seriously have no idea why that subscription didn't work, but thanks to #fgalan I did manage to read the logs so I am posting the subscription that did trigger the event to cygnus:
(curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
],
"attributes": [
"temperature"
],
"reference": "http://localhost:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"pressure"
]
}
],
"throttling": "PT5S"
}
EOF
Thank you #fgalan one more time!
I want to subscribe Orion to send notifications to Cygnus. Then cygnus will save all data in mysql database. I use this script to subscribe the speed attribute of car1.
(curl 130.206.118.44:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Fiware-Service: vehicles' --header 'Fiware-ServicePath: /4wheels' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "car",
"isPattern": "false",
"id": "car1"
}
],
"attributes": [
"speed",
"oil_level"
],
"reference": "http://192.168.1.49:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"speed"
]
}
],
"throttling": "PT1S"
}
EOF
But when I update the speed attribute of car 1, cygnus doesn't update the database.
Databases available:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)
Some information about my cygnus service and my cygnus configuration (systemctl status cygnus):
cygnus.service - SYSV: cygnus
Loaded: loaded (/etc/rc.d/init.d/cygnus)
Active: active (exited) since Wed 2015-10-21 17:54:07 UTC; 8min ago
Process: 31566 ExecStop=/etc/rc.d/init.d/cygnus stop (code=exited, status=0/SUCCESS)
Process: 31588 ExecStart=/etc/rc.d/init.d/cygnus start (code=exited, status=0/SUCCESS)
Oct 21 17:54:05 cygnus systemd[1]: Starting SYSV: cygnus...
Oct 21 17:54:05 cygnus su[31593]: (to cygnus) root on none
Oct 21 17:54:07 cygnus cygnus[31588]: Starting Cygnus mysql... [ OK ]
Oct 21 17:54:07 cygnus systemd[1]: Started SYSV: cygnus.
agent_mysql.conf:
# main configuration
cygnusagent.sources = http-source
cygnusagent.sinks = mysql-sink
cygnusagent.channels = mysql-channel
# source configuration
cygnusagent.sources.http-source.channels = mysql-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# url target
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.default_service = def_serv
cygnusagent.sources.http-source.handler.default_service_path = def_servpath
cygnusagent.sources.http-source.handler.events_ttl = 10
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
#Orion MysqlSink Configuration
cygnusagent.sinks.mysql-sink.channel = mysql-channel
cygnusagent.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.OrionMySQLSink
cygnusagent.sinks.mysql-sink.enable_grouping = false
# mysqldb ip
cygnusagent.sinks.mysql-sink.mysql_host = 127.0.0.1
# mysqldb port
cygnusagent.sinks.mysql-sink.mysql_port = 3306
cygnusagent.sinks.mysql-sink.mysql_username = root
cygnusagent.sinks.mysql-sink.mysql_password = 12345
cygnusagent.sinks.mysql-sink.attr_persistence = column
cygnusagent.sinks.mysql-sink.table_type = table-by-destination
# configuracao do canal mysql
cygnusagent.channels.mysql-channel.type = memory
cygnusagent.channels.mysql-channel.capacity = 1000
cygnusagent.channels.mysql-channel.transactionCapacity = 100
After read this question, I changed my agent_mysql.conf in this line:
cygnusagent.sinks.mysql-sink.attr_persistence = column to cygnusagent.sinks.mysql-sink.attr_persistence = row and restarted the service. Then I updated orion entity and I queried database and nothing happened.
Cygnus log file: http://pastebin.com/B2FNKcVf
Note: My JAVA_HOME is set.
As you can see in the logs you posted, there is a problem with the Cygnus log file:
java.io.FileNotFoundException: ./logs/cygnus.log (No such file or directory)
After that, Cygnus stops. You must check your configuration regarding log4j, everything is at /usr/cygnus/conf/log4j.properties (it should exists, it is created by the RPM... if not existing -because you italled from sources instead of the RPM-, it must be created from the available template). In addition, can you post your instance configuration file? Anyway, which version are you running?
EDIT 1
Recently, we have found another user dealing with the same error, and the problem was the content of the /usr/cygnus/conf/log4j.properties file was:
flume.root.logger=INFO,LOGFILE
flume.log.dir=./log
flume.log.file=flume.log
Instead of what the template contains:
flume.root.logger=INFO,LOGFILE
flume.log.dir=/var/log/cygnus/
flume.log.file=flume.log
Once changed it worked because the RPM creates /var/log/cygnus but not ./log.
I just installed freeradius-server-2.2.0 from tarball.
I want to connect freeradius using mysql server.
But when i running radius on foreground with radiusd -X and I got error message like bellow :
Fri Mar 8 13:44:46 2013 : Debug: group_membership_query = "SELECT groupname FROM radusergroup WHERE username = '%{SQL-User-Name}' ORDER BY priority"
Fri Mar 8 13:44:46 2013 : Debug: connect_failure_retry_delay = 60
Fri Mar 8 13:44:46 2013 : Debug: simul_count_query = ""
Fri Mar 8 13:44:46 2013 : Debug: simul_verify_query = "SELECT radacctid, acctsessionid, username, nasipaddress, nasportid, framedipaddress, callingstationid, framedprotocol FROM radacct WHERE username = '%{SQL-User-Name}' AND acctstoptime IS NULL"
Fri Mar 8 13:44:46 2013 : Debug: postauth_query = "INSERT INTO radpostauth (username, pass, reply, authdate) VALUES ( '%{User-Name}', '%{%{User-Password}:-%{Chap-Password}}', '%{reply:Packet-Type}', '%S')"
Fri Mar 8 13:44:46 2013 : Debug: safe-characters = "#abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.-_: /"
Fri Mar 8 13:44:46 2013 : Debug: }
Fri Mar 8 13:44:46 2013 : Error: Could not link driver rlm_sql_mysql: rlm_sql_mysql.so: cannot open shared object file: No such file or directory
Fri Mar 8 13:44:46 2013 : Error: Make sure it (and all its dependent libraries!) are in the search path of your system's ld.
Fri Mar 8 13:44:46 2013 : Error: /usr/local/etc/raddb/sql.conf[22]: Instantiation failed for module "sql"
Fri Mar 8 13:44:46 2013 : Error: /usr/local/etc/raddb/sites-enabled/default[177]: Failed to find "sql" in the "modules" section.
Fri Mar 8 13:44:46 2013 : Error: /usr/local/etc/raddb/sites-enabled/default[69]: Errors parsing authorize section.
It's also happen on my freeradius ldap?
[error LDAP]
/usr/local/etc/raddb/modules/ldap[29]: Failed to link to module 'rlm_ldap': rlm_ldap.so: cannot open shared object file: No such file or directory
/usr/local/etc/raddb/sites-enabled/default[305]: Failed to find "ldap" in the "modules" section.
/usr/local/etc/raddb/sites-enabled/default[305]: Failed to parse "ldap"
How i can solve this issue ?
i've installed libmysqlclient-dev on my ubuntu.
I'm running on Ubuntu 12.04.
Help me to solve this.
Thanks
You did re-run configure right, after installing the development headers?
When building from source you will need to do a:
./configure <options> && make && make install
To actually install the module.