Core OWASP ModSecurity - Allowing JSON - json

I've had ModSecurity and the Core OWASP Rule Set ver.2.2.5 installed for some months now, but a JSON endpoint on the site has recently stopped responding, and the Apache log gets the following:
[Tue Jul 21 10:41:12 2015] [error] [client 194.54.11.146] ModSecurity:
Warning. Match of "streq %{SESSION.IP_HASH}" against "TX:ip_hash"
required. [file
"/etc/modsecurity/activated_rules/modsecurity_crs_16_session_hijacking.conf"]
[line "35"] [id "981059"] [msg "Warning - Sticky SessionID Data
Changed - IP Address Mismatch."] [hostname "************"] [uri
"/api/campaigns/d3c735cb-0773-11e4-98bd-02f651afdab5"] [unique_id
"Va4hyKwfKiYAAAYSLigAAAAJ"]
[Tue Jul 21 10:41:12 2015] [error] [client 194.54.11.146] ModSecurity:
Warning. Match of "streq %{SESSION.UA_HASH}" against "TX:ua_hash"
required. [file
"/etc/modsecurity/activated_rules/modsecurity_crs_16_session_hijacking.conf"]
[line "36"] [id "981060"] [msg "Warning - Sticky SessionID Data
Changed - User-Agent Mismatch."] [hostname "************"] [uri
"/api/campaigns/d3c735cb-0773-11e4-98bd-02f651afdab5"] [unique_id
"Va4hyKwfKiYAAAYSLigAAAAJ"]
[Tue Jul 21 10:41:12 2015] [error] [client 194.54.11.146] ModSecurity:
Warning. Operator EQ matched 2 at TX:sticky_session_anomaly. [file
"/etc/modsecurity/activated_rules/modsecurity_crs_16_session_hijacking.conf"]
[line "37"] [id "981061"] [msg "Possible Session Hijacking - IP
Address and User-Agent Mismatch."] [hostname "************"] [uri
"/api/campaigns/d3c735cb-0773-11e4-98bd-02f651afdab5"] [unique_id
"Va4hyKwfKiYAAAYSLigAAAAJ"]
[Tue Jul 21 10:41:12 2015] [error] [client 194.54.11.146] ModSecurity:
Warning. Match of "rx ^%{tx.allowed_request_content_type}$" against
"TX:0" required. [file
"/etc/modsecurity/activated_rules/modsecurity_crs_30_http_policy.conf"]
[line "64"] [id "960010"] [msg "Request content type is not allowed by
policy"] [data "application/json"] [severity "WARNING"] [tag
"POLICY/ENCODING_NOT_ALLOWED"] [tag "WASCTC/WASC-20"] [tag
"OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/EE2"] [tag "PCI/12.1"]
[hostname "************"] [uri
"/api/campaigns/d3c735cb-0773-11e4-98bd-02f651afdab5"] [unique_id
"Va4hyKwfKiYAAAYSLigAAAAJ"]
I'm new to mod_security and the OWASP rules (I basically followed the guide here) but as I understand, rules are scored, and if a request passes a threshold, it's nuked. I assume this is what I'm seeing here.
The final one is the one that concerns me - "application/json" should certainly be allowed. From looking at /etc/modsecurity/modsecurity_crs_10_setup.conf, I see:
setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf'
My question is:
1. Can I just add application/json in here to make the error go away?
2. Is that the correct way to do it?

Yes you can so it reads like this:
setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json'
Yes that is the correct way of doing this.

Related

MediaWiki treats CLTF text as RCE

Help. I have a weird problem on a MediaWiki 1.37.2 install. We (me & the musician) are trying to get a wiki going for Clan Lord Tune Format music (like midi but with a–g, "." for flat, "[…]" for chords so "[DFA.]" is a possible chord). However, when we try to edit or create a page with CLTF in it, sometimes MediaWiki throws an exception thinking it’s an RCE attack:
ModSecurity: Warning. Pattern match "(?:\$(?:\((?:\(.\)|.)\)|\{.\})|[<>]\(.\)) at ARGS:text at ARGS:text. [file "…/apache2/template/etc/mod_sec3_CRS/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "366"] [id "932130"] [msg "Remote Command Execution: Unix Shell Expression Found"] [data "Matched Data:
and
[severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [tag "PCI/6.5.2"]
at the end.
How do I disable or modify the security module to not throw the exception on this text and not flag it as an RCE attack so it can be saved in the wiki? Is there a setting or format of the text that will allow the CLTF formatted music?
Thanks in advance for any answers that fix this.
*EDIT: I found this: https://www.mediawiki.org/wiki/ModSecurity and it says to turn off the security module with an .htaccess file. Does this go in the MediaWiki directory or the web root?
**EDIT 2: we also found this: https://anto.online/guides/how-to-disable-modsecurity-rules-that-cause-403-errors/ and are not sure which one is the for the OWASP_CRS are the codes: "capec/1000/152/248/88" ?
full log (minus personally identifiable info removed with ellipsis: … ):
[Wed May 18 08:27:38.649383 2022] [:error] [pid 3490:tid 3408850568960] [client …] [client …] ModSecurity: Warning. Pattern match "(?:\\$(?:\\((?:\\(.\\)|.)\\)|\\{.\\})|[<>]\\(.\\))" at ARGS:wpTextbox1. [file "…/apache2/template/etc/mod_sec3_CRS/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "366"] [id "932130"] [msg "Remote Command Execution: Unix Shell Expression Found"] [data "Matched Data: >((#150[e]4ppp[b]2pp[e]2pp[e]2p[d]3ppp[e]2pp|1[e]2pp[e]2p!ppp)2)4([=egb]8[e]4ppp[b]2p[=egb]8p[e]2pp[e]2p[=df#b]8[d]3ppp[e]2p[=egb]8p|1[e]2pp[e]2p!ppp)2([=cea]8[a]4ppp[e]2p[=cea]8p[a]2pp[a]2p[=df#b]8[d]3ppp[f#]2p|1[=cea]8p[e]2pp[e]2p![=egb]8pppp)2(([=egb]8[e]4ppp[b]2p[=egb]8p[e]2pp[e]2p[=df#b]8[d]3ppp[e]2p|1[=egb]8p[e]2pp[e]2p![=egb]pppp)2)2([=egb]8[e]4ppp[b]2p[=egb]8p[e]2pp[e]2p[=df#b]8[d]3ppp[e]2p[=egb]8p|1[e]2pp[e]2p!ppp)2([=cea]8[a]4ppp[e]2p[=cea]8p[a]2pp[a]2p[=df#b]8[d]3ppp[f#]2p|1[=cea..."] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.2"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [tag "PCI/6.5.2" [hostname "…"] [uri "…bardsfield/index.php"] [unique_id "YoUQak6BVgxiylKa6BNQVgAAAAE"], referer: https://…/bardsfield/index.php?title=…&action=edit
Okay. So, We got all the modSec rule IDs exempted that we needed (there we 6 security violations) to by our hosting ISP’s server admin. So ,this case is closed.
Try this exclusion rule (put it into REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf):
SecRule REQUEST_FILENAME "#endsWith /index.php" \
"id:80,\
phase:1,\
pass,\
t:none,\
nolog,\
chain"
SecRule ARGS:action "#streq submit" \
"t:none,\
chain"
SecRule &ARGS:action "#eq 1" \
"t:none,\
ctl:ruleRemoveTargetById=930120;ARGS:wpTextbox1,\
ctl:ruleRemoveTargetById=932100;ARGS:wpTextbox1,\
ctl:ruleRemoveTargetById=932130;ARGS:wpTextbox1,\
ctl:ruleRemoveTargetById=941100;ARGS:wpTextbox1,\
ctl:ruleRemoveTargetById=941160;ARGS:wpTextbox1"

Could not initialize corosync configuration API error 12

Unable to initialize corosync running inside a docker container. The corosync-cfgtool -s command yields the following:
Could not initialize corosync configuration API error 12
The /etc/corosync/corosync.conf file has the following:
compatibility: whitetank
totem {
version: 2
secauth: off
threads: 0
interface {
ringnumber: 0
bindnetaddr: 127.0.0.1
mcastaddr: 239.255.1.1
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
logfile: /var/log/corosync.log
to_syslog: yes
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
The /var/log/corosync.log file shows the following:
May 02 20:13:22 corosync [MAIN ] Could not set SCHED_RR at priority 99: Operation not permitted (1)
May 02 20:13:22 corosync [MAIN ] Could not lock memory of service to avoid page faults: Cannot allocate memory (12)
May 02 20:13:22 corosync [MAIN ] Corosync Cluster Engine ('1.4.6'): started and ready to provide service.
May 02 20:13:22 corosync [MAIN ] Corosync built-in features: nss
May 02 20:13:22 corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
May 02 20:13:22 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
May 02 20:13:22 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
I was running the following in a bash script:
service corosync start
service corosync status
corosync-cfgtool -s
Apparently it was running too quickly and not giving corosync enough time to initialize. Changing the script to the following seems to have worked:
service corosync start
service corosync status
sleep 5
corosync-cfgtool -s
I now see the following output from corosync-cfgtool -s:
Printing ring status.
Local node ID 16777343
RING ID 0
id = 127.0.0.1
status = ring 0 active with no faults

Why wont Cygnus receive a subscription on my CentOS 6.7?

I just finished testing the entire thing on my virtual machine environment and now I am trying to launch it on the dedicated server. And now I ran into a completely new issue. First I confirmed that I have both context Broker and centos running (on 1026 and 5050 respectively):
[root#centos conf]# netstat -ntlpd
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1026 0.0.0.0:* LISTEN 1321/contextBroker
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1282/mongod
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1791/mysqld
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1260/sshd
tcp 0 0 :::1026 :::* LISTEN 1321/contextBroker
tcp 0 0 :::8081 :::* LISTEN 2481/java
tcp 0 0 :::22 :::* LISTEN 1260/sshd
tcp 0 0 :::5050 :::* LISTEN 2481/java
[root#centos conf]# service cygnus status
Cygnus 1 status...
cygnus-flume-ng (pid 2481) is running...
Then I confirmed that I have data on contextBroker because this command gave me an appropriate response:
(curl localhost:1026/v1/queryContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
]
}
EOF
Following the workaround to an issue with root user and logging I fixed the log4j.properties and changed the follwing:
flume.log.dir=/var/log/cygnus
I then started cygnus and got the following log:
Starting an ordered shutdown of Cygnus
Stopping sources
Stopping http-source (lyfecycle state=START)
All the channels are empty
Stopping channels
Stopping mysql-channel (lyfecycle state=START)
Stopping sinks
Stopping mysql-sink (lyfecycle state=START)
Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplic$
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.11.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
17 Dec 2015 13:35:37,684 INFO [main] (com.telefonica.iot.cygnus.nodes.CygnusApplication.main:235) - Starting a Jetty server listening on port 8081 (Management Interface)
17 Dec 2015 13:35:37,700 INFO [main] (org.mortbay.log.Slf4jLog.info:67) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
17 Dec 2015 13:35:37,700 INFO [main] (com.telefonica.iot.cygnus.nodes.CygnusApplication.main:238) - Starting Cygnus application
17 Dec 2015 13:35:37,700 INFO [Thread-1] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,713 INFO [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61) - Configuration provider starting
17 Dec 2015 13:35:37,715 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/cygnus/conf/agent_1.conf
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,725 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:mysql-sink
17 Dec 2015 13:35:37,755 INFO [Thread-1] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:8081
17 Dec 2015 13:35:37,764 WARN [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid:319) - Agent configuration for 'cygunsagent' does not contain any channels. Marking it as invalid.
17 Dec 2015 13:35:37,765 WARN [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:127) - Agent configuration invalid for agent 'cygunsagent'. It will be removed.
17 Dec 2015 13:35:37,766 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [cygnusagent]
17 Dec 2015 13:35:37,766 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150) - Creating channels
17 Dec 2015 13:35:37,778 INFO [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40) - Creating instance of channel mysql-channel type memory
17 Dec 2015 13:35:37,782 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel mysql-channel
17 Dec 2015 13:35:37,783 INFO [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39) - Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
17 Dec 2015 13:35:37,791 INFO [conf-file-poller-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.<init>:75) - Cygnus version (0.11.0.2a9c87fb7fd6156225e2eed7fbc9792f1d9c5dfe)
17 Dec 2015 13:35:37,807 INFO [conf-file-poller-0] (com.telefonica.iot.cygnus.handlers.OrionRestHandler.configure:141) - Startup completed
17 Dec 2015 13:35:37,826 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: mysql-sink, type: com.telefonica.iot.cygnus.sinks.OrionMySQLSink
17 Dec 2015 13:35:37,839 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel mysql-channel connected to [http-source, mysql-sink]
17 Dec 2015 13:35:37,843 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTT$
17 Dec 2015 13:35:37,844 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel mysql-channel
17 Dec 2015 13:35:37,910 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: CHANNEL, name: mysql-channel, registered successfully.
17 Dec 2015 13:35:37,910 INFO [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: CHANNEL, name: mysql-channel started
17 Dec 2015 13:35:37,911 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink mysql-sink
17 Dec 2015 13:35:37,913 INFO [lifecycleSupervisor-1-1] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.start:153) - [mysql-sink] Startup completed
17 Dec 2015 13:35:37,915 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184) - Starting Source http-source
17 Dec 2015 13:35:37,916 INFO [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:92) - Grouping rules read:
17 Dec 2015 13:35:37,916 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101) - Shutting down configuration: { sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HT$
17 Dec 2015 13:35:37,917 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:105) - Stopping Source http-source
17 Dec 2015 13:35:37,920 ERROR [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.parseGroupingRules:165) - Error while parsing the Json-based grouping rules file. Details=null
17 Dec 2015 13:35:37,921 WARN [lifecycleSupervisor-1-2] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:98) - Grouping rules syntax has errors
17 Dec 2015 13:35:37,948 INFO [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,973 INFO [lifecycleSupervisor-1-2] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,974 INFO [lifecycleSupervisor-1-2] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110) - Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
17 Dec 2015 13:35:37,974 INFO [lifecycleSupervisor-1-2] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: SOURCE, name: http-source started
17 Dec 2015 13:35:37,974 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state$
17 Dec 2015 13:35:37,974 INFO [conf-file-poller-0] (org.mortbay.log.Slf4jLog.info:67) - Stopped SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,975 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:139) - Component type: SOURCE, name: http-source stopped
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:145) - Shutdown Metric for type: SOURCE, name: http-source. source.start.time == 1450355737974
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:151) - Shutdown Metric for type: SOURCE, name: http-source. source.stop.time == 1450355737975
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append-batch.accepted == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append-batch.received == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append.accepted == 0
17 Dec 2015 13:35:37,976 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.append.received == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.events.accepted == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.events.received == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: SOURCE, name: http-source. src.open-connection.count == 0
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.source.http.HTTPSource.stop:172) - Http source http-source stopped. Metrics: SOURCE:http-source{src.events.accepted=0, src.open-connection.count=0, src.append.receiv$
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:115) - Stopping Sink mysql-sink
17 Dec 2015 13:35:37,977 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor#76da521f counterGroup:{ name:nul$
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:125) - Stopping Channel mysql-channel
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:171) - Stopping component: org.apache.flume.channel.MemoryChannel{name: mysql-channel}
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:139) - Component type: CHANNEL, name: mysql-channel stopped
17 Dec 2015 13:35:37,987 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:145) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.start.time == 1450355737910
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:151) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.stop.time == 1450355737987
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.capacity == 1000
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.current.size == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.put.attempt == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.put.success == 0
17 Dec 2015 13:35:37,988 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.take.attempt == 1
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:167) - Shutdown Metric for type: CHANNEL, name: mysql-channel. channel.event.take.success == 0
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTT$
17 Dec 2015 13:35:37,989 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel mysql-channel
17 Dec 2015 13:35:37,989 INFO [lifecycleSupervisor-1-3] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: CHANNEL, name: mysql-channel started
17 Dec 2015 13:35:37,992 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173) - Starting Sink mysql-sink
17 Dec 2015 13:35:37,993 INFO [lifecycleSupervisor-1-8] (com.telefonica.iot.cygnus.sinks.OrionMySQLSink.start:153) - [mysql-sink] Startup completed
17 Dec 2015 13:35:37,993 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184) - Starting Source http-source
17 Dec 2015 13:35:37,993 INFO [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:92) - Grouping rules read:
17 Dec 2015 13:35:37,994 ERROR [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.parseGroupingRules:165) - Error while parsing the Json-based grouping rules file. Details=null
17 Dec 2015 13:35:37,994 WARN [lifecycleSupervisor-1-4] (com.telefonica.iot.cygnus.interceptors.GroupingInterceptor.initialize:98) - Grouping rules syntax has errors
17 Dec 2015 13:35:37,994 INFO [lifecycleSupervisor-1-4] (org.mortbay.log.Slf4jLog.info:67) - jetty-6.1.26
17 Dec 2015 13:35:37,996 INFO [lifecycleSupervisor-1-4] (org.mortbay.log.Slf4jLog.info:67) - Started SocketConnector#0.0.0.0:5050
17 Dec 2015 13:35:37,996 INFO [lifecycleSupervisor-1-4] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:94) - Component type: SOURCE, name: http-source started
Then I tried to subscribe the previously mentioned data to cygnus:
(curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
],
"attributes": [
"pressure"
"temperature"
],
"reference": "http://localhost:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"pressure",
"temperature"
]
}
],
"throttling": "PT1S"
}
EOF
Even after I updated the information on context Broker thinking it would trigger an event:
(curl localhost:1026/v1/updateContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"contextElements": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1",
"attributes": [
{
"name": "temperature",
"type": "float",
"value": "333"
},
{
"name": "pressure",
"type": "integer",
"value": "555"
}
]
}
],
"updateAction": "APPEND"
}
EOF
But the cygnus log remainged exactly the same and its like nothing even got through to it. Which is odd considering my agent_1.conf:
# Copyright 2014 Telefónica Investigación y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es
#=============================================
# To be put in APACHE_FLUME_HOME/conf/agent.conf
#
# General configuration template explaining how to setup a sink of each of the available types (HDFS, CKAN, MySQL).
#=============================================
# The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
# ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
# Regarding sinks, you can use multiple types at the same time; the only requirement is to provide a channel for each
# one of them (this example shows how to configure 3 sink types at the same time). Even, you can define more than one
# sink of the same type and sharing the channel in order to improve the performance (this is like having
# multi-threading).
cygnusagent.sources = http-source
cygnusagent.sinks = mysql-sink
cygnusagent.channels = mysql-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = mysql-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = Trace_Data
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = Sensor
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts gi
# TimestampInterceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# GroupinInterceptor, do not change
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
# Grouping rules for the GroupingInterceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.gi.grouping_rules_conf_file = /usr/cygnus/conf/grouping_rules.conf
# ============================================
# OrionMySQLSink configuration
# channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel
# sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.OrionMySQLSink
# true if the grouping feature is enabled for this sink, false otherwise
cygnusagent.sinks.mysql-sink.enable_grouping = false
# the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = 127.0.0.1
# the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306
# a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = root
# password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = klasika
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = column
# select the table type from table-by-destination and table-by-service-path
cygnusagent.sinks.mysql-sink.table_type = table-by-destination
# number of notifications to be included within a processing batch
cygnusagent.sinks.mysql-sink.batch_size = 1
# timeout for batch accumulation
cygunsagent.sinks.mysql-sink.batch_timeout = 30
#=============================================
# mysql-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory
# capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100
#============================================
It has 5050 and notify as the reference address. I double checked the cygnus_instance_1.conf as well and it is pointing at agent_1.conf
#####
#
# Configuration file for apache-flume
#
#####
# Copyright 2014 Telefonica Investigación y Desarrollo, S.A.U
#
# This file is part of fiware-cygnus (FI-WARE project).
#
# fiware-cygnus is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any
# later version.
# fiware-cygnus is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with fiware-cygnus. If not, see
# http://www.gnu.org/licenses/.
#
# For those usages not covered by the GNU Affero General Public License please contact with iot_support at tid dot es
# Who to run cygnus as. Note that you may need to use root if you want
# to run cygnus in a privileged port (<1024)
CYGNUS_USER=cygnus
# Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf
# Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_1.conf
# Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
# naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnusagent
# Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log
# Administration port. Must be unique per instance
ADMIN_PORT=8081
# Polling interval (seconds) for the configuration reloading
POLLING_INTERVAL=30
This is the content of my config folder in /usr/cygnus/conf:
[root#centos conf]# ls
agent_1.conf cygnus_instance_1.conf flume-env.sh grouping_rules.conf krb5.conf krb5_login.conf log4j.properties.template
agent.conf.template cygnus_instance.conf.template flume-env.sh.template grouping_rules.conf.template krb5.conf.template log4j.properties README.md
I noticed that there is an exacted mirror of this in /etc/cygnus/conf but I didn't touch anything because the installation only intructs me to use the /usr/ folder.
Here is my Mysql create statement. In this table I am expecting to receive the context broker data but I get nothing of course since the log didnt register anything.
CREATE TABLE sensor_room1_room (
sensorID INT NOT NULL AUTO_INCREMENT,
recvTime mediumtext,
fiwareservicepath text,
entityId text,
entityType text,
pressure text,
pressure_md text,
temperature text,
temperature_md text,
PRIMARY KEY (sensorID));
Edit 1:
Here is the listener
[root#centos conf]# nc -l 5050
But when I tried subscribing or updating context, nothing was received on the listener side. I am not taking into account the client side of nc: nc 127.0.0.1 5050 because it successfully sends everything i type (even gibberish).
I also tried the test: /usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_1.conf -n cygnusagent -Dflume.root.logger=DEBUG,console. I tried both 5050 and 8081 ports to subscribe to and then update context but nothing is read on the console.
Since I seriously have no idea why that subscription didn't work, but thanks to #fgalan I did manage to read the logs so I am posting the subscription that did trigger the event to cygnus:
(curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type: application/json' \
--header 'Accept: application/json' -d #- | python -mjson.tool) <<EOF
{
"entities": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
],
"attributes": [
"temperature"
],
"reference": "http://localhost:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"pressure"
]
}
],
"throttling": "PT5S"
}
EOF
Thank you #fgalan one more time!

web2py with MongoHQ deploy at openshift: RuntimeError: no driver available ('pymongo',)

I'm trying to deploy my app to openshift, but I got error message. Following are my logs:
==> app-root/logs/web2py.log <==
127.5.151.1, 2014-06-01 18:08:10, GET, /admin/static/images/questions.png, HTTP/1.1, 304, 0.010710
127.5.151.1, 2014-06-01 18:11:12, HEAD, /, HTTP/1.1, 303, 0.000809
127.5.151.1, 2014-06-01 18:11:13, HEAD, /, HTTP/1.1, 303, 0.004810
127.5.151.1, 2014-06-01 18:41:05, GET, /MarkIt/, HTTP/1.1, 500, 5.459672
127.5.151.1, 2014-06-01 18:49:36, GET, /MarkIt/, HTTP/1.1, 500, 5.658112
127.5.151.1, 2014-06-01 18:54:00, GET, /MarkIt/, HTTP/1.1, 500, 5.682422
127.5.151.1, 2014-06-01 19:10:50, HEAD, /, HTTP/1.1, 303, 0.000960
127.5.151.1, 2014-06-01 19:10:50, HEAD, /, HTTP/1.1, 303, 0.001024
127.5.151.1, 2014-06-01 19:56:22, GET, /MarkIt/, HTTP/1.1, 500, 5.487744
127.5.151.1, 2014-06-01 20:04:03, GET, /MarkIt/, HTTP/1.1, 500, 5.431559
==> app-root/logs/python.log <==
[Sun Jun 01 20:04:02 2014] [error] DEBUG: connect attempt 4, connection error:
[Sun Jun 01 20:04:02 2014] [error] Traceback (most recent call last):
[Sun Jun 01 20:04:02 2014] [error] File "/var/lib/openshift/538b85575973cad5cc000794/app-root/runtime/repo/libs/gluon/dal.py", line 7409, in __init__
[Sun Jun 01 20:04:02 2014] [error] self._adapter = ADAPTERS[self._dbname](**kwargs)
[Sun Jun 01 20:04:02 2014] [error] File "/var/lib/openshift/538b85575973cad5cc000794/app-root/runtime/repo/libs/gluon/dal.py", line 5240, in __init__
[Sun Jun 01 20:04:02 2014] [error] if do_connect: self.find_driver(adapter_args)
[Sun Jun 01 20:04:02 2014] [error] File "/var/lib/openshift/538b85575973cad5cc000794/app-root/runtime/repo/libs/gluon/dal.py", line 746, in find_driver
[Sun Jun 01 20:04:02 2014] [error] raise RuntimeError("no driver available %s" % str(self.drivers))
[Sun Jun 01 20:04:02 2014] [error] RuntimeError: no driver available ('pymongo',)
97.77.53.27 - - [01/Jun/2014:20:03:56 -0400] "GET /MarkIt/ HTTP/1.1" 500 843 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14"
I'm using MongoHQ, the app runs totally fine if I run it locally.
The way I connect to MongoHQ is in db.py:
if not request.env.web2py_runtime_gae:
## if NOT running on Google App Engine use SQLite or other DB
#db = DAL('sqlite://storage.sqlite',pool_size=1,check_reserved=['all'])
db = DAL("mongodb://username:password#oceanic.mongohq.com:10017/MarkIt",
check_reserved=["mongodb_nonreserved",], adapter_args={"safe":False})
Only one of my applications in web2py uses MongoHQ and that's the one I could not open. Other work fine.
I think it might be dependency issue but I don't know how to resolve it.
I also did rhc cartridge add mongodb-2.4 -a my app but it does not help.

Vagrant Chef Recipe Checksum Calculations

I am running a mysql recipe that is failing. When I do vagrant up after a halt it claims that grants.sql template's checksum has changed causing it to re-run when it shouldn't.
[default] [Wed, 28 Mar 2012 12:58:48 -0700] INFO: Processing template[/etc/mysql/grants.sql] action create (mysql::server line 128)
: stdout
[default] [Wed, 28 Mar 2012 12:58:48 -0700] DEBUG: Current content's checksum: 3992e44304b56cebdbd4bf23183ddd78f877539c025227546e19098b0b5872ca
: stdout
[default] [Wed, 28 Mar 2012 12:58:48 -0700] DEBUG: Rendered content's checksum: f967f212b3e7b25a08ed35d086938846c188f6e9980a1ecc42635136841587a4
: stdout
[default] [Wed, 28 Mar 2012 12:58:48 -0700] INFO: template[/etc/mysql/grants.sql] backed up to /var/chef/backup/etc/mysql/grants.sql.chef-20120328125848
: stdout
[default] [Wed, 28 Mar 2012 12:58:48 -0700] INFO: template[/etc/mysql/grants.sql] updated content
: stdout
[default] [Wed, 28 Mar 2012 12:58:48 -0700] INFO: template[/etc/mysql/grants.sql] sending run action to execute[mysql-install-privileges] (immediate)
: stdout
[default] [Wed, 28 Mar 2012 12:58:48 -0700] INFO: Processing execute[mysql-install-privileges] action run (mysql::server line 137)
: stdout
[default] [Wed, 28 Mar 2012 12:58:48 -0700] INFO: execute[mysql-install-privileges] sh(/usr/bin/mysql -u root -p"evanta" < /etc/mysql/grants.sql)
: stdout
[default] [Wed, 28 Mar 2012 12:58:48 -0700] ERROR: execute[mysql-install-privileges] (mysql::server line 137) has had an error
[Wed, 28 Mar 2012 12:58:48 -0700] ERROR: template[/etc/mysql/grants.sql] (/tmp/vagrant-chef-1/chef-solo-1/mysql/recipes/server.rb:128:in `rescue in from_file') had an error:
execute[mysql-install-privileges] (mysql::server line 137) had an error: Chef::Exceptions::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /usr/bin/mysql -u root -p"evanta" < /etc/mysql/grants.sql ----
STDOUT:
STDERR: ERROR 1396 (HY000) at line 12: Operation CREATE USER failed for 'root'#'%'
---- End output of /usr/bin/mysql -u root -p"evanta" < /etc/mysql/grants.sql ----
Ran /usr/bin/mysql -u root -p"evanta" < /etc/mysql/grants.sql returned 1
Any ideas how these checksums are completed and how to fix this?
The best way to debug these issues is to take a look at the new file it created, in this case at /etc/mysql/grants.sql, and then to look at the backup at /var/chef/backup/etc/mysql/grants.sql.chef-20120328125848 (from the logs you posted). The backup is always made, so you can compare the contents of the two, and proceed to fix the Chef recipe to make sure it generates the same content.