I have some problems parsing json logs that were received from docker container. I know this question is probably a duplicate but none of the solutions found, including the documentation (https://docs.fluentd.org/filter/parser), helped.
Now my fluentd.conf like this:
<source>
#type forward
port 24224
bind "0.0.0.0"
</source>
<match mylog>
#type opensearch
#log_level "info"
host "opensearch-node1"
port 9200
logstash_format true
logstash_dateformat "%Y%m%d"
logstash_prefix "mylog"
</match>
<filter **>
#type parser
key_name "log"
hash_value_field "log"
reserve_data true
<parse>
#type "json"
</parse>
</filter>
But it feels like the filter is not working because the log field does not change
Result in OpenSearch:
{
"_index": "mylog-20221024",
"_id": "93cECoQBW3_q0OULbk2_",
"_version": 1,
"_score": null,
"_source": {
"container_id": "47de762b4d113478ee125417abd9e6ec7d963aa0d77758fc92cf3a18b0d2ad86",
"container_name": "/mylog",
"source": "stdout",
"log": "{\"time\":\"2022-10-24T15:42:48.480757224+03:00\",\"id\":\"\",\"remote_ip\":\"10.73.133.144\",\"host\":\"localhost:80\",\"method\":\"GET\",\"uri\":\"/metrics\"}",
"#timestamp": "2022-10-24T12:42:48.000000000+00:00"
},
"fields": {
"#timestamp": [
"2022-10-24T12:42:48.000Z"
]
},
"sort": [
1666615368000
]
}
Expected:
{
"_index": "mylog-20221024",
"_id": "93cECoQBW3_q0OULbk2_",
"_version": 1,
"_score": null,
"_source": {
"container_id": "47de762b4d113478ee125417abd9e6ec7d963aa0d77758fc92cf3a18b0d2ad86",
"container_name": "/mylog",
"source": "stdout",
"log": "{\"time\":\"2022-10-24T15:42:48.480757224+03:00\",\"id\":\"\",\"remote_ip\":\"10.73.133.144\",\"host\":\"localhost:80\",\"method\":\"GET\",\"uri\":\"/metrics\"}",
"time": "2022-10-24T15:42:48.480757224+03:00",
"id":"",
"remote_ip":"10.73.133.144",
"host":"localhost:80",
"method":"GET",
"uri":"/metrics",
"#timestamp": "2022-10-24T12:42:48.000000000+00:00",
},
"fields": {
"#timestamp": [
"2022-10-24T12:42:48.000Z"
]
},
"sort": [
1666615368000
]
}
Thanks!
The solution to the problem turned out to be very simple. The <filter> should be placed before the <match>. Now my fluentd.conf like this:
<source>
#type forward
port 24224
bind "0.0.0.0"
</source>
<filter mylog**>
#type parser
format json
key_name log
reserve_data true
reserve_time true
inject_key_prefix logs.
</filter>
<match mylog>
#type opensearch
#log_level "info"
host "opensearch-node1"
port 9200
logstash_format true
logstash_dateformat "%Y%m%d"
logstash_prefix "mylog"
</match>
Related
Environment:
Ansible: 2.10.12
Python: 3.6.8
Jinja2: 3.0.1
Given an API response, I'm trying to grab the address, but getting "variable is not defined". Here's a playbook that has the API response as a var:
---
- name: test
gather_facts: no
hosts: localhost
vars:
pool_details: {
"allow": "",
"cache_control": "no-store, no-cache, must-revalidate",
"changed": false,
"connection": "close",
"content_length": "668",
"content_security_policy": "default-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; img-src 'self' data: http://127.4.1.1 http://127.4.2.1",
"content_type": "application/json; charset=UTF-8",
"cookies": {},
"cookies_string": "",
"date": "Tue, 24 Aug 2021 16:34:47 GMT",
"elapsed": 0,
"expires": "-1",
"failed": false,
"json": {
"items": [
{
"address": "10.200.136.22",
"connectionLimit": 0,
"dynamicRatio": 1,
"ephemeral": "false",
"fqdn": {
"autopopulate": "disabled"
},
"fullPath": "/Common/sensor01:443",
"generation": 103,
"inheritProfile": "enabled",
"kind": "tm:ltm:pool:members:membersstate",
"logging": "disabled",
"monitor": "default",
"name": "sensor01:443",
"partition": "Common",
"priorityGroup": 0,
"rateLimit": "disabled",
"ratio": 1,
"selfLink": "https://localhost/mgmt/tm/ltm/pool/~Common~sensor01/members/~Common~sensor01:443?ver=16.1.0",
"session": "monitor-enabled",
"state": "up"
}
],
"kind": "tm:ltm:pool:members:memberscollectionstate",
"selfLink": "https://localhost/mgmt/tm/ltm/pool/~Common~sensor01/members?ver=16.1.0"
},
"msg": "OK (668 bytes)",
"pragma": "no-cache",
"redirected": false,
"server": "Jetty(9.2.22.v20170606)",
"status": 200,
"strict_transport_security": "max-age=16070400; includeSubDomains",
"x_content_type_options": "nosniff",
"x_frame_options": "SAMEORIGIN",
"x_xss_protection": "1; mode=block"
}
tasks:
- debug:
var: pool_details
- debug:
var: pool_details.json.items[0].address
The full error, with verbosity enabled is:
pool_details.json.items[0].address: 'VARIABLE IS NOT DEFINED!: builtin_function_or_method object has no element 0'
My second debug to grab the address isn't working ("not defined"), & I can't figure out why. If I take the same json, & pipe it to jq, it works just fine:
pbpaste | jq '.json.items[0].address'
"10.200.136.22"
items is a Python dictionary method.
To return the value to the key "items":
- debug:
var: pool_details.json['items'][0].address
If there are more items in the list you might want to use json_query, e.g.
- set_fact:
list_address: "{{ pool_details.json['items']|
json_query('[].address') }}"
gives
list_address:
- 10.200.136.22
If you don't want to, or can't, install JmesPath, the next option would be map attribute, e.g. the code below gives the same result
- set_fact:
list_address: "{{ pool_details.json['items']|
map(attribute='address')|
list }}"
Establishing a connection to an MQTT 3.1.1 endpoint following the description here and
the Operating Devops Commands end up in an invalid json 400 response. Even the example MQTT-Bidirectional gets refused with a 400. So this is why i am posting this question here to get hints what i am currently doing wrong and what i can do to get it right to help others running in the same issue. Here is my curl request:
gd#gd:~/ditto/mosquitto$ curl -X POST 'http://{ditto-url}/devops/piggyback/connectivity?timeout=10000' -u devops:foobar -H 'Content-Type: application/json' -d createMQTT_connection.json
{"status":400,"error":"json.invalid","message":"Failed to parse JSON string 'createMQTT_connection.json'!","description":"Check if the JSON was valid (e.g. on https://jsonlint.com) and if it was in required format."}
The hint to check my json file gives back that my json is valid.
Here is how my json file currently looks like:
{
"targetActorSelection": "/system/sharding/connection",
"headers": {
"aggregate": false
},
"piggybackCommand": {
"type": "connectivity.commands:createConnection",
"connection": {
"id": "mqtt-example-connection-123",
"name": "mmqtt-example-connection-123",
"connectionType": "mqtt",
"connectionStatus": "open",
"failoverEnabled": true,
"uri": "tcp://{mqtt-broker-url}:1883",
"sources": [
{
"addresses": [
"{ditto-url}/#"
],
"authorizationContext": ["nginx:ditto"],
"qos": 0,
"filters": []
}
],
"targets": [
{
"address": "{ditto-url}/{{ thing:id }}",
"topics": [
"_/_/things/twin/events"
],
"authorizationContext": ["nginx:ditto"],
"qos": 0
}
]
}
}
}
Someone got an idea why this json is not valid?
Thanks for the support and hints to solve this issue!
[EDIT]
First of all a "-f" makes more sense for the curl request:
curl -X POST -u devops:foobar 'http://{ditto-url}:8080/devops/piggyback/connectivity?timeout=10000' -f createMQTT_connection_1.json
curl: (22) The requested URL returned error: 400 Bad Request
curl: (6) Could not resolve host: createMQTT_connection_1.json
Second here the update json (with the result shown above)
{
"targetActorSelection": "/system/sharding/connection",
"headers": {
"aggregate": false
},
"piggybackCommand": {
"type": "connectivity.commands:createConnection",
"connection": {
"id": "mqtt-example-connection-123",
"connectionType": "mqtt",
"connectionStatus": "open",
"failoverEnabled": true,
"uri": "tcp://{MQTT-Broker-url}:1883",
"sources": [{
"addresses": ["ditto-tutorial/#"],
"authorizationContext": ["nginx:ditto"],
"qos": 0,
"filters": []
}],
"targets": [{
"address": "ditto-tutorial/{{ thing:id }}",
"topics": [
"_/_/things/twin/events",
"_/_/things/live/messages"
],
"authorizationContext": ["nginx:ditto"],
"qos": 0
}]
}
}
}
I think the problem you face is curl related.
Please have a look here on how to send json data from a file: https://stackoverflow.com/a/18614411/5058051
Seems the # is missing in your case when specifying the file location.
I query the Cisco ACI to attain the advanced vmm provider details for a specific EPG.
The result is successful.
I then register the result to a variable.
I try to search that variable and obtain\extract a single specific piece of information such as 'dn' or 'encap' as this would allow me to use the information in other plays.
Unfortunately i'm unable to extract the information as the result comes back in an usual format. Looking at a debug on the register variable, it would appear it's a dictionary variable but no matter what I try the only item i'm able to access is the 'current' item.
All other items are not registered as dictionary items.
I have tried to change the variable to a list but still i'm unable to attain the information I require.
I've searched forums to see if the there is a methodology to convert the variable from a json result or dictionary variable to a string and then grep for the information but no success.
Ideally I would like to extract the information without installing additional 'apps'.
Will be very grateful if someone can advise how to search for a specific result from an irregular nested result which doesn't list the items in a correct dictionary format.
- name: Access VMM provider Information
hosts: apics
gather_facts: false
connection: local
#
vars:
ansible_python_interpreter: /usr/bin/python3
#
tasks:
- name: Play 1 Obtain VMM Provider Information
aci_epg_to_domain:
hostname: "{{ apics.hostname }}"
username: "{{ apics.username }}"
password: "{{ apics.password }}"
tenant: Tenant_A
ap: AP_Test
epg: EPG_Test
domain: DVS_Dell
domain_type: vmm
vm_provider: vmware
state: query
validate_certs: no
register: DVS_Result
#
- set_fact:
aci_result1: "{{ DVS_Result.current }}"
- set_fact:
aci_result2: "{{ DVS_Result.fvRsDomAtt.attributes.dn }}"
#
- debug:
msg: "{{ DVS_Result }}"
- debug:
var=aci_result1
- debug:
var=aci_result2
DVS_Result
ok: [apic1r] => {
"msg": {
"changed": false,
"current": [
{
"fvRsDomAtt": {
"attributes": {
"annotation": "",
"bindingType": "none",
"childAction": "",
"classPref": "encap",
"configIssues": "",
"delimiter": "",
"dn": "uni/tn-TN_prod/ap-AP_Test/epg-EPG_Test/rsdomAtt-[uni/vmmp-VMware/dom-DVS_Dell]",
"encap": "unknown",
"encapMode": "auto",
"epgCos": "Cos0",
"epgCosPref": "disabled",
"extMngdBy": "",
"forceResolve": "yes",
"instrImedcy": "lazy",
"lagPolicyName": "",
"lcOwn": "local",
"modTs": "2019-08-18T20:52:13.570+00:00",
"mode": "default",
"monPolDn": "uni/tn-common/monepg-default",
"netflowDir": "both",
"netflowPref": "disabled",
"numPorts": "0",
"portAllocation": "none",
"primaryEncap": "unknown",
"primaryEncapInner": "unknown",
"rType": "mo",
"resImedcy": "lazy",
"secondaryEncapInner": "unknown",
"state": "missing-target",
"stateQual": "none",
"status": "",
"switchingMode": "native",
"tCl": "infraDomP",
"tDn": "uni/vmmp-VMware/dom-DVS_Dell",
"tType": "mo",
"triggerSt": "triggerable",
"txId": "8646911284551354729",
"uid": "15374"
}
}
}
],
"failed": false
}
}
######################################
### aci_result1
ok: [apic1r] => {
"aci_result1": [
{
"fvRsDomAtt": {
"attributes": {
"annotation": "",
"bindingType": "none",
"childAction": "",
"classPref": "encap",
"configIssues": "",
"delimiter": "",
"dn": "uni/tn-TN_prod/ap-AP_Test/epg-EPG_Test/rsdomAtt-[uni/vmmp-VMware/dom-DVS_Dell]",
"encap": "unknown",
"encapMode": "auto",
"epgCos": "Cos0",
"epgCosPref": "disabled",
"extMngdBy": "",
"forceResolve": "yes",
"instrImedcy": "lazy",
"lagPolicyName": "",
"lcOwn": "local",
"modTs": "2019-08-18T20:52:13.570+00:00",
"mode": "default",
"monPolDn": "uni/tn-common/monepg-default",
"netflowDir": "both",
"netflowPref": "disabled",
"numPorts": "0",
"portAllocation": "none",
"primaryEncap": "unknown",
"primaryEncapInner": "unknown",
"rType": "mo",
"resImedcy": "lazy",
"secondaryEncapInner": "unknown",
"state": "missing-target",
"stateQual": "none",
"status": "",
"switchingMode": "native",
"tCl": "infraDomP",
"tDn": "uni/vmmp-VMware/dom-DVS_Dell",
"tType": "mo",
"triggerSt": "triggerable",
"txId": "8646911284551354729",
"uid": "15374"
}
}
}
]
}
############################################
### aci_result2
fatal: [apic1r]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'fvRsDomAtt'\n\nThe error appears to be in '/etc/ansible/playbooks/cisco/aci/create_bd_ap_epg3.yml': line 37, column 8, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - set_fact:\n ^ here\n"}
Use json_query. For example
- debug:
msg: "{{ DVS_Result.current|
json_query('[].fvRsDomAtt.attributes.dn') }}"
I have logs like the following:
{
"log": {
"header": {
"key": "value",
"nested": "{\"key1\":\"value\",\"key2\":\"value\"}",
"dateTime": "2019-05-08T20:58:06+00:00"
},
"body": {
"path": "/request/path/",
"method": "POST",
"ua": "curl/7.54.0",
"resp": 200
}
}
}
I'm trying to aggregate logs using fluentd and I want the entire record to be JSON. The specific problem is the "$.log.header.nested" field, which is a JSON string. How can I parse and replace that string with its contents?
For clarity, I'd like the logs output by fluentd to look like this:
{
"log": {
"header": {
"key": "value",
"nested": {
"key1": "value",
"key2": "value"
},
"dateTime": "2019-05-08T20:58:06+00:00"
},
"body": {
"path": "/request/path/",
"method": "POST",
"ua": "curl/7.54.0",
"resp": 200
}
}
}
I've found a way to parse the nested field as JSON, but storing to back to the same key it was parsed from isn't clear. It doesn't seem like hash_value_field supports storing to a nested key. Is there some other way to accomplish this?
The following config seems to accomplish what I want. However, I'm not sure if this is the best way. I assume using ruby is far less performant. Any improvements to this are welcome.
<filter logs>
#type parser
key_name "$.log.header.nested"
hash_value_field "parsed_nested"
reserve_data true
remove_key_name_field true
<parse>
#type json
</parse>
</filter>
<filter logs>
#type record_transformer
enable_ruby true
<record>
parsed_nested ${record["log"]["header"]["nested"] = record["parsed_nested"]}
</record>
remove_keys parsed_nested
</filter>
We are upgrading our Data pipeline version from 3.3.2 to 5.8, so those bootstrap actions on old AMI release have changed to be setup using configuration and specifying them under classification / property definition.
So my Json looks like below
{
"enableDebugging": "true",
"taskInstanceBidPrice": "1",
"terminateAfter": "2 Hours",
"name": "ExportCluster",
"taskInstanceType": "m1.xlarge",
"schedule": {
"ref": "Default"
},
"emrLogUri": "s3://emr-script-logs/",
"coreInstanceType": "m1.xlarge",
"coreInstanceCount": "1",
"taskInstanceCount": "4",
"masterInstanceType": "m3.xlarge",
"keyPair": "XXXX",
"applications": ["hadoop","hive", "tez"],
"subnetId": "XXXXX",
"logUri": "s3://pipelinedata/XXX",
"releaseLabel": "emr-5.8.0",
"type": "EmrCluster",
"id": "EmrClusterWithNewEMRVersion",
"configuration": [
{ "ref": "configureEmrHiveSite" }
]
},
{
"myComment": "This object configures hive-site xml.",
"name": "HiveSite Configuration",
"type": "HiveSiteConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},
{
"myComment": "This object sets a hive-site configuration
property value.",
"name":"hive-exec-compress-output",
"type": "Property",
"id": "hive-exec-compress-output",
"key": "hive.exec.compress.output",
"value": "true"
}
],
"parameters": []
With the above Json file it gets loaded into Data Pipeline but throws an error saying
Object:HiveSite Configuration
ERROR: 'HiveSiteConfiguration'
Object:ExportCluster
ERROR: 'configuration' values must be of type 'null'. Found values of type 'null'
I am not sure what this really means and could you please let me know if i am specifying this correctly which i think i am according to http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html
The below block should have the name as "EMR Configuration" only then its recognized correctly by the AWS Data pipeline and the Hive-site.xml is being set accordingly.
{
"myComment": "This object configures hive-site xml.",
"name": "EMR Configuration",
"type": "EmrConfiguration",
"id": "configureEmrHiveSite",
"classification": "hive-site",
"property": [
{"ref": "hive-exec-compress-output" }
]
},