I have a task which performs a GET request to a page. The response's body is a JSON like the following.
{
"ips": [
{
"organization": "1233124121",
"reverse": null,
"id": "1321411312",
"server": {
"id": "1321411",
"name": "name1"
},
"address": "x.x.x.x"
},
{
"organization": "2398479823",
"reverse": null,
"id": "2418209841",
"server": {
"id": "234979823",
"name": "name2"
},
"address": "x.x.x.x"
}
]
}
I want to extract the fields id and address, and tried (for id field):
tasks:
- name: get request
uri:
url: "https://myurl.com/ips"
method: GET
return_content: yes
status_code: 200
headers:
Content-Type: "application/json"
X-Auth-Token: "0010101010"
body_format: json
register: json_response
- name: copy ip_json content into a file
copy: content={{json_response.json.ips.id}} dest="/dest_path/json_response.txt"
but I get this error:
the field 'args' has an invalid value, which appears to include a variable
that is undefined. The error was: 'list object' has no attribute 'id'..
Where is the problem?
The error was: 'list object' has no attribute 'id'
json_response.json.ips is a list.
You either need to choose one element (first?): json_response.json.ips[0].id.
Or process this list for example with map or json_query filters if you need all ids.
Ansible command to copy to file:
copy:
content: "{{ output.stdout[0] }}"
dest: "~/ansible/local/facts/{{ inventory_hostname }}.txt"
Related
I am using log4js to log data to a file in my app. I want to display some of this data in my Grafana dashboard and for that I am using Promtail to read logs from the file, pre-process it and send it to Loki. In Loki, I want to filter the data based on the parsed values.
Here is an example of my logs:
[2023-02-12T04:01:23.587] [DEBUG] default - {
"message_id": 123,
"from": {
"id": 123,
"is_bot": false,
"first_name": "XXX",
"last_name": "XXXXX",
"username": "XXXXX",
"is_premium": true
},
"chat": {
"id": 123,
"title": "XXXXX",
"username": "XXXXX",
"type": "supergroup"
},
"date": 123,
"message_thread_id": 123,
"text": "XXX XXXXX XXXXX"
}
Here is my current Promtail configuration:
server:
http_listen_port: 80
grpc_listen_port: 9095
log_level: debug
positions:
filename: /tmp/positions.yaml
clients:
- url: http://192.168.1.64:3100/loki/api/v1/push
scrape_configs:
- job_name: patriotbot
pipeline_stages:
- multiline:
firstline: \[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}\]
- regex:
expression: '^\[.*\]\s\w*\s-\s(\{.*\})'
name: log_entry
- json:
source: log_entry
name: only_json
expressions:
from_id: 'from.id'
is_bot: 'from.is_bot'
first_name: 'from.first_name'
last_name: 'from.last_name'
username: 'from.username'
chat_id: 'chat.id'
chat_title: 'chat.title'
chat_type: 'chat.type'
- output:
source: only_json
static_configs:
- targets:
- localhost
labels:
job: patriot
type: all
__path__: /logs/all.log
I have two issues with my current configuration:
My logs are saved as multiple lines, making them difficult to parse. I have attempted to fix this issue with the multiline stage, but which is displayed properly in Loki, though maybe it won't work for following parsing.
The timestamp and metadata in front of the JSON object is preventing it from being parsed properly. Should I get rid it before sending it to the pipe and parse?
So can somebody suggest changes to my configuration that would allow me to properly parse these multiline logs and extract the relevant data?
I used the code to convert json data from Yarn REST API to Prometheus data type:
https://github.com/prometheus-community/json_exporter.
However, it printed errors:
level=error ts=2021-07-08T06:31:03.712Z caller=collector.go:83 msg="Failed to extract value for metric" path={.capacity} err="capacity is not found" metric="Desc{fqName: "queues_capacity", help: "information on queues", constLabels: {}, variableLabels: [type]}"
I was wondering if there is some wrong in my configuration of YAML file (such as in terms of nested json) or just the reason about the code.
my yaml config is:
metrics:
- name: queues
path: "{ .scheduler.schedulerInfo.queues.queue }"
help: information on queues
type: object
labels:
type: '{.type}'
values:
capacity: '{.capacity}'
and part of the json file is:
{
"scheduler": {
"schedulerInfo": {
"type": "capacityScheduler",
"capacity": 100,
"usedCapacity": 1.0526316,
"maxCapacity": 100,
"queueName": "root",
"queues": {
"queue": [
{
"type": "capacitySchedulerLeafQueueInfo",
"capacity": 10,
"usedCapacity": 10.526316,
"maxCapacity": 100,
use [*] to get all object first, so it should be:
path: "{ .scheduler.schedulerInfo.queues.queue[*] }"
according to this: https://kubernetes.io/docs/reference/kubectl/jsonpath/
I am trying to use the ansible.builtin.lookup plugin in order to read a JSON file from the local directory and then pass it as the payload to the ansible.builtin.uri module to send a POST message to a URI endpoint.
Following are the contents of my JSON file (config.json):
{
"Configuration": {
"Components": [
{
"Name": "A",
"Attributes": [
{
"Name": "A1",
"Value": "1",
"Set On Import": "True",
"Comment": "Read and Write"
},
{
"Name": "A2",
"Value": "2",
"Set On Import": "True",
"Comment": "Read and Write"
}
]
}
]
}
}
I need to send the above JSON content as the below string in the payload to ansible.builtin.uri module:
"{\"Configuration\": {\"Components\": [{\"Name\": \"A\", \"Attributes\": [{\"Name\": \"A1\", \"Value\": \"1\", \"Set On Import\": \"True\", \"Comment\": \"Read and Write\"}, {\"Name\": \"A2\", \"Value\": \"2\", \"Set On Import\": \"True\", \"Comment\": \"Read and Write\"}]}]}}"
I am trying to use the lookup plugin with the to_json filter to read and format the JSON content. Following is my playbook:
- name: import scp
ansible.builtin.uri:
url: "https://{{ inventory_hostname }}/api/config/actions/import"
user: "{{ user }}"
password: "{{ password }}"
method: POST
headers:
Accept: "application/json"
Content-Type: "application/json"
body:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
Buffer: "{{ lookup('file', 'config.json') | to_json }}"
body_format: json
status_code: 202
validate_certs: no
force_basic_auth: yes
However, the uri module double escapes all the new-line and tab characters. Following is the how the payload is sent when I run the playbook:
"invocation": {
"module_args": {
"attributes": null,
"body": {
"Buffer": "\"{\\n\\t\\\"Configuration\\\": {\\n\\t\\t\\\"Components\\\": [\\n\\t\\t\\t{\\n\\t\\t\\t\\t\\\"Name\\\": \\\"A\\\",\\n\\t\\t\\t\\t\\\"Attributes\\\": [\\n\\t\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\t\\\"Name\\\": \\\"A1\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Value\\\": \\\"1\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Set On Import\\\": \\\"True\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Comment\\\": \\\"Read and Write\\\"\\n\\t\\t\\t\\t\\t},\\n\\t\\t\\t\\t\\t{\\n\\t\\t\\t\\t\\t\\t\\\"Name\\\": \\\"A2\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Value\\\": \\\"2\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Set On Import\\\": \\\"True\\\",\\n\\t\\t\\t\\t\\t\\t\\\"Comment\\\": \\\"Read and Write\\\"\\n\\t\\t\\t\\t\\t}\\n\\t\\t\\t\\t]\\n\\t\\t\\t}\\n\\t\\t]\\n\\t}\\n}\"",
"Parameters": {
"IgnoreCertificateWarning": "Enabled",
"Type": "LOCAL_FILE",
"Target": "ALL"
},
},
"body_format": "json",
...
},
Could you please let me know how I can format the payload with uri module? Appreciate any help.
Edited (5/11/2021):
I made the changes as suggested by #mdaniel in his response and used string filter instead of to_json. With the suggested change, I can see the JSON being formatted properly into a string with newline ('\n') and tab ('\t') characters. I tried to use the replace filter to remove the \n and \t characters. However, now the whole string is converted back into the JSON.
Following is the playbook and the output when using the string filter alone:
...
body:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
Buffer: "{{ lookup('file', 'config.json') | string }}"
$ ansible-playbook import_file.yml -i hosts --tags
...
"body": {
"HostPowerState": "On",
"Buffer": "{\n\t\"Configuration\": {\n\t\t\"Components\": [\n\t\t\t{\n\t\t\t\t\"Name\": \"A\",\n\t\t\t\t\"Attributes\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"A1\",\n\t\t\t\t\t\t\"Value\": \"1\",\n\t\t\t\t\t\t\"Set On Import\": \"True\",\n\t\t\t\t\t\t\"Comment\": \"Read and Write\"\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"A2\",\n\t\t\t\t\t\t\"Value\": \"2\",\n\t\t\t\t\t\t\"Set On Import\": \"True\",\n\t\t\t\t\t\t\"Comment\": \"Read and Write\"\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t]\n\t}\n}",
"Parameters": {
"IgnoreCertificateWarning": "Enabled",
"Type": "LOCAL_FILE",
"Target": "ALL"
},
},
Following is the playbook and the output when using replace filter in conjunction with string filter:
...
body:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
Buffer: "{{ lookup('file', 'config.json') | string | replace('\n', '') | replace('\t', '') }}"
...
$ ansible-playbook import_file.yml -i hosts --tags
...
"body": {
"Buffer": {
"Configuration": {
"Components": [
{
"Attributes": [
{
"Comment": "Read and Write",
"Name": "A1",
"Set On Import": "True",
"Value": "1"
},
{
"Comment": "Read and Write",
"Name": "A2",
"Set On Import": "True",
"Value": "2"
}
],
"Name": "A"
}
]
}
},
"Parameters": {
"IgnoreCertificateWarning": "Enabled",
"Type": "LOCAL_FILE",
"Target": "ALL"
},
},
...
Any pointers on how I remove the \n and \t characters from the string?
You have used to_json on a dict value that is, itself, going to be to_json-ed; ansible cannot transmit a python dict over HTTP, so any yaml structure that is not already a string needs to be converted into one first
What you'll want is just that lookup result (which will return a str, not a dict) and then ansible will apply to_json to the whole body: value for the aforementioned reason
However, because ansible is trying to be "helpful", it will auto-coerce a yaml value that it finds starting with { back into a dict -- that's why you just need to send the result of lookup through the | string filter to reinforce to ansible that yes, you really do want it to remain a str in that context
...
body:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
Buffer: "{{ lookup('file', 'config.json') | string }}"
updated answer approach
In light of the comment discussion that the dict coercion was continuing to be a problem, and the leading space concerned the OP, the alternative approach is to build up the actual payload structure completely, and only "JSON-ify" it before transmission, to keep ansible and jinja on the same page about the data types:
- name: import scp
vars:
body_dict:
Parameters:
Type: "LOCAL_FILE"
Target: "ALL"
IgnoreCertificateWarning: "Enabled"
# this will be filled in before submission
# Buffer:
whitespace_free_config_json: >-
{{ lookup('file', 'config.json')
| regex_replace('[\t\n]', '')
| string
}}
ansible.builtin.uri:
...
body: >-
{{ body_dict
| combine({"Buffer": whitespace_free_config_json})
| to_json }}
body_format: json
status_code: 202
I am new to Ansible. I am trying to fetch some API info from a centralized server in the json format, parse the json and set "certificate" to "yes" for each and every server and make a PUT operation back. GET and PUT operations in my code are working fine.
I am not able to find correct syntax to parse listofservers.json.
What is the syntax to display a specific json element??
Can you help me on how to parse the json to change "Certificate" value to Yes for all the servers?
My script
- hosts: serverlist
vars_prompt:
- name: "USER"
prompt: "Enter Admin Username"
private: no
- name: "PASS"
prompt: "Enter Admin Password"
private: yes
tasks:
- name: "Get the current values of all Servers"
delegate_to: localhost
register: listofservers
check_mode: no
uri:
validate_certs: "no"
user: "{{USER}}"
password: "{{PASS}}"
url: "https://console.exmaple.com/adminapi/serverlist"
method: "GET"
body_format: "json"
return_content: "no"
status_code: "200"
- name: Print json dictionary
debug:
var: listofservers.json
Output
TASK [Print json dictionary] *****************************************************************************************
ok: [centalServer.example.com] =>
{
"listofservers.json": {
"serverlist": [
{
"id": 1,
"servername": "redhat.example.com",
"apachestatus": "running",
"certificate": "no"
},
{
"id": 2,
"servername": "solaris.example.com",
"apachestatus": "down",
"certificate": "yes"
} ] } }
Q: "Change "certificate" value to "yes" for all servers."
A: The task below does the job
- set_fact:
listofservers: "{{ {'json':
{'serverlist': serverlist}} }}"
vars:
serverlist: "{{ listofservers.json.serverlist|
map('combine', {'certificate': 'yes'})|
list }}"
- debug:
var: listofservers.json
give
{
"listofservers.json": {
"serverlist": [
{
"apachestatus": "running",
"certificate": "yes",
"id": 1,
"servername": "redhat.example.com"
},
{
"apachestatus": "down",
"certificate": "yes",
"id": 2,
"servername": "solaris.example.com"
}
]
}
}
Q: "What is the syntax to display a specific JSON element?"
A: There are two main options 1) Ansible filters and 2) json_query, and many variations which depend on the structure of the data and requested query
For example, display first servername where id is equal to 1
- debug:
msg: "{{ listofservers.json.serverlist|
selectattr('id', 'eq', 1)|
map(attribute='servername')|
first }}"
gives
"msg": "redhat.example.com"
The same result will give json_query below
- debug:
msg: "{{ listofservers.json.serverlist|
json_query('[?id == `1`].servername')|
first }}"
I have the following files
user.json
"user": {
"id": 1,
"name": "nameuser",
"online": true,
"profile": {
"photo": "",
"validated": true,
"popular": true,
"suspect": false,
"moderator": false,
"age": "22 ani",
"gender_id": "M"
}
}
profile.raml
displayName: Profile
get:
description: Get profile data
queryParameters:
userId:
description: The user id for which we are requesting the profile data
type: integer
required: true
responses:
200:
body:
application/json:
example: |
{
"user": !include user.json,
"details": {
"friend": true
}
}
The user json is present in more examples and I want to reuse it.
I'm using raml2html and it compiles it to
so how do I do this ?
I have used parameters successfully in the past. You will not be able to put a parameter inside an included file because RAML views all included files as strings. But you can do something like this in your profile.raml:
example: |
{
"user": <<userItem>>,
"details": {
"friend": true
}
}
The RAML 200 Tutorial has a good explanation and code examples (see snippets below) on how to declare parameters and them pass them in. I highly recommend reading the entire tutorial though.
resourceTypes:
- collection:
description: Collection of available <<resourcePathName>> in Jukebox.
get:
description: Get a list of <<resourcePathName>>.
responses:
200:
body:
application/json:
example: |
<<exampleCollection>>
/songs:
type:
collection:
exampleCollection: !include jukebox-include-songs.sample