I have an instance resource along these lines:
masters:
type: OS::Heat::ResourceGroup
properties:
count: { get_param: num_masters }
resource_def:
type: heat_stack_server.yaml
properties:
name:
str_replace:
template: cluster_id-k8s_type-%index%
params:
cluster_id: { get_param: cluster_id }
k8s_type: master
cluster_env: { get_param: cluster_env }
cluster_id: { get_param: cluster_id }
type: master
image: { get_param: master_image }
flavor: { get_param: master_flavor }
key_name: { get_resource: keypair }
net: { get_resource: net }
subnet: { get_resource: subnet }
secgrp:
- { get_resource: master-secgrp }
- { get_resource: node-secgrp }
floating_network: { get_param: external_net }
net_name:
str_replace:
template: openshift-ansible-cluster_id-net
params:
cluster_id: { get_param: cluster_id }
depends_on:
- interface
It creates num_masters. Now, I want to guarantee these masters will be created in different availability zones (so that when one of them fails, the other will continue to work).
Say, I have 3 AZ and num_masters == 5. How to spread them, so that zone1 contains nodes 1 and 4, zone2 - 2 and 5, and so on?
Ansible has that loop.cycle thing where you could pass over a list of options over and over. Any ideas how to do it in OS?
OK, I found a solution. I see that someone ticked my question up, so I understand that there's someone else searching for a solution, so I'd better share mine.
You rarely use (and I certainly don't) Heat in isolation from other Configuration Management frameworks. I use it alongside Ansible. So in order to spread nodes between availability zones (AZ) you may consider to prepare this spread yourself. First, I have in my Ansible vars file a list of all AZ available (sorry for the pun):
zones:
- 'zone1'
- 'zone2'
Alternatively, you can query Openstack for that list. When you have it, you fill it into the environment file of your stack like this:
{% set zone_cycler = cycler( *zones ) %}
master_availability_zones: [{% for n in range(1,master_number+1) %}"{{ zone_cycler.next() }}"{% if not loop.last %}{{','}} {% endif %}{% endfor %}]
So for five hosts and two zones you'll get this:
master_availability_zones: ["zone1","zone2","zone1","zone2","zone1"]
Then you pass this list into your host resource group like this:
master_availability_zones:
type: comma_delimited_list
label: Master Availability zones
description: Availability zone mapping for masters
master_nodes:
type: OS::Heat::ResourceGroup
properties:
count: { get_param: master_number }
resource_def:
type: master_template.yaml
properties:
...
availability_zones: { get_param: master_availability_zones }
index: "%index%"
...
Don't forget to pass along index variable as well, you'll need it on the other side, in master_template.yaml:
master_node:
type: OS::Nova::Server
properties:
...
availability_zone: { get_param: [ availability_zones, { get_param: index } ] }
...
Voila, you now have scalable procedure accomodating for arbitrary host and zone numbers.
Related
I have a variable template
var1.yml
variables:
- name: TEST_DB_HOSTNAME
value: 10.123.56.222
- name: TEST_DB_PORTNUMBER
value: 1521
- name: TEST_USERNAME
value: TEST
- name: TEST_PASSWORD
value: TEST
- name: TEST_SCHEMANAME
value: SCHEMA
- name: TEST_ACTIVEMQNAME
value: 10.123.56.223
- name: TEST_ACTIVEMQPORT
value: 8161
When I run the below pipeline
resources:
repositories:
- repository: templates
type: git
name: pipeline_templates
ref: refs/heads/master
trigger:
- none
variables:
- template: templates/var1.yml#templates
pool:
name: PoolA
steps:
- pwsh: |
Write-Host "${{ convertToJson(variables) }}"
I get the output
{
build.sourceBranchName: master,
build.reason: Manual,
system.pullRequest.isFork: False,
system.jobParallelismTag: Public,
system.enableAccessToken: SecretVariable,
TEST_DB_HOSTNAME: 10.123.56.222,
TEST_DB_PORTNUMBER: 1521,
TEST_USERNAME: TEST,
TEST_PASSWORD: TEST,
TEST_SCHEMANAME: SCHEMA,
TEST_ACTIVEMQNAME: 10.123.56.223,
TEST_ACTIVEMQPORT: 8161
}
How can I modify the pipeline to extract only the key value from the result set that starts with "Test_" and store into another variable in the same format so that I could be used in other tasks in the same pipeline ?
OR iterate through the objects that has keys "Test_" and get the value for the same ?
The output you have shown is invalid JSON and cannot be transformed with JSON. Assuming that it were valid JSON:
{
"build.sourceBranchName": "master",
"build.reason": "Manual",
"system.pullRequest.isFork": "False",
"system.jobParallelismTag": "Public",
"system.enableAccessToken": "SecretVariable",
"TEST_DB_HOSTNAME": "10.123.56.222",
"TEST_DB_PORTNUMBER": 1521,
"TEST_USERNAME": "TEST",
"TEST_PASSWORD": "TEST",
"TEST_SCHEMANAME": "SCHEMA",
"TEST_ACTIVEMQNAME": "10.123.56.223",
"TEST_ACTIVEMQPORT": 8161
}
then you can use the to_entries or with_entries filters of jq to get an object containing only those keys which start with "TEST_":
with_entries(select(.key|startswith("TEST_")))
This will give you a new object as output:
{
"TEST_DB_HOSTNAME": "10.123.56.222",
"TEST_DB_PORTNUMBER": 1521,
"TEST_USERNAME": "TEST",
"TEST_PASSWORD": "TEST",
"TEST_SCHEMANAME": "SCHEMA",
"TEST_ACTIVEMQNAME": "10.123.56.223",
"TEST_ACTIVEMQPORT": 8161
}
The convertToJson() function is a bit messy, as the "json" it creates is not, in fact, a valid json.
There are several possible approaches I can think of:
Use convertToJson() to pass the non-valid json to a script-step, convert it to a valid json and then extract the relevant values. I have done this before and it typically works, if you have control over the data in the variables. The downside is that there is risk that the conversion to valid json can fail.
Create a yaml-loop that iterates the variables and extract the ones that begins with Test_. You can find examples of how to write a loop here, but basically, it would look like this:
- stage:
variables:
firstVar: 1
secondVar: 2
Test_thirdVar: 3
Test_forthVar: 4
jobs:
- job: loopVars
steps:
- ${{ each var in variables }}:
- script: |
echo ${{ var.key }}
echo ${{ var.value }}
displayName: handling ${{ var.key }}
If applicable to your use case, you can create complex parameters (instead of variables) for only the Test_ variables. Using this, you could use the relevant values directly and would not need to extract a subset from your variable list. Note however, that parameters are inputs to a pipeline and can be adjusted before execution. Example:
parameters:
- name: non-test-variables
type: object
default:
firstVar: 1
secondVar: 2
- name: test-variables
type: object
default:
Test_thirdVar: 3
Test_forthVar: 4
You can use these by referencing ${{ parameters.Test_thirdVar }} in the pipeline.
I have the following json file called cust.json :
{
"customer":{
"CUST1":{
"zone":"ZONE1",
"site":"ASIA"
},
"CUST2":{
"zone":"ZONE2",
"site":"EUROPE"
}
}
}
I am using this json file in my main.yml to get a list of customers (CUST1 and CUST2).
main.yml:
- name: Include the vars
include_vars:
file: "{{ playbook_dir }}/../default_vars/cust.json"
name: "cust_json"
- name: Generate customer config
include_tasks: create_config.yml
loop: "{{ cust_json.customer }}"
I was hoping the loop will basically pass each customer's code (eg CUST1) to create_config.yml, so that something like the following can happen:
create_config.yml:
- name: Create customer config
block:
- name: create temporary file for customer
tempfile:
path: "/tmp"
state: file
prefix: "my customerconfig_{{ item }}."
suffix: ".tgz"
register: tempfile
- name: Setup other things
include_tasks: "othercustconfigs.yml"
Which will result in :
The following files being generated : /tmp/mycustomerconfig_CUST1 and /tmp/mycustomerconfig_CUST2
The tasks within othercustconfigs.yml be run for CUST1 and CUST2.
Questions :
Running the ansible, it fails at this point:
TASK [myrole : Generate customer config ] ************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {
"msg": "Invalid data passed to 'loop', it requires a list, got this instead: {u'CUST1': {u'site': u'ASIA', u'zone': u'ZONE1'}, u'CUST2': {u'site': u'EUROPE', u'zone': uZONE2'}}. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup."
}
How do I loop the JSON so that it would get the list of customers (CUST1 and CUST2) correctly? loop: "{{ cust_json.customer }}" clearly doesnt work.
If I manage to get the above working, is it possible to pass the result of the loop to the next include_tasks: "othercustconfigs.yml ? SO basically, passing the looped items from main.yml , then to config.yml, and then to othercustconfigs.yml. Is this possible?
Thanks!!
J
cust_json.customer is a hashmap containing one key for each customer, not a list.
The dict2items filter can transform this hashmap into a list of elements each containing a key and value attribute, e.g:
- key: "CUST1"
value:
zone: "ZONE1"
site: "ASIA"
- key: "CUST2"
value:
zone: "ZONE2"
site: "EUROPE"
With this in mind, you can transform your include to the following:
- name: Generate customer config
include_tasks: create_config.yml
loop: "{{ cust_json.customer | dict2items }}"
and the relevant task in your included file to:
- name: create temporary file for customer
tempfile:
path: "/tmp"
state: file
prefix: "my customerconfig_{{ item.key }}."
suffix: ".tgz"
register: tempfile
Of course you can adapt all this to use the value element where needed, e.g. item.value.site
You can see the following documentations for in depth info and alternative solutions:
https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#dict-filter
https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#iterating-over-a-dictionary
https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#with-dict
https://jinja.palletsprojects.com/en/2.11.x/templates/#dictsort
I'm calling a webservice and returning some JSON. I want to conditionally run a subsequent task, based on whether a particular name value is found.
For example, set a value nameExists if and only if the values array contains a name field of myDemo. So in this case, nameExists would be defined:
...
"failed": false,
"json": {
"values": [{
"id": "1234",
"name": "myDemo"
},
{
"id": "6789",
"name": "myDemo2"
}]
},
"msg": "OK (100 bytes)"
...
Here's what I'm currently trying:
# Call API
- name: Call API
uri:
url: myURL
method: POST
register: apiCheckResult
- name: Debug Auto tags
debug:
msg: "{{ item.name }}"
loop: "{{ apiCheckResult.json['values'] }}"
when: item.name == "myDemo"
register: tagExists
This works, in a way, but it gives me the full JSON output, all I need is a true / false.
Am I on the right track or is there a better way to achieve this?
You don't (normally) use a debug task to set variables. You probably want to use set_fact. If I understand your question correctly, you want to set a boolean tagExists to true if one of the items in the values list of the API response contains the name myDemo. That might look like this:
- set_fact:
tagExists: "{{ apiCheckResult.json|json_query('values[?name == `myDemo`]') }}"
"But wait!", you say, "that's not a boolean!". While you are correct, you can treat it like on. For example, after having set tagExists using that task, you could do this:
- debug:
msg: "The tag exists!"
when: tagExists
This works because a non-empty list evaluates as a true value in a boolean context (and an empty list evaluates as false). The json_query expression above returns a non-empty list when there is a match, and an empty list otherwise.
If you really want a boolean, you could do this instead:
- set_fact:
tagExists: "{{ true if apiCheckResult.json|json_query('values[?name == `myDemo`]') else false }}"
I have been stuck to get a particular json object if the value of a key matches a variable (string).
My json file looks like this:
"totalRecordsWithoutPaging": 1234,
"jobs": [
{
"jobSummary": {
"totalNumOfFiles": 0,
"jobId": 8035,
"destClientName": "BOSDEKARLSSP010",
"destinationClient": {
"clientId": 10,
"clientName": "BOSDEKARLSSP010"
}
}
},
{
"jobSummary": {
"totalNumOfFiles": 0,
"jobId": 9629,
"destClientName": "BOSDEKARLSSP006",
"destinationClient": {
"clientId": 11,
"clientName": "BOSDEKARLSSP006"
}
}
},
.....
]
}
I read this json with result: "{{ lookup('file','CVExport-short.json') | from_json }}" and I can get only one value of destClientName key with the following code:
- name: Iterate JSON
set_fact:
app_item: "{{ item.jobSummary }}"
with_items: "{{ result.jobs }}"
register: app_result
- debug:
var: app_result.results[0].ansible_facts.app_item.destClientName
My goal is to get the value of jobIdif the value of destClientName matches some other variable or string in any jobSummary.
I don't still have much knowledge in Ansible. So, any help would be much appreciated.
Update
Ok, I have found one solution.
- name: get job ID
set_fact:
job_id: "{{ item.jobSummary.jobId }}"
with_items: "{{ result.jobs}}"
when: item.jobSummary.destClientName == '{{ target_vm }}'
- debug:
msg: "{{job_id}}"
But I think there might be a better solution than this. Any idea how?
Ansible's json_query filter let's you perform complex filtering of JSON documents by applying JMESPath expressions. Rather than looping over the jobs in the the result, you can get the information you want in a single step.
We want to query all jobs in which have a destClientName that matches the value in target_vm. Using literal values, the expression yielding that list of jobs would look like this:
jobs[?jobSummary.destClientName == `BOSDEKARLSSP006`]
The result of this, when applied to your sample data, would be:
[
{
"jobSummary": {
"totalNumOfFiles": 0,
"jobId": 9629,
"destClientName": "BOSDEKARLSSP006",
"destinationClient": {
"clientId": 11,
"clientName": "BOSDEKARLSSP006"
}
}
}
]
From this result, you want to extract the jobId, so we rewrite the expression like this:
jobs[?jobSummary.destClientName == `BOSDEKARLSSP006`]|[0].jobSummary.jobId
Which gives us:
9629
To make this work in a playbook, you'll want to replace the literal hostname in this expression with the value of your target_vm variable. Here's a complete playbook that demonstrates the solution:
---
- hosts: localhost
gather_facts: false
# This is just the sample data from your question.
vars:
target_vm: BOSDEKARLSSP006
results:
totalRecordsWithoutPaging: 1234
jobs:
- jobSummary:
totalNumOfFiles: 0
jobId: 8035
destClientName: BOSDEKARLSSP010
destinationClient:
clientId: 10
clientName: BOSDEKARLSSP010
- jobSummary:
totalNumOfFiles: 0
jobId: 9629
destClientName: BOSDEKARLSSP006
destinationClient:
clientId: 11
clientName: BOSDEKARLSSP006
tasks:
- name: get job ID
set_fact:
job_id: "{{ results|json_query('jobs[?jobSummary.destClientName == `{}`]|[0].jobSummary.jobId'.format(target_vm)) }}"
- debug:
var: job_id
Update re: your comment
The {} in the expression is a Python string formatting sequence that
is filled in by the call to .format(target_vm). In Python, the
expression:
'The quick brown {} jumped over the lazy {}.'.format('fox', 'dog')
Would evaluate to:
The quick brown fox jumped over the lazy dog.
And that's exactly what we're doing in that set_fact expression. I
could instead have written:
job_id: "{{ results|json_query('jobs[?jobSummary.destClientName == `' ~ target_vm ~ '`]|[0].jobSummary.jobId') }}"
(Where ~ is the Jinja stringifying concatenation operator)
I've looked at a few other related questions and I still can't seem to find what I'm looking for. This is an example JSON payload being sent to an API I'm writing:
{
"publishType": "Permitable",
"electricalPanelCapacity": 0.0,
"roofConstruction": "Standard/Pitched",
"roofType": "Composition Shingle",
"systemConstraint": "None",
"addedCapacity": 0.0,
"isElectricalUpgradeRequired": false,
"cadCompletedBy": "94039",
"cadCompletedDate": "2017-02-01T02:18:15Z",
"totalSunhourDeficit": 5.0,
"designedSavings": 5.0,
"isDesignedWithinTolerance": "N/A",
"energyProduction": {
"january": 322.40753170051255,
"february": 480.61501312589826,
"march": 695.35215022905118,
"april": 664.506907341219,
"may": 877.53769491124172,
"june": 785.56924358327,
"july": 782.64347308783363,
"august": 760.1123565793057,
"september": 574.67050827435878,
"october": 524.53797441350321,
"november": 324.31132291046379,
"december": 280.46921069200033
},
"roofSections": [{
"name": "North East Roof 4",
"roofType": "Composition Shingle",
"azimuth": 55.678664773137086,
"tilt": 15.0,
"solmetricEstimate": 510.42831656979456,
"shadingLoss": 14.0,
"systemRating": 580.0,
"sunHours": 0.88004882167205956,
"moduleCount": 2,
"modules": [{
"moduleRating": 290.0,
"isovaPartNumber": "CDS-MON-007070",
"partCount": 2
}]
}, {
"name": "South West Roof 3",
"roofType": "Composition Shingle",
"azimuth": 235.67866481720722,
"tilt": 38.0,
"solmetricEstimate": 3649.1643776261653,
"shadingLoss": 59.0,
"systemRating": 5220.0,
"sunHours": 0.69907363556056812,
"moduleCount": 18,
"modules": [{
"moduleRating": 290.0,
"isovaPartNumber": "CDS-MON-007070",
"partCount": 18
}]
}, {
"name": "South East Roof",
"roofType": "Composition Shingle",
"azimuth": 145.67866477313709,
"tilt": 19.0,
"solmetricEstimate": 2913.1406926526984,
"shadingLoss": 31.0,
"systemRating": 2900.0,
"sunHours": 1.0045312733285168,
"moduleCount": 10,
"modules": [{
"moduleRating": 290.0,
"isovaPartNumber": "CDS-MON-007070",
"partCount": 10
}]
}],
"SystemConfiguration": {
"inverters": [{
"isovaPartNumber": "ENP-INV-007182",
"partCount": 30
}]
}
}
Describing all the beginning parameters was easy.
/post/new-cad/{serviceNumber}:
post:
summary: Publish a new CAD record.
description: Creates a new CAD record under the provided service number and returns the name of the new CAD record, the unique SF ID, and the deep link URL for Salesforce.
parameters:
- name: serviceNumber
in: path
description: The service number for the solar project you're interested in publishing to.
required: true
type: string
- name: publishType
in: formData
description: The type of CAD record to publish (Proposal, Permitable, or AsBuilt).
required: true
type: string
- name: electricalPanelCapacity
in: formData
required: true
type: number
format: double
- name: roofConstruction
in: formData
description: New, Flat Roof, Open Beam, Standard/Pitched
required: true
type: string
- name: roofType
in: formData
description: Composition Shingle, Membrane (Rubber, TPO, PVC, EPDM), Metal - Corrugated (S-Curve), Metal - Standing Seam, Metal - Trapezoidal, Multi Roof Type, Rolled Comp, Silicone, Tar & Gravel, Tile - Flat, Tile - S-Curve, or Tile - W-Curve
type: string
- name: systemConstraint
in: formData
description: Usage, None, Roof, Electrical, Shading, or 10kW Max
required: true
type: string
- name: addedCapacity
in: formData
required: true
type: number
format: double
- name: isElectricalUpgradeRequired
in: formData
type: boolean
- name: cadCompletedBy
in: formData
description: Employee ID of record author.
type: number
required: true
- name: cadCompletedDate
in: formData
description: The date the CAD record was completed.
type: string
format: date
required: true
- name: totalSunhourDeficit
in: formData
type: number
format: double
- name: designedSavings
in: formData
type: number
format: double
- name: isDesignedWithinTolerance
in: formData
type: string
description: Yes, No, or N/A
And yields the expected result in Swagger-UI:
But now I'm struggling with the last parts of the example JSON payload above. I'm unsure how to express the energyProduction key which is an object with a key for each month of the year. I'm also unsure how to describe roofSections which is an array of objects and systemConfiguration which is an object with a property inverters whose value is an array of objects.
I'm going over the swagger documentation quite a bit but I'm still pretty confused and hoping maybe someone here can explain things a little better to me.
I figured it out. Turns out formData is not what I should have been using for my parameters. Instead I needed to use body and define the structure of the JSON that would populate the body. Here is the completed design file using a body parameter with an object schema and describes all the nested objects and arrays as well.
/new-cad/{serviceNumber}:
post:
summary: Publish a new CAD record.
description: Creates a new CAD record under the provided service number and returns the name of the new CAD record, the unique SF ID, and the deep link URL for Salesforce.
parameters:
- name: serviceNumber
in: path
description: The service number for the solar project you're interested in publishing to.
required: true
type: string
- name: cadData
in: body
description: A JSON payload containing the data required to publish a new CAD record.
required: true
schema:
type: object
properties:
publishType:
type: string
default: "Proposal"
enum: ["Proposal","Permitable","AsBuilt"]
electricalPanelCapacity:
type: number
roofConstruction:
type: string
default: "New"
enum: ["New","Flat Roof","Open Beam","Standard/Pitched"]
roofType:
type: string
enum: ["Composition Shingle","Membrane (Rubber, TPO, PVC, EPDM)","Metal - Corrugated (S-Curve)","Metal - Standing Seam","Metal - Trapezoidal","Multi Roof Type","Rolled Comp","Silicone","Tar & Gravel","Tile - Flat","Tile - S-Curve","Tile - W-Curve"]
systemConstraint:
type: string
default: "None"
enum: ["None","Usage","Roof","Electrical","Shading","10kW Max"]
addedCapacity:
type: number
default: 0
isElectricalUpgradeRequired:
type: boolean
cadCompletedBy:
type: string
cadCompletedDate:
type: string
totalSunhourDeficit:
type: number
designedSavings:
type: number
isDesignedWithinTolerance:
type: string
default: "N/A"
enum: ["N/A","Yes","No"]
energyProduction:
type: object
properties:
january:
type: number
february:
type: number
march:
type: number
april:
type: number
may:
type: number
june:
type: number
july:
type: number
august:
type: number
september:
type: number
october:
type: number
november:
type: number
december:
type: number
roofSections:
type: array
items:
type: object
properties:
name:
type: string
roofType:
type: string
enum: ["Composition Shingle","Membrane (Rubber, TPO, PVC, EPDM)","Metal - Corrugated (S-Curve)","Metal - Standing Seam","Metal - Trapezoidal","Multi Roof Type","Rolled Comp","Silicone","Tar & Gravel","Tile - Flat","Tile - S-Curve","Tile - W-Curve"]
azimuth:
type: number
tilt:
type: number
solmetricEstimate:
type: number
shadingLoss:
type: number
systemRating:
type: number
sunHours:
type: number
moduleCount:
type: integer
modules:
type: array
items:
type: object
properties:
moduleRating:
type: number
isovaPartNumber:
type: string
partCount:
type: integer
systemConfiguration:
type: object
properties:
inverters:
type: array
items:
type: object
properties:
isovaPartNumber:
type: string
partCount:
type: integer
tags:
- NEW-CAD
responses:
200:
description: CAD record created successfully.
schema:
type: object
properties:
cadName:
type: string
sfId:
type: string
sfUrl:
type: string
examples:
cadName: some name
sfId: a1o4c0000000GGAQA2
sfUrl: http://some-url-to-nowhere.com
204:
description: No project could be found for the given service number.
500:
description: Unexpected error. Most likely while communicating with Salesforce.
schema:
type: string
So now I can still get the serviceNumber from the path while everything else comes in the request body. One thing to keep in mind here is that you cannot use all the same Swagger Data Types. For example I tried to use double for one of the properties and Swagger complained that it couldn't parse type double. I was very confused until I finally found the section of the docs describing the difference between formData parameters and a body parameter (of which you can only have one, because it describes the entire request body). Basically you can only use data types that are supported by the JSON schema.
Swagger-UI now shows a single textarea instead of multiple input fields for each parameter. Not as pretty but it works great. You can click the "Example Value" box on the right and it places a predefined JSON template in the textarea for you so you can just fill in the values.
If you are just learning Swagger like I am I hope this helps!