Trying to pass parameters from Master to child template - json

I'm trying to pass list parameters from master to child template, however I'm running into two errors.. These are my current parameters on the master template.
"Parameters": {
"ELBSubnets": {
"Default": "subnet-5d8fea67,subnet-3e35cf15",
"Type": "CommaDelimitedList"
},
"LCKeyPair": {
"Default": "key-master",
"Type": "String"
},
"LCSecurityGroups": {
"Default": "sg-10a15c74,sg-880e5fec",
"Type": "CommaDelimitedList"
}
},
They are being referenced in this method on the same template when passing on to the child template.
"ChildTempate1": {
"Properties": {
"Parameters": {
"ELBSubnets": {
"Ref": "ELBSubnets"
},
"KeyPair": {
"Ref": "LCKeyPair"
},
"LCSecurityGroups": {
"Ref": "LCSecurityGroups"
}
},
On the child template, they are declared exactly the same.
"Parameters": {
"ELBSubnets": {
"Type": "CommaDelimitedList"
},
"LCKeyPair": {
"Type": "String"
},
"LCSecurityGroups": {
"Type": "CommaDelimitedList"
}
},
And they're being referenced in this method in the child template.
"KeyName": {
"Ref": "LCKeyPair"
},
"SecurityGroups": {
"Fn::Join": [
",",
[
{
"Ref": "LCSecurityGroups"
}
]
]
}
},
This is another part of the template.
"Subnets": {
"Fn::Join": [
",",
[
{
"Ref": "ELBSubnets"
}
]
]
}
},
When I attempt to use the fn::join on the master template, it says
"Template validation error: Template error: every Fn::Join object requires two parameters, (1) a string delimiter and (2) a list of strings to be joined or a function that returns a list of strings (such as Fn::GetAZs) to be joined."
When I don't use fn::join on the master template the error is
Value of property Parameters must be an object with String (or simple type) properties
Regardless of whether I have fn::join on the same parameters in the child template.
Both templates can be found here: https://github.com/slimg00dy/Troposphere-CloudformationTests

The issue is that when using Ref on a CommaDelimitedList, you pass a list, so it passes "[a,b,c]" rather than "a,b,c"
So on the master template I've used Join(",") to pass lists, and Join(" ") to pass strings. This way they're Referenced as "a,b,c" and " String"
On the child template I've set the parameters as CommaDelimitedLists for the lists and String from the String. This is because of the way they're being passed from the master template.
I then used Ref on the Child template without the use of Join() on the child template. This created the CommaDelimitedLists into Lists and the Strings that were joined with Join(" ") passed as strings.
Here is my parameter declaration on the master Template.
"Parameters": {
"ELBSubnets": {
"Default": "subnet-5d8fea67,subnet-3e35cf15",
"Type": "CommaDelimitedList"
},
"LCKeyPair": {
"Default": "key-master",
"Type": "CommaDelimitedList"
},
"LCSecurityGroups": {
"Default": "sg-10a15c74,sg-880e5fec",
"Type": "CommaDelimitedList"
}
},
Here is how I used Join(","), Join(" ") and Ref. Since Ref converts the CommaDelimitedLists into Lists, I used Join(",") to keep them as CommaDelimitedLists and pass them on to the child template.
As for the KeyPair String, I made sure that it was declared as a CommaDelimitedList on the parent template and joined with Join(" "), this effectively made it into a string when referencing to the child template.
"Parameters": {
"ELBSubnets": {
"Fn::Join": [
",",
{
"Ref": "ELBSubnets"
}
]
},
"LCKeyPair": {
"Fn::Join": [
" ",
{
"Ref": "LCKeyPair"
}
]
},
"LCSecurityGroups": {
"Fn::Join": [
",",
{
"Ref": "LCSecurityGroups"
}
]
}
},
On the child template, they're declared like so.
"Parameters": {
"ELBSubnets": {
"Type": "CommaDelimitedList"
},
"LCKeyPair": {
"Type": "String"
},
"LCSecurityGroups": {
"Type": "CommaDelimitedList"
}
},
And they are all referenced normally without the use of Join on the child template.
Subnets": {
"Ref": "ELBSubnets"
}
There could have been many different ways to do this. I could have had the joins on the child template rather than the parent template. However I prefer to keep one template complicated and the rest as clean as possible. Hope this helps others down the line.
I also should have been able to pass the list as a list on the child template, however then I received the error "Unknown parameter type: List"

CommaDelimtedList can only be parsed once, so in the parent template, you need to set the type as "String", then in the child template, keep it as CommaDelimitedList, as before.
The problem is happening, because CloudFormation is parsing the CommaDelimitedList in the parent template, and then it's trying to parse it again in the child template.

So my previous answer was a little confusing - but I think the basics still stand. In the end you shouldn't have to Join() a CommaDelimitedList when it is used in an argument that expects a list. I just looked at your most recent updates to the repo you shared, and I think you're on the right track. If you need more help, let me know!

Related

jq output is empty when tag name does not exist

When I run the jq command to parse a json document from the amazon cli I have the following problem.
I’m parsing through the IP address and a tag called "Enviroment". The enviroment tag in the instance does not exist therefore it does not throw me any result.
Here's an example of the relevant output returned by the AWS CLI
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
I’m running the following command
aws ec2 describe-instances --filters "Name=tag:Name,Values=Balance-OTA-SS_a" | jq -c '.Reservations[].Instances[] | ({IP: .PrivateIpAddress, Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)})'
## output
empty
How do I show the IP address in the output of the command even if the enviroment tag does not exist?
Regards,
Let's assume this input:
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
This is the format returned by describe-instances, but with all the irrelevant fields removed.
Note that tags is always a list of objects, each of which has a Key and a Value. This format is perfect for from_entries, which can transform this list of tags into a convenient mapping object. Try this:
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags|from_entries.Environment)
}
{"IP":"10.0.0.1","Ambiente":"alpha"}
{"IP":"10.0.0.2","Ambiente":null}
That answers how to do it. But you probably want to understand why your approach didn't work.
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)
}
The .[] filter you're using on the tags can return zero or multiple results. Similarly, the select filter can eliminate some or all items. When you apply this inside an object constructor (the expression from { to }), you're causing that whole object to be created a variable number of times. You need to be very careful where you use these filters, because often that's not what you want at all. Often you instead want to do one of the following:
Wrap the expression that returns multiple results in an array constructor [ ... ]. That way instead of outputting the parent object potentially zero or multiple times, you output it once containing an array that potentially has zero or multiple items. E.g.
[.Tags[]|select(.Key=="Environment")]
Apply map to the array to keep it an array but process its contents, e.g.
.Tags|map(select(.Key=="Environment"))
Apply first(expr) to capture only the first value emitted by the expression. If the expression might emit zero items, you can use the comma operator to provide a default, e.g.
first((.Tags[]|select(.Key=="Environment")),null)
Apply some other array-level function, such as from_entries.
.Tags|from_entries.Environment
You can either use an if ... then ... else ... end construct, or //. For example:
.Reservations[].Instances[]
| {IP: .PrivateIpAddress} +
({Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)}
// null)

Can a JSON schema specify required properties in object OR array?

I am struggling to understand if it's possible to write a json schema that requires certain properties, but also allows those properties to be in different areas of the json file (e.g. a property value can be in the main top-level object OR it can be in an array - it just needs to be somewhere).
For example, I have some devices that collect multiple temperature records over the course of a few hours and send the records in batches. However, some of the devices send the software version once in the main object, while others send the software version along with each hourly temperature record (inside a "records" array).
Example 1 (swversion sent once in main object):
{
"name": "device1",
"swversion": "1.3.abc2",
"records": [
{
"time": "10am",
"temp": 2
},
{
"time": "11am",
"temp": 4
}
]
}
Example 2 (swversion sent inside "records" array):
{
"name": "device1",
"records": [
{
"time": "10am",
"temp": 2,
"swversion": "1.3.abc2"
},
{
"time": "11am",
"temp": 4,
"swversion": "1.3.abc2"
}
]
}
Using these examples, I would like to write my schema definition as follows (the first two bullets are easy, the last one is where I'm struggling):
Main object requires name property and records array
records array can contain objects where time and temp would be required
swversion is required somewhere (could be in the main object or inside records array)
Is there a feature I'm missing in json-schema that enforces required properties, yet allows the flexibility for said properties to be anywhere (e.g. within an object OR an array), as long as they are present somewhere?
The anyOf keyword is a boolean OR operation. At least one of the schemas must pass for the keyword to pass. The first schema requires that the "swversion" property is present at the top level. The second schema requires that the "swversion" property is required in each of the items in the "record" array.
{
... define the easy stuff here, then ...,
"anyOf": [
{ "required": ["swversion"] },
{
"properties": {
"records": {
{ "items": { "required": ["swversion"] } }
}
}
}
]
}
In this example, "swversion" could appear in both places. If you only want to ensure that it only appear in one place (top level or items), you can use oneOf instead of anyOf.
Is there a feature .. that allows the flexibility for said properties to be anywhere
Not directly, but it's not difficult to express this. You can define the structure of "swversion" itself in a definition that is re-used via a reference.
In pseudocode, that would be:
any of:
the main object contains a "swversion" property,
all the items under "records" contain a "swversion" property
In code:
{
"$defs": {
"swversion": {
"type": "string",
.. other constraints?
}
},
"type": "object",
"properties": {
... other property definitions ...,
"records": {
"items": {
"type": "object",
... other definitions for the mandatory portion of records ...
}
}
},
"anyOf": [
{
"$comment": "swversion is a member of the main object",
"required": [ "swversion" ],
"properties": {
"swversion": {
"$ref": "#/$defs/swversion"
}
},
{
"$comment": "swversion is a member of all the items under the records property",
"properties": {
"records": {
"items": {
"type": "object",
"required": [ "swversion" ],
"properties": {
"swversion": {
"$ref": "#/$defs/swversion"
}
}
}
}
}
}
],
}
Note that if you are using JSON Schema version draft7 or earlier, change $defs to definitions.

Conditionally determine the required-ness of a field

I need to refer to a sub-schema of certain property (Kind in the example) from a different property in the schema, and then enforce some more conditions on it. Important thing to note is I cannot make those changes where I've defined Kind, I need to refer to it from some other property and then add conditionals on top of it.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"Kind": {
"$id": "#/properties/Kind",
"type": "string",
"enum": [
"Foo",
"Bar"
]
}
},
"allOf": [
{
"if": {
"$ref": "#/properties/Kind",
"const": "Foo"
},
"then": {
"required": [
"MyField"
]
}
}
]
}
A json object like below should fail the validation, because MyField property is absent
{
"Kind": "Foo"
}
I don't want the following solution, since this is just a simplified version and ultimately I want to refer to Kind value from another property. If I do following, then #/properties/Kind is interpreted relative to where I refer Kind so it doesn't refer to the Kind at the top level. I want a solution which uses the $ref and $id keywords.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"Kind": {
"$id": "#/properties/Kind",
"type": "string",
"enum": [
"Foo",
"Bar"
]
}
},
"allOf": [
{
"if": {
"properties": {"Kind":{
"const":"Foo"
}}
},
"then": {
"required": [
"MyField"
]
}
}
]
}
To summarize, let's say I've following JSON structure. The last allOf statement is what I need to add.
- Kind ( enum of One,Two)
- Other
- MyField
- ConditionField
- allOf ( which enforces the required-ness of MyField based on ConditionField)
- allOf ( MyField should be not-required if Kind is One)
[ To add this last conditional, I need to reference the value of Kind.
I'm hoping providing $id to Kind and referring to it with $ref should be my approach,
which doesn't seem to be working]
To summarize even further, I would get my answer if we're able to get the first snippet work using $id and $ref.
There seem to be some misunderstandings that are making it difficult to fully understand the problem here, but one part of the edited question makes enough sense that I think I can start things off and we can iterate on the answer as necessary.
Let's start with some of the things that don't make sense in hopes that it helps clarify possible misunderstandings.
$ref can't change the behavior of a schema. If you can't do something without $ref, then you can't make the schema behave another way by introducing $ref. The only exception to that rule is recursive schemas, which would require an infinitely large and repeating schema without using $ref.
I'm not sure what you are trying to get from $id, but it's pretty safe to say you don't need it for this. In any case, the $id used the question is invalid. An anchor can not have a / in it. Even if it was valid, it would be redundant because you can reference that location with the same JSON Pointer without an anchor.
MyField should be not-required if Kind is One
I'm not sure if "not-required" means forbidden or optional. Everything is optional by default in JSON Schema, so if you meant optional, there is nothing to do here. Therefore, I'll assume for now that you mean forbidden. Here's what that would look like.
{
"type": "object",
"properties": {
"Kind": { "enum": ["One", "Two"] },
"Other": {
"type": "object",
"properties": {
"MyField": {}
}
}
},
"allOf": [
{
"if": {
"properties": {
"Kind": { "const": "One" }
},
"required": ["Kind"]
},
"then": {
"properties": {
"Other": {
"not": { "required": ["MyField"] }
}
}
}
}
]
}

Validate each JSON node with different JSON schema

Im trying to make a system monitor, which is highly customisable by user. This customization is achieved by using JSON file for modeling look of system monitor. The JSON could look like this.
{
"_": "WINDOW",
"name": "myWindow",
"children": [
{
"_": "CPU",
"name": "cpuMonitor",
"freq_Unit": "MHZ"
},
{
"_": "NETWORK",
"name": "network",
"unit": "Kb/s"
},
{
"_": "DISK",
"name": "disk"
}
],
"background": "red"
}
As you can see, each object coresponds to this schema.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"name":"Component",
"type": "object",
"properties":{
"_": {
"type": "string"
},
"name":{
"type":"string"
},
"childern":{
"type":"array"
}
},
"required": ["_","name"]
}
But each component has also its own schema definition. I'd like to parse whole JSON and validate each node for different schema (first if its component and then to its corresponding schema).
I had look at rapidJson and other libraries, but I didnt find solution for validating nodes for different schema. Do you know any library which could do that? Or is it even possible to validate JSON in this way?
All feedback on how to solve this will be appreciated.
Edit: Corrected schema :(
There's a simple approach involved with that, use the oneOf pattern declaration to specify the layout of the array elements. Inside these nested declarations, you specify the fixed identifier (probably the content of your _ field) as a constant, so that there is only one nested schema matching each of your panel types.
Notes:
I had to specify the constant type identifier using the enum specifier because the regular constant specifier didn't work with the library I was using. This may also have been an oversight in the revision of the specification that it was based on.
A different approach is to split the the validation steps. You simply verify that the elements of the array are objects and that they have a string field _ containing one of the supported types. When iterating over the array, you then validate each field individually according to its _ field.
In addition to Ulrich's answer, here's an example of what I'd do:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Component",
"type": "object",
"definitions": {
"base": {
"properties": {
"name": { "type": "string" },
"children": {
"type": "array",
"items": { "$ref": "#" }
}
},
"required": [ "_", "name" ]
},
"cpu": {
"properties": {
"_": { "const": "CPU" },
"freq_Unit": "MHZ"
}
},
"network": {
"properties": {
"_": { "const": "NETWORK" },
"unit": "Kb/s"
}
},
"disk": {
"properties": {
"_": { "const": "DISK" }
}
},
"window": {
"properties": {
"_": { "const": "WINDOW" },
"background": { "enum": [ "red", "orange", "yellow", ... ] }
}
}
},
"allOf": [
{ "$ref": "#/definitions/base" },
{
"oneOf": [
{ "$ref": "#/definitions/cpu" },
{ "$ref": "#/definitions/network" },
{ "$ref": "#/definitions/disk" },
{ "$ref": "#/definitions/window" }
]
}
]
}
First, we require that any instance MUST adhere to base which declares _ and name as required properties. Additionally, we declare a children array property that requires all items also match this schema (giving us a recursive behavior). This doesn't really do much except that it allows us to declare these things in one place instead of having to declare them in the other three definitions.
(Note that we don't declare _ in the properties list. This means that any value will pass for this portion of the schema. We clean it up in the next part. If you want to ensure that future components are declared with strings, then you can add a "type": "string" requirement to that property, but I don't feel it's necessary unless others are authoring those components.)
Second, we declare each of our specific types as separate definitions, using the const keyword to isolate the one we want. This construct is analogous to a switch (or case) statement. If the instance doesn't match one of these explicit options, it fails. If it's missing one of the required base properties, it fails.
This will get you where you want to be.
To take it further, there are two more things you can do:
Add required to the other definitions to say that the specific properties are also required (e.g. freq_Unit for the cpu definition).
Declare each of the definitions in separate files. This would allow you to add a new definition by simply adding a new file and referencing it in the main schema. In my opinion, it's a bit cleaner. Some people prefer to have it all in one file, though.

JSON schema draft4 VS JSON schema draft3

What are the features present in the schema draft 4 that are not in the JSON schema draft 3 produced by IETF ?
From the change logs:
New keywords
anyOf (match at least one schema in the schema array),
allOf (match all schemas in the schema array),
oneOf (match exactly one schema in the schema array),
not (do not match the schema),
multipleOf (replaces divisibleBy),
minProperties and maxProperties (the minimum and maximum number of members in an object instance),
definitions (standardized container for inlined subschemas).
Removed:
disallow
extends
divisbleBy
Changed in functionality:
Type
When the value is an array, schemas are no longer allowed as elements. Also, the array must have at least one element.
Before
{
"type": [ "string", { "other": "schema" } ]
}
Now
{
"anyOf": [
{ "type": "string" },
{ "other": "schema" }
]
}
Required
Before, it was an attribute of subschemas in properties. It is now a first level keyword playing the same role, and has a string array as an argument.
Before
{
"properties": {
"p": {
"type": "string",
"required": true
},
"q": {
"type": "string",
"required": true
}
}
}
Now
{
"properties": {
"p": { "type": "string" },
"q": { "type": "string" }
},
"required": [ "p", "q" ]
}
Dependencies
A single string in a property dependency is no longer allowed, only arrays are allowed
Before
{
"dependencies": { "a": "b" }
}
Now
{
"dependencies": { "a": [ "b" ] }
}
If you're interested in a deep dive, you can review a diff between the two drafts on the IETF site.
However, if you're looking for a simpler summary of changes, Geraint Luff and Francis Galiegue created a changelog page on the project's github wiki that lists the changes, additions, and removals.