AWS CloudFormation - using !Ref inside !Sub - json

I'm writing AWS CloudFormation template (using yaml) which creates AWS Service Catalog Product.
I'm getting the template for the product using parameter S3FilePath which has a value like the above path: https://bucket.s3-eu-west-1.amazonaws.com/template.yml.
The URL to the file needs to be send in a JSON format as shown here (this example works):
Resources:
Type: AWS::ServiceCatalog::CloudFormationProduct
Properties:
Description: Example Product
Distributor: xyz
Name: ExampleProduct
Owner: xyz
ProvisioningArtifactParameters:
- Description: Example Product
Info: { "LoadTemplateFromURL": "https://bucket.s3-eu-west-1.amazonaws.com/template.yml" }
Name: Version1
I tried to replace the URL using !Sub and !Ref as shown below:
Parameters:
S3FilePath:
Type: String
Description: file name
Resources:
Type: AWS::ServiceCatalog::CloudFormationProduct
Properties:
Description: Example Product
Distributor: xyz
Name: ExampleProduct
Owner: xyz
ProvisioningArtifactParameters:
- Description: Example Product
Info: !Sub
- '{ "LoadTemplateFromURL": "${FILEPATH}" }'
- {FILEPATH: !Ref S3FilePath}
Name: Version1
But the CloudFormation stack fails with the error: "invalid input".
I guess I am building the JSON in a wrong way, I tried to use \ before each ' " ' but it didn't help either and I couldn't find an example which explain how to build this correctly. There is no problem with the S3FilePath parameter.
Can you please advice how to use the !Sub and !Ref correctly to build the JSON? Thanks.

Here is an example: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html#w2ab1c25c28c59c11
Despite the documentation saying the Info parameter is JSON, the example shows just a name/value pair (Map): https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-servicecatalog-cloudformationproduct-provisioningartifactproperties.html
Try formatting your string as
Info: !Sub
- "LoadTemplateFromURL": "${FILEPATH}"
- {FILEPATH: !Ref S3FilePath}

You can reference any Parameters or LogicalResourceId directly inside a !Sub like so:
ProvisioningArtifactParameters:
- Description: Example Product
Info: !Sub '{ "LoadTemplateFromURL": "${S3FilePath}" }'
Name: Version1
This should work totally fine. The way you were doing substitutions is useful when you want to use conditions and/or mapping inside a !Sub.

I think it should be simply:
ProvisioningArtifactParameters:
- Description: Example Product
Info:
LoadTemplateFromURL: !Ref S3FilePath
Name: Version1
This is at least what I have in my own AWS::ServiceCatalog::CloudFormationProduct templates.
ProvisioningArtifactParameters:
- DisableTemplateValidation: false
Info:
LoadTemplateFromURL: !Ref S3FilePath

Related

Json issue in Cloudformation for glue workflow

I have created below cloudformation yaml for glue workflow.
Parameter GlWFproperties defined as type string
Type: AWS::Glue::Workflow
Properties:
DefaultRunProperties: !Ref GlWFproperties
Description: !Ref GlWFdescription
MaxConcurrentRuns: !Ref GlWFMaxConcurrentRuns
Name: !Ref GlWFname
I am passing the parameters to this template while deploying it.
If i deploy this template without DefaultRunProperties it works.But when i pass values to "DefaultRunProperties" it fails with error "Internal Failure".
DefaultRunProperties is defined as Type JSON in aws documentation.
Values i am passing to this is as
{ "sql": "inno", "format": "aro"}
also tried
"{ \"sql\": \"inno\", \"format\": \"aro\"}"

Parse byte array for yaml value - kubebuilder

We are using kubebuilder to build our custom controller, the problem is we are not able to parse raw data as it comes empty when you apply the file.
I’ve created very minimal example which describe the issue.
apiVersion: mygroup.test.com/v1alpha1
kind: Rawtest
metadata:
name: rawtest-sample
spec:
system:
type: test
provider:
type: aws
infrastructureConfig:
kind: InfrastructureConfig
apiVersion: v1alpha1
networks:
vpc:
cidr: aaa
zones:
- name: abc
internal: 123
workers:
- name: myworker
machine:
type: "mt"
image:
name: name1
version: "2"
maximum: 2
minimum: 1
maxUnavailable: 0
volume:
type: a1
size: 20Gi
zones:
- zone1
In runtime I was able to get the the spec.system.type value=test and spec.system.provider.type value="aws", however I wasn’t able to get all the data under the infrastructureConfig: (line 10) any idea how can I overcome this ?
I’ve created this very simple project to demonstrate the issue ,
See the api/type folder, after getting the reconcile object (after apply the config/sample/ file ,you see that the infrastructureconfig and all related data are
https://github.com/JennyMet/
Here is the code which is trying to read the raw value
https://github.com/JennyMet/kuberaw/blob/master/controllers/rawtest_controller.go#L57
&rawtest should contain all the data
please see the type
https://github.com/JennyMet/kuberaw/blob/master/api/v1alpha1/rawtest_types.go#L32
raw type
https://github.com/gardener/gardener/blob/bf32324d9d1a366d8a0a7514956dc39c2f22f7b7/pkg/apis/core/v1beta1/types_shoot.go#L945
https://github.com/gardener/gardener/blob/bf32324d9d1a366d8a0a7514956dc39c2f22f7b7/pkg/apis/core/types_shoot.go#L774
https://github.com/gardener/gardener/blob/bf32324d9d1a366d8a0a7514956dc39c2f22f7b7/vendor/k8s.io/apimachinery/pkg/runtime/types.go#L94:6
I need a way to make it work in the kubebuilder, as while I apply the file I dont get the values in debug ...
debug pic
if something is missing please let me know.
I did the following
visit a site such as https://yaml.to-go.online/ convert the yaml to a go Struct
make a struct
t := AutoGenerated{}
I assume that the yaml data is loaded into memory as []byte
using "gopkg.in/yaml.v2", Unmarshall the yaml to the struct
err := yaml.Unmarshal([]byte(data), &t)
the infrastructureconfig field is available as t.Spec.System.Provider.InfrastructureConfig
see https://play.golang.org/p/syx8v7gAmDH

Invalid type for io.k8s.api.core.v1.ConfigMapEnvSource got "array" expected "map"

I've a kubernetes cronjob manifest file.In that file I've defined enviornment variables.I'm generating yaml using a shell script but while using the yaml using kubectl create -f. I'm getting the following validation error
error validating "cron.yaml": error validating data: [ValidationError(CronJob.spec.jobTemplate.spec.template.spec.containers[0].envFrom[0].configMapRef): invalid type for io.k8s.api.core.v1.ConfigMapEnvSource: got "array", expected "map".
Can anyone suggest me how to resolve this?
You have a mistake in the syntax.
There are two approaches, using valueFrom for individual values or envFrom for multiple values.
valueFrom is used inside the env attribute.valueFrom will inject the value of a a key from the referenced configMap.
spec:
template:
spec:
containers:
- name: ad-sync
image: foo.azurecr.io/foobar/ad-sync
command: ["dotnet", "AdSyncService.dll"]
args: []
env:
- name: AdSyncService
valueFrom:
configMapKeyRef:
name: ad-sync-service-configmap
key: log_level
envFrom is used direct inside the container attribute.envFrom will inject All configMap keys as environment variables
spec:
template:
spec:
containers:
- name: ad-sync
image: foo.azurecr.io/foobar/ad-sync
command: ["dotnet", "AdSyncService.dll"]
envFrom:
- configMapRef:
name: ad-sync-service-configmap

How to embed JSON string as the value in a Kubernetes Secret

Part of our orchestration uses envsubst to update a YAML template file with our desired values.
envsubst < "${SECRET_TEMPLATE}" | kubectl apply -f -
The value for our keyword config is a JSON string:
data=$(jq -c . ${JSON_FILE})
This results in YAML that looks like this (trimmed for brevity):
apiVersion: v1
kind: Secret
metadata:
name: reporting-config
type: Opaque
data:
config: {"database": "foo"}
This apparently worked in some earlier versions of Kube, I wanna say 1.8. Anyways, we are running 1.15 and now kubectl interprets this as a map type and complains:
error: error validating "STDIN": error validating data: ValidationError(Secret.data.config): invalid type for io.k8s.api.core.v1.Secret.data: got "map", expected "string"; if you choose to ignore these errors, turn validation off with --validate=false
Is there a trick to doing this now. I've played around with quoting and various places, escaping quotes, and all that jazz and nada.
* update 1 *
Using stringData still results in the same error:
apiVersion: v1
kind: Secret
metadata:
name: monsoon-storage-reporting-config
type: Opaque
stringData:
config: {"database": "foo"}
error: error validating "STDIN": error validating data: ValidationError(Secret.stringData.config): invalid type for io.k8s.api.core.v1.Secret.stringData: got "map", expected "string"; if you choose to ignore these errors, turn validation off with --validate=false
You may use stringData as below. Note the pipe after stringData:
apiVersion: v1
kind: Secret
metadata:
name: monsoon-storage-reporting-config
type: Opaque
stringData: |
config: {"database": "foo"}
I had to base64 encode the value
$ echo {"database": "foo"} | base64
e2RhdGFiYXNlOiBmb299Cg==
and then use the base64 encoded value in the data: field
apiVersion: v1
kind: Secret
metadata:
name: reporting-config
type: Opaque
data:
config: e2RhdGFiYXNlOiBmb299Cg==
Also note this on base64 encoding:
When using the base64 utility on Darwin/macOS users should avoid using the -b option to split long lines. Conversely Linux users should add the option -w 0 to base64 commands or the pipeline base64 | tr -d '\n' if -w option is not available.

Convert Json reponse in swagger with description

I try to specify the response from GET method with Swagger Editor.
But when I look in Swagger UI, the response JSON doesn't show.
My declaration swagger :
/clients/{id}:
get:
consumes:
- application/hal+json
produces:
- application/hal+json
# This is array of GET operation parameters:
parameters:
# An example parameter that is in query and is required
- name: id
in: path
required: true
type: string
# Expected responses for this operation:
responses:
# Response code
200:
description: Succes
# A schema describing your response object.
# Use JSON Schema format
schema:
example:
application/json: |
- {"produits":[{"idOffre":{"codeBanque":"038","identifiant":"123"},"idProduit":{"codeBanque":"061","identifiant":"123"}},{"idOffre":{"code":"038","identifiant":"123"},"idProduit":{"code":"061","identifiant":"123"}}]}
.....
In Swagger UI , the box where is written Response Class (Status 200) > Model Schema there is an empty json like that -> {}
I'm not sure swagger-ui supports examples, however I fixed some errors in you swagger extract:
example renamed examples
examples at same indentation level as schema (in you example, example is processed as a property of the response schema, I'm not sure this is what you want)
schema must describe response model, complete TODO
Fixed version:
swagger: '2.0'
info:
title: Clients
description: API
version: "1.0.0"
host: api.test.com
schemes:
- https
basePath: /v1
produces:
- application/json
paths:
/clients/{id}:
get:
consumes:
- application/hal+json
produces:
- application/hal+json
parameters:
- name: id
in: path
required: true
type: string
responses:
200:
description: Succes
schema:
type: array
items:
properties:
produits:
type: string
description: TODO
examples:
application/json: |
- {"produits":[{"idOffre":{"codeBanque":"038","identifiant":"123"},"idProduit":{"codeBanque":"061","identifiant":"123"}},{"idOffre":{"code":"038","identifiant":"123"},"idProduit":{"code":"061","identifiant":"123"}}]}