I am currently studying AWS. I'm creating a CloudFormation Template. And I'm stuck with this problem.
I need to convert this JSON code below into YAML and add it to a template.
And the converters in google say
"Unexpected Token : In JSON At Position 10".
"Fn::Join": [
"\n",
[
"#!/bin/bash",
"yum -y update",
"echo 1 > /proc/sys/net/ipv4/ip_forward",
"echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects",
"/sbin/iptables -t nat -A POSTROUTING -o eth0 -s 0.0.0.0/0 -j
MASQUERADE",
"/sbin/iptables-save > /etc/sysconfig/iptables",
"mkdir -p /etc/sysctl.d/",
"cat <<EOF > /etc/sysctl.d/nat.conf",
"net.ipv4.ip_forward = 1",
"net.ipv4.conf.eth0.send_redirects = 0",
"EOF \n"
]
]
I need to then add it to this YAML code below.
NatInstance:
Type: AWS::EC2::Instance
Properties:
InstanceType: t2.micro
ImageId: !Ref AmazonLinuxAMIID
SubnetId: !Ref PublicSubnet1
SecurityGroupIds:
- !Ref AppSecurityGroup
IamInstanceProfile: !Ref InstanceProfile
Tags:
- Key: Name
Value: NAT Instance
Related
I have been installing and using audiowaveform on AWS Elastic Beanstalk via an ebextensions but for some reason I now get an error during the build process:
...
[ 72%] Building CXX object CMakeFiles/audiowaveform_tests.dir/src/WavFileWriter.cpp.o
[ 73%] Building C object CMakeFiles/audiowaveform_tests.dir/src/madlld-1.1p1/bstdfile.c.o
[ 75%] Building CXX object CMakeFiles/audiowaveform_tests.dir/test/FileFormatTest.cpp.o
/opt/src/audiowaveform/audiowaveform-master/test/FileFormatTest.cpp:26:25:
fatal error: gmock/gmock.h: No such file or directory #include "gmock/gmock.h" ^ compilation terminated.
make[2]: *** [CMakeFiles/audiowaveform_tests.dir/test/FileFormatTest.cpp.o] Error 1
make[1]: *** [CMakeFiles/audiowaveform_tests.dir/all] Error 2 make: *** [all] Error 2.
Below is the ebextension that I have been using:
packages:
yum:
make: []
cmake: []
gcc-c++: []
gd-devel: []
boost-devel: []
sources:
/opt/src/audiowaveform: https://github.com/bbc/audiowaveform/archive/master.zip
/opt/src/googletest: https://github.com/google/googletest/archive/release-1.10.0.tar.gz
/opt/src/libmad: https://sourceforge.net/projects/mad/files/libmad/0.15.1b/libmad-0.15.1b.tar.gz/download
/opt/src/libsndfile: http://www.mega-nerd.com/libsndfile/files/libsndfile-1.0.28.tar.gz
/opt/src/libid3tag: https://sourceforge.net/projects/mad/files/libid3tag/0.15.1b/libid3tag-0.15.1b.tar.gz/download
files:
/usr/lib/pkgconfig/mad.pc:
mode: "000755"
owner: root
group: root
content: |
prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
Name: mad
Description: MPEG audio decoder
Requires:
Version: 0.15.1b
Libs: -L${libdir} -lmad
Cflags: -I${includedir}
/etc/ld.so.conf.d/libsndfile:
mode: "000644"
owner: root
group: root
content: |
/usr/lib/
commands:
00-install-libmad-make:
command: "sed -i '/-fforce-mem/d' configure"
cwd: /opt/src/libmad/libmad-0.15.1b/
01-install-libmad-make:
command: "./configure --prefix=/usr && make && make install"
cwd: /opt/src/libmad/libmad-0.15.1b
test: "[ ! -f /usr/local/bin/audiowaveform ]"
02-install-libsndfile-make:
command: "./configure --prefix=/usr --disable-static --docdir=/usr/share/doc/libsndfile-1.0.28 && make && make install"
cwd: /opt/src/libsndfile/libsndfile-1.0.28
test: "[ ! -f /usr/local/bin/audiowaveform ]"
03-install-libid3tag-make:
command: "./configure --prefix=/usr && make && make install"
cwd: /opt/src/libid3tag/libid3tag-0.15.1b
test: "[ ! -f /usr/local/bin/audiowaveform ]"
04-change-mod-audiowaveform:
command: "chmod -R 755 audiowaveform-master"
cwd: /opt/src/audiowaveform
05-audiowaveform-ln-test:
command: "ln -s ../../googletest/googletest-release-1.10.0/googletest ./googletest"
cwd: /opt/src/audiowaveform/audiowaveform-master/
test: "[ ! -L ./googletest ]"
06-audiowaveform-ln-mock:
command: "ln -s ../../googletest/googletest-release-1.10.0/googlemock ./googlemock"
cwd: /opt/src/audiowaveform/audiowaveform-master/
test: "[ ! -L ./googlemock ]"
07-audiowaveform-mkdir-build:
command: "sudo mkdir build"
cwd: /opt/src/audiowaveform/audiowaveform-master/
test: "[ ! -d ./build ]"
08-audiowaveform-cmake:
command: "cmake .."
cwd: /opt/src/audiowaveform/audiowaveform-master/build/
test: "[ ! -f /usr/local/bin/audiowaveform ]"
09-audiowaveform-make:
command: "make"
cwd: /opt/src/audiowaveform/audiowaveform-master/build/
test: "[ ! -f /usr/local/bin/audiowaveform ]"
10-audiowaveform-make-install:
command: "make install"
cwd: /opt/src/audiowaveform/audiowaveform-master/build/
test: "[ ! -f /usr/local/bin/audiowaveform ]"
11-ldconfig:
command: "ldconfig"
Anyone know of something that has changed? So far I haven't found anything.
Do I need to use a different process? I am really not sure what to do next. I have created an new application and environment and tried deploying to it to make sure I haven't messed up the current one but I get the same error.
Well it looks like it is a version issue. This was on our dev environment and found out it was trying to install version 1.5.1 and our production version is on 1.4.2. So I changed the
/opt/src/audiowaveform: https://github.com/bbc/audiowaveform/archive/master.zip
to
/opt/src/audiowaveform: https://github.com/bbc/audiowaveform/archive/refs/tags/1.4.2.zip
and changed all the references to audiowaveform-master to audiowaveform-1.4.2 in the commands below and it seems to work.
Currently we are on a deprecated platform so am transitioning to a supported platform at the same time so I haven't made sure that it works completely but at least it builds it.
I have a JSON with key pairs and I want to access the values from Rundeck Options dynamically during the job execution.
For shell script, we can do a $RD_OPTIONS_<>.
Similarly is there some format I can use in a JSON file?
Just use #option.myoption# in a inline-script step.
You need a tool to use on an inline script step to manipulate JSON files on Rundeck. I made an example using JQ. Alternatively, you can use bash script-fu to reach the same goal.
For example, using this JSON file:
{
"books": [{
"fear_of_the_dark": {
"author": "John Doe",
"genre": "Mistery"
}
}]
}
Update the file with the following jq call:
To test directly in your terminal
jq '.books[].fear_of_the_dark += { "ISBN" : "9999" }' myjson.json
On Rundeck Inline-script
echo "$(jq ''.books[].fear_of_the_dark += { "ISBN" : "#option.isbn#" }'' myjson.json)" > myjson.json
Check how looks on an inline-script job (check here to know how to import the job definition to your Rundeck instance).
- defaultTab: nodes
description: ''
executionEnabled: true
id: d8f1c0e7-a7c6-43d4-91d9-25331cc06560
loglevel: INFO
name: JQTest
nodeFilterEditable: false
options:
- label: isbn number
name: isbn
required: true
plugins:
ExecutionLifecycle: null
scheduleEnabled: true
sequence:
commands:
- description: original file content
exec: cat myjson.json
- description: pass the option and save the content to the json file
fileExtension: .sh
interpreterArgsQuoted: false
script: 'echo "$(jq ''.books[].fear_of_the_dark += { "ISBN" : "#option.isbn#"
}'' myjson.json)" > myjson.json'
scriptInterpreter: /bin/bash
- description: modified file content (after jq)
exec: cat myjson.json
keepgoing: false
strategy: node-first
uuid: d8f1c0e7-a7c6-43d4-91d9-25331cc06560
Finally, check the result.
Here you can check more about executing scripts on Rundeck and here more about the JQ tool.
Hi community,
I have been struggling with an issue in ansible issue for days now.
Everything is executed wihtin a Jenkins pipeline.
The ansible command looks like:
sh """
ansible-playbook ${env.WORKSPACE}/cost-optimization/ansible/manage_dynamo_db.yml \
--extra-vars '{"projectNameDeployConfig":${projectNameDeployConfig},"numberOfReplicas":${numberOfReplicas},"dynamodbtask":${dynamodbtask}}'
"""
And the playbooks is:
playbook.yml
---
- hosts: localhost
vars:
numberOfReplicas: "{{numberOfReplicas}}"
dynamodbtask: "{{dynamodbtask}}"
namespace: "{{projectNameDeployConfig}}"
status: "{{status}}"
- tasks:
- name: "Get replica number for the pods"
command: aws dynamodb put-item --table-name pods_replicas
register: getResult
when: dynamodbtask == "get"
- name: "Update replica number for specified pods"
command: |
aws dynamodb put-item
--table-name pods_replicas
--item '{"ProjectNameDeployConfig":{"S":{{namespace}}},"NumberReplicas":{"N":{{numberOfReplicas}}}}'
register: updatePayload
when: dynamodbtask == "put" and getResult is skipped
However, there is always the following error:
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["aws", "dynamodb", "put-item", "--table-name",
"pods_replicas", "--item", "{\"ProjectNameDeployConfig\":{\"S\":LERN-PolicyCenterV10},\"NumberReplicas\":
{\"N\":0}}"], "delta": "0:00:01.702107", "end": "2020-02-09 16:58:26.055579",
"msg": "non-zero return code", "rc": 255, "start": "2020-02-09 16:58:24.353472", "stderr": "\nError parsing parameter '--item': Invalid JSON: No JSON object could be decoded\nJSON received: {\"ProjectNameDeployConfig\":{\"S\":LERN-PolicyCenterV10},\"NumberReplicas\":{\"N\":0}}", "stderr_lines": ["", "Error parsing parameter '--item': Invalid JSON: No JSON object could be decoded", "JSON received: {\"ProjectNameDeployConfig\":{\"S\":LERN-PolicyCenterV10},\"NumberReplicas\":{\"N\":0}}"], "stdout": "", "stdout_lines": []}
There are two answers to your question: the simple one and the correct one
The simple one is that had you actually fed the JSON into jq, or python -m json.tool, you would have observed that namespace is unquoted:
"{\"ProjectNameDeployConfig\":{\"S\": LERN-PolicyCenterV10 },\"NumberReplicas\": {\"N\":0}}"
where I added a huge amount of space, but didn't otherwise alter the quotes
The correct answer is that you should never use jinja2 to try and assemble structured text when there are filters that do so for you.
What you actually want is to use the to_json filter:
- name: "Update replica number for specified pods"
command: |
aws dynamodb put-item
--table-name pods_replicas
--item {{ dynamodb_item | to_json | quote }}
vars:
dynamodb_item:
"ProjectNameDeployConfig":
"S": '{{ projectNameDeployConfig }}'
"NumberReplicas":
"N": 0
register: updatePayload
when: dynamodbtask == "put" and getResult is skipped
although you'll notice that I changed your variable name because namespace is the name of a type in jinja2, so you can either call it ns or I just used the interpolation value from your vars: block at the top of the playbook, as it doesn't appear that it changed from then
When I want to exctract the current value of some container env variabe I could use jsonpath with syntax like:
kubectl get pods -l component='somelabel' -n somenamespace -o \
jsonpath='{.items[*].spec.containers[*].env[?(#.name=="SOME_ENV_VARIABLE")].value}')
That will return me the value of env varialbe with the name SOME_ENV_VARIABLE. Pod section with container env variables in json will look like this:
"spec": {
"containers": [
{
"env": [
{
"name": "SOME_ENV_VARIABLE",
"value": "some_value"
},
{
"name": "ANOTHER_ENV_VARIABLE",
"value": "another_value"
}
],
When I want to patch some value in my deployment I'm using commands with syntax like:
kubectl -n kube-system patch svc kubernetes-dashboard --type='json' -p="[{'op': 'replace', 'path': '/spec/ports/0/nodePort', 'value': $PORT}]"
But how can I patch a variable with 'op': 'replace' in cases where I need to use expression like env[?(#.name=="SOME_ENV_VARIABLE")]? Which syntax I should use?
Rather than kubectl patch command, you can make use of kubectl set env to update environment variable of k8s deployment.
envvalue=$(kubectl get pods -l component='somelabel' -n somenamespace -o jsonpath='{.items[*].spec.containers[*].env[?(#.name=="SOME_ENV_VARIABLE")].value}')
kubectl set env deployment/my-app-deploy op=$envvalue
Hope this helps.
Most of them haven't provide proper commands just use as simple as it is =>
kubectl set env deployment/deploy_name APP_VERSION=value -n namespace
op: replace
path: /spec/template/spec/containers/0/env/0/name
value: YOUR_VARIABLE_NAME
op: replace
path: /spec/template/spec/containers/0/env/0/value
value: YOUR_VARIABLE_VALUE
I am trying to create an Openshift template for a Job that passes the job's command line arguments in a template parameter using the following template:
apiVersion: v1
kind: Template
metadata:
name: test-template
objects:
- apiVersion: batch/v2alpha1
kind: Job
metadata:
name: "${JOB_NAME}"
spec:
parallelism: 1
completions: 1
autoSelector: true
template:
metadata:
name: "${JOB_NAME}"
spec:
containers:
- name: "app"
image: "batch-poc/sample-job:latest"
args: "${{JOB_ARGS}}"
parameters:
- name: JOB_NAME
description: "Job Name"
required: true
- name: JOB_ARGS
description: "Job command line parameters"
Because the 'args' need to be an array, I am trying to set the template parameter using JSON syntax, e.g. from the command line:
oc process -o=yaml test-template -v=JOB_NAME=myjob,JOB_ARGS='["A","B"]'
or programmatically through the Spring Cloud Launcher OpenShift Client:
OpenShiftClient client;
Map<String,String> templateParameters = new HashMap<String,String>();
templateParameters.put("JOB_NAME", jobId);
templateParameters.put("JOB_ARGS", "[ \"A\", \"B\", \"C\" ]");
KubernetesList processed = client.templates()
.inNamespace(client.getNamespace())
.withName("test-template")
.process(templateParameters);
In both cases, it seems to fail because Openshift is interpreting the comma after the first array element as a delimiter and not parsing the remainder of the string.
The oc process command sets the parameter value to '["A"' and reports an error: "invalid parameter assignment in "test-template": "\"B\"]"".
The Java version throws an exception:
Error executing: GET at: https://kubernetes.default.svc/oapi/v1/namespaces/batch-poc/templates/test-template. Cause: Can not deserialize instance of java.util.ArrayList out of VALUE_STRING token\n at [Source: N/A; line: -1, column: -1] (through reference chain: io.fabric8.openshift.api.model.Template[\"objects\"]->java.util.ArrayList[0]->io.fabric8.kubernetes.api.model.Job[\"spec\"]->io.fabric8.kubernetes.api.model.JobSpec[\"template\"]->io.fabric8.kubernetes.api.model.PodTemplateSpec[\"spec\"]->io.fabric8.kubernetes.api.model.PodSpec[\"containers\"]->java.util.ArrayList[0]->io.fabric8.kubernetes.api.model.Container[\"args\"])
I believe this is due to a known Openshift issue.
I was wondering if anyone has a workaround or an alternative way of setting the job's parameters?
Interestingly, if I go to the OpenShift web console, click 'Add to Project' and choose test-template, it prompts me to enter a value for the JOB_ARGS parameter. If I enter a literal JSON array there, it works, so I figure there must be a way to do this programmatically.
We worked out how to do it; template snippet:
spec:
securityContext:
supplementalGroups: "${{SUPPLEMENTAL_GROUPS}}"
parameters:
- description: Supplemental linux groups
name: SUPPLEMENTAL_GROUPS
value: "[14051, 14052, 48, 65533, 9050]"
In our case we have 3 files :
- environment configuration,
- template yaml
- sh file which run oc process.
And working case looks like this :
environment file :
#-- CORS ---------------------------------------------------------
cors_origins='["*"]'
cors_acceptable_headers='["*","Authorization"]'
template yaml :
- apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: plugin-common-cors
annotations:
kubernetes.io/ingress.class: ${ingress_class}
config:
origins: "${{origins}}"
headers: "${{acceptable_headers}}"
credentials: true
max_age: 3600
plugin: cors
sh file running oc :
if [ -f templates/kong-plugins-template.yaml ]; then
echo "++ Applying Global Plugin Template ..."
oc process -f templates/kong-plugins-template.yaml \
-p ingress_class="${kong_ingress_class}" \
-p origins=${cors_origins} \
-p acceptable_headers=${cors_acceptable_headers} \
-p request_per_second=${kong_throttling_request_per_second:-100} \
-p request_per_minute=${kong_throttling_request_per_minute:-2000} \
-p rate_limit_by="${kong_throttling_limit_by:-ip}" \
-o yaml \
> yaml.tmp && \
cat yaml.tmp | oc $param_mode -f -
[ $? -ne 0 ] && [ "$param_mode" != "delete" ] && exit 1
rm -f *.tmp
fi
The sh file should read environment file.