Azure key Vault value integration inside a JSON of an application - json

I have Mesosphere DC/OS installed on Azure cluster, running tomcat apps as services, those services are configured using JSON files holding the ports and passwords of the apps. My Manager wants to use Azure Key Vault to store the passwords and secrets of the apps, i created the vault and stored in it the secrets i need.
This is a part of my JSON values which i need to replace:
(i cut only the fields with the values i want to replace from the Vault)
"APP_ACCESS_SERVICE_PASSWORD": "AppPW",
"CASSANDRA_DB_PASSWORD": "App_uat_PW",
"UAMS_ORACLE_PASSWORD": "App_uat_PW",
"PUBLISH_DB_PASSWORD": "ogw",
"App-PUBLISH_DB_PASSWORD": "App_uat1",
"EMP_DB_PASSWORD": "App_uat1",
How can i replace the passwords in my JSON with the values in the key vault ? i mean putting a URL to the password instead of the password is not an option, any Idea how can i input the values in the key vault into the JSON instead of the static values ?
From Azure documentation i see that i can access the values using URL, for example:
https://Contoso-Vault2.vault.azure.net/secrets/ExamplePassword
but using URL instead of the value is not an option. (it won't work)

I assume you have a pipeline but here's how you could do it.
First create your secrets in your desired key vault. I have created an app using Python and the Azure sdk which lets you create multiple secrets in a key vault very easily - https://github.com/TechyTish/AzurePy/blob/main/create-azure-secrets-README.md
lets say your secrets are stored as (secretName: secretValue):
appAccessService: 1234
cassandraDB: hello567
...
etc
#filename: example.json
"APP_ACCESS_SERVICE_PASSWORD": "{#appAccessServicePW#}",
"CASSANDRA_DB_PASSWORD": "{#cassandraDB#}",
"UAMS_ORACLE_PASSWORD": "{#uamsOraclePW#}",
"PUBLISH_DB_PASSWORD": "{#publishDBPW#}",
"App-PUBLISH_DB_PASSWORD": "{#appPublishPW#}",
"EMP_DB_PASSWORD": "{#empDBPW#}",
Create a YAML pipeline which will have 2 tasks:
Extract keyvault secrets
Replace the .json key values with the secrets
#filename: azure-pipeline.yml
# Azure Key Vault
# Download Azure Key Vault secrets
- task: AzureKeyVault#2
inputs:
connectedServiceName: # Azure subscription - you will need to create an service connection in the repo which has access policies to the keyvault
keyVaultName: # Name of existing key vault
secretsFilter: '*' # Downloads all secrets for the key vault
runAsPreJob: true # Runs before the job starts
Underneath the previous task add another one.
This looks for any key value with the {# pre-fix and #} suffix in the .json file and the variables (below) after this task will replace the value in the .json file with the value of the secrets you assigned it.
#filename: azure-pipeline.yml
- task: qetza.replacetokens.replacetokens-task.replacetokens#3
inputs:
targetFiles: "$(Pipeline.Workspace)/codepath/jsonFileName.json"
encoding: "auto"
writeBOM: true
verbosity: "detailed"
actionOnMissing: "warn"
keepToken: false
tokenPrefix: "{#"
tokenSuffix: "#}"
displayName: Perform variable substitution in json file
- script: echo "$(<folderName/example.json)" #open json file in the terminal to show the changes
#.json value: point to secret name
variables:
appAccessServicePW: $(appAccessService)
cassandraDB: $(cassandraDB)
#....
#etc
Run the pipeline (tick the box for debugging) and your output of .json file should be:
#filename: appsettings.json
"APP_ACCESS_SERVICE_PASSWORD": "1234",
"CASSANDRA_DB_PASSWORD": "hello567"
#....
#etc
This way the only time your secrets are revealed is when the pipeline is run on whichever host agent (virtual machine) you use.

Related

Azure linux VM deployment .json failing with inputting a password for deployment

Apologies in advance I am extremely new to this I am trying to run the following code and don't understand the error when attempting to generate a password for his azuredeploy.json runbook (or whatever its called)
$ az group deployment create --resource-group ProjectSpaghetti --template-uri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/docker-simple-on-ubuntu/azuredeploy.jsonubuntu/azuredeploy.json
This command is implicitly deprecated because command group 'group deployment' is deprecated and will be removed in a future release. Use 'deployment group' instead.
Please provide string value for 'adminUsername' (? for help): luke
Please provide string value for 'dnsNameForPublicIP' (? for help): lasagne
Please provide securestring value for 'adminPasswordOrKey' (? for help):
Deployment failed. Correlation ID: 55abf44d-92e4-4a7f-9c0e-ed7e5353d586. {
"error": {
"code": "InvalidParameter",
"message": "The value of parameter linuxConfiguration.ssh.publicKeys.keyData is invalid.",
"target": "linuxConfiguration.ssh.publicKeys.keyData"
}
}
I assumed because it used the phrase "adminPasswordOrKey" I would be able to use a password of reasonable complexity. Is that not the case? I'm a real newb at this just having a play really.
You see this error because you seemed to have provided a password instead of the default SSH Public key for the adminPasswordOrKey parameter that this template parameter expects.
You could take either of the following steps next:
Pass on the SSH Public key it expects
If you do not already have one, generate a new keypair and pass the contents of the public key as follows:
az group deployment create --resource-group <resource-group-name> --template-uri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/docker-simple-on-ubuntu/azuredeploy.json --parameters adminPasswordOrKey="ssh-rsa AAAAB..."
Specify an alternate authentication mechanism
Override the authenticationType to accept a password instead of a sshPublicKey. Check this blog for a quick walk through.
You could also set your parameters using a parameter file. Check this document for other ways: Deploy resources with ARM templates and Azure CLI

Environment Variables in a JavaScript Github Action

I am writing a javascript github action. Inside my action.yml, I have the following:
runs:
using: node12
main: ./index.js
Inside my index.js, I am calling an api which requires a secret key. I want to use my own secret key. I don't want users who use my action to define their own key. How can I add my secret key to the file as a secret ort env variable ?
Secrets need to come from somewhere. Even if you encrypt your token for the external api and host it directly in the action's code, you still need a passphrase to decrypt it and that needs to come from a secrets entry by the user of your action as it would need to be hosted in their own repo or organization.
Example of encrypting a secret if you wanted to, but still requires a passphrase, therefore we are back on the same boat.
gpg --symmetric --cipher-algo AES256 my_secret.json
The reason this situation is not ideal is because from my understanding, you want to use one token for all users of your action. Therefore the secret will need to be public in some way as multiple users will need to have that token, so either you host the token directly in your action or tell users to provide it to you through their secrets setup as an action variable or env. Here is an example of the user sharing the token in two different ways to your action.
steps:
- name: Hello world action
with: # Set the secret as an input
super_secret: ${{ secrets.SuperSecret }}
env: # Or as an environment variable
super_secret: ${{ secrets.SuperSecret }}

Nested JSON path with dot/period in azure

The goal is to use the az iot edge deployment update command to change a module in an azure iot hub/edge deployment. The attempt to do this uses the property-path within the deployment configuration json to replace the image path. The problem is that there is a dot in a json property properties.desired and attempts of escaping it have been futile. The file is a default azure deployment configuration file.
Command format
az iot edge deployment update --deployment-id <name-of-deployment> --hub-name <name-of-iot-hub> --set <json-path>=<new-value>
First part of the deployment configuration (json)
The goal is to change the value of image
{
"content": {
"modulesContent": {
"$edgeAgent": {
"properties.desired": {
"modules": {
"demoimage1-latest": {
"settings": {
"image": "demoworkspac2478a907.azurecr.io/demoimage1:6",
The most obvious attempt
az iot edge deployment update --deployment-id demoimage1-6 --hub-name iot-hubski --set content.modulesContent.'$edgeAgent'.'properties.desired'.modules.'demoimage1-latest'.settings.image=demoworkspac2478a907.azurecr.io/demoimage1:5
Gives
Couldn't find 'properties' in 'content.modulesContent.$edgeAgent.properties.desired.modules.demoimage1-latest'. Available options: ['properties.desired']
Status
Many things have been tried using both bash (ubuntu LTS vm) and powershell (win10)
[properties.desired]
'[properties.desired]'
['properties.desired']
properties\.desired
properties.desired`
properties.desired
'..."properties.desired"...'
'...\"properties.desired\"...'
'$edgeAgent'[properties.desired]
'$edgeAgent'['properties.desired']
^[properties.desired^]
^^[properties.desired^^]
``[properties.desired]
```[properties.desired``]`
you need to manually strinigying the $edgeHub JSON.
az iot edge deployment update --deployment-id testedge --hub-name microwaves --set content.modulesContent.'$edgeHub'="{'properties.desired': {'routes': {'route': 'FROM /messages/* INTO $upstream'},'schemaVersion': '1.0','storeAndForwardConfiguration': {'timeToLiveSecs': 7201}}}"
However it doesn't do anything because of content being immutable. Items that can be updated by az iot edge deployment update command: labels, metrics, priority and targetCondition. labels and metrics do not allow values with ‘.’ in the name.

Can I use a more restrictive key with Jekyll Algolia?

I am building my Jekyll site with Algolia search.
The documentation about jekyll-algolia says the admin key must be provided in the environment variable ALGOLIA_API_KEY.
However, another page about API key security says
Your admin API key is the most sensitive key: it provides full control of all your indices and data. The admin API key should always be kept secure. Do NOT release it to anybody or do NOT use it in any application, and always create a new key that will be more restrictive. This API key should almost exclusively be used to generate other - more limited - API Keys that will then be used to search and perform indexing operations.
Reading the second page, I'm trying to create a more restrictive key for use with jekyll-algolia in CI builds of my Jekyll website:
However I still get complaints from bundle exec jekyll algolia:
ibug#ubuntu:~/iBug.github.io$ ALGOLIA_API_KEY="0123456789abcdef0123456789abcdef" bundle exec jekyll algolia
Configuration file: /home/wsl/iBug.github.io/_config.yml
Processing site...
AutoPages: Disabled/Not configured in site.config.
Pagination: Complete, processed 1 pagination page(s)
Jekyll Feed: Generating feed for posts
GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
Extracting records...
Updating records in index iBug_website...
Records to delete: 428
Records to add: 420
[✗ Error] Invalid credentials
The jekyll-algolia plugin could not connect to your application ID using the
API key your provided.
Make sure your API key has access to your 14DZKASAEJ application.
You can find your API key in your Algolia dashboard here:
https://www.algolia.com/licensing
ibug#ubuntu:~/iBug.github.io$ echo $?
1
How should I do that? Or must I provide the admin key in CI environments?
Minimum API key ACLs required to allow indexing with jekyll-algolia are deleteIndex, addObject, deleteObject and 'editSettings`.
If one of those ACLs is not set you get an error like this :
[jekyll-algolia] Error:
403: Cannot PUT to
https://APP_ID.algolia.net/1/indexes/your_folder/settings:
{"message":"Method not allowed with this API key","status":403} (403)
In your case, the error message indicates that your application ID is not connected with the API_KEY you provide.
Check your application ID in your Algolia dashboard, and verify that you have a correct algolia.application_id entry in your _config.yml.
If you provide the right application_id and one of her API key, it must work, otherwise it's an Algolia problem.

POSTMAN - Newman Run Iteration Json File on Specific Endpoint

I have a postman collection that contains some endpoint. Something like this
my-collection
|- Get All Users
|- Get User
|- Delete User
I also have some newman scenario/iteration json files (1. get-all-users.json, 2. get-user.json, 3. delete-user.json).
How do i run single scenario/iteration json file to specific endpoint on my-collection?
Lets say get-all-users.json run for Get All Users endpoint only, instead of run for all endpoints.
By 'single endpoint', you mean running a single sub-folder for 1 iteration, say, 'Get All Users'. Right?
If so, you can export your my-collection as a JSON on your system (No need to individually export the sub-folders to get-all-users.json, get-user.json, delete-user.json).
You can use --folder option (See here for reference). Just give the folder name you want to run, i.e., "Get All Users"
So, your newman command should be (your environment json and global json being "env.json" and "global.json" respectively) :
newman run "<my_collection_path>/my-collection.json" --folder "Get All Users" -e "<env_path>/env.json" -g "<global_path>/global.json"
This way you should be able to run a requests in a single sub-folder at a time.