How to use the output of the invokeLambda in jenkins pipeline? - json

I am invoking a lambda function from jenkins declarative pipeline. Now i want to use its output in other the pipeline. I am trying the following code :
def health=invokeLambda([awsAccessKeyId: 'xxxx', awsRegion: 'rrrrr', awsSecretKey: 'kkkkk', functionName: 'yyyyyy', payload: '', synchronous: true]);
when i try echo "$health" iam getting null.
Does anyone know how to use the output of lambda function in jenkinsfile?

I Assume you are using the AWS Lambda Plugin which is a relatively old plugin and was designed for freestyle jobs and not for pipelines.
While you can use it in a pipeline script you will not be able to get the returned value as the plugin is designed to update environment variables with the results - which works grate in freestyle jobs but is not supported in pipelines.
To achieve what you want you can use the new pipeline oriented AWS Steps Plugin which contains many AWS related steps which are designed for pipeline usage and allow easy access to the output of the steps. It is also more secure and allows much more capabilities then the old plugin - especially for pipelines.
In your case you can use the invokeLambda step that will return the output is you expect:
withAWS(region:'eu-central-1', credentials:'nameOfSystemCredentials') {
def result = invokeLambda(
functionName: 'myLambdaFunction',
payload: [ "key": "value", "anotherkey" : [ "another", "value"] ]
)
}

Related

key-value of JSON object not stored in Azure App Config as expected when reading from App Config

I'm developing an Azure Function which has to consume JSON as input and then trigger a hybrid CI/CD pipeline split between on-prem and Azure DevOps. To split configuration from code I intend to use an Azure App Configuration store to retrieve configuration settings that the Function will use to trigger the correct pipeline depending on JSON input. I'm completely new to App Config but have tried to investigate how to properly use it. However, I have stumbled into a perplexing issue and can't find an explanation for it. I apologize if I have missed something obvious out there.
For the purpose of this question I have abstracted away any business-related terminology.
Imagine I have a JSON object stored in a file TestStructure.json that looks like this:
{
"TestStructure": {
"Repository1": {
"RepositoryName": "Repository1",
"RepositoryUrl": "https://url.repository1.com/"
},
"Repository2": {
"RepositoryName": "Repository2",
"RepositoryUrl": "https://url.repository2.com/"
},
"Repository3": {
"RepositoryName": "Repository3",
"RepositoryUrl": "https://url.repository3.com/"
}
}
}
I store this in my App Config using the Azure CLI with the following command:
az appconfig kv import -n <myAppConfigName> -s file --format json --path "C:\workspace\TestStructure.json" --content-type "application/json" --separator . --depth 2
The command yields the following key-value pairings:
---------------- Key Values Preview ----------------
Adding:
{"key": "TestStructure.Repository1", "value": "{\"RepositoryName\": \"Repository1\", \"RepositoryUrl\": \"https://url.repository1.com/\"}"}
{"key": "TestStructure.Repository2", "value": "{\"RepositoryName\": \"Repository2\", \"RepositoryUrl\": \"https://url.repository2.com/\"}"}
{"key": "TestStructure.Repository3", "value": "{\"RepositoryName\": \"Repository3\", \"RepositoryUrl\": \"https://url.repository3.com/\"}"}
These keys are what I expect to find in my App Config store.
Going to the App Config in the Azure Portal I find that the JSON object has been stored correctly, i.e. the keys are TestStructure.Repository1, TestStructure.Repository2 and so forth, all with their corresponding values as the Azure CLI command reported back. This screenshot verifies it:
Now, to the actual problem. When I try to fetch a key from my App Config I get some weird behavior.
I have put together a simple Console App in .NET 6 to test how to read from the App Config:
1 using Microsoft.Extensions.Configuration;
2
3 var config = new ConfigurationBuilder()
4 .AddAzureAppConfiguration("MyConnectionString")
5 .Build();
6
7 var repository = config["TestStructure.Repository1"] // Returns null
It doesn't make sense to me why line 7 returns null, so I attached a debugger to inspect the ConfigurationRoot object a bit further and found the following:
What is going on here? Inspecting the config object reveals that the actual keys to index with are stored as TestStructure.Repository1:RepositoryName and not TestStructure.Repository1 and then the corresponding values.
Thank you for taking your time to read my question. I hope I have expressed clearly what I am trying to achieve and what my problem is.

What is a useful Azure IoT Hub JSON message structure for consumption in Time Series Insights

The title sounds quite comprehensive, but my baseline question is quite simple, I guess.
Context
I Azure, I have an IoT hub, which I am sending messages to. I use a modified version one of the samples from the Azure Iot SDK for python.
Sending works fine. However, instead of a string, I send a JSON structure.
When I watch the events flowing into the IoT hub, using the Cloud shell, it looks like this:
PS /home/marcel> az iot hub monitor-events --hub-name weathertestiothub
This extension 'azure-cli-iot-ext' is deprecated and scheduled for removal. Please remove and add 'azure-iot' instead.
Starting event monitor, use ctrl-c to stop...
{
"event": {
"origin": "raspberrypi-zero-wh",
"payload": "{ \"timestamp\": \"1608643863720\", \"locationDescription\": \"Attic\", \"temperature\": \"21.941\", \"relhumidity\": \"71.602\" }"
}
}
Issue
The data seems fine, except the payload looks strange here. BUT, the payload is literally what I send from the device, using the SDK sample.
Is this the correct way to do it? At the end, I have a very hard time to actually get the data into the Time Series Insights model. So I guess, my structure is to be blamed.
Question
What is a recommended JSON data structure to send to the IoT hub for later use?
You should add the following 2 lines to your message in your python SDK sample:
msg.content_encoding = "utf-8"
msg.content_type = "application/json"
This should resolve your formatting concern.
We've also updated our samples to reflect this: https://github.com/Azure/azure-iot-sdk-python/blob/master/azure-iot-device/samples/sync-samples/send_message.py
I ended up using the tip by #elhorton, but it was not the key change. Nonetheless, the formatting in the Azure Shell Monitor looks now much better:
"event": {
"origin": "raspberrypi-zero-wh",
"payload": {
"temperature": 21.543947753906245,
"humidity": 69.22964477539062,
"locationDescription": "Attic"
}
}
The key was:
include the message source time in ISO format
from datetime import datetime
timestampIso = datetime.now().isoformat()
message.custom_properties["iothub-creation-time-utc"] = timestampIso
Using the locationDescription as the Time Series ID Property See https://learn.microsoft.com/en-us/azure/time-series-insights/how-to-select-tsid (Maybe I could also have taken the iothub-connection-device-id, but I did not test that alone specifically)
I guess using "iothub-connection-device-id" will make "raspberrypi-zero-wh" as the name of the time series instance. I agree with your choice of using "locationDescription" as TSID; so Attic becomes the time series instance name, temperature and humidity will be your variables.

Pass information between two Jenkins jobs in a Pipeline

I want to migrate an existing job to Jenkins Pipeline and this process I try to migrate the different plugins attached to this job to use the new syntax.
However, not all plugins provide a corresponding wrapper syntax.
In the current case, I want to allocate three separate ports (embedded DB, container, and process engine) so that the build can run independently from other builds on the same machine. In the classic Jenkins job, we could use the Port Allocator Plugin, but it's not (yet) available via Pipeline Syntax.
My idea is to trigger a classic build that uses the port allocator plugin and returns the free ports that I can then use in a later build to start up the required services.
node {
stage("Allocate Ports") {
build job: "allocate-port", parameters: [ // <- the classic build
string(name: "name1", value: "PORT_DB"),
string(name: "name2", value: "PORT_CONTAINER"),
string(name: "name3", value: "PORT_ENGINE")
]
}
stage("Integration Tests") {
sh """
run-test \
-db=${PORT_DB} \
-container=${PORT_CONTAINER} \
-engine=${PORT_ENGINE}
"""
}
}
Is there a good way to store the results from the allocate-port build and return it to the enclosing pipeline?

Jenkins json REST api does not show failcount, skipCount and totalCount for a console job output

I am planning to use following data in one of my dashboards that I am running on dashing.io framework. Is there any plugin or setting that I am missing in the Jenkins job or server that is restricting it to set this data in Jenkins REST API for particular test?
{
"failCount" : 1,
"skipCount" : 0,
"totalCount" : 62,
"urlName" : "reportTest"
},
The following Jenkins-API-page gives you a detailed overview about all tests executed in your build-job if your Jenkins is provided with the JUnit-Plugin:
https://{jenkins}/job/{jobname}/{jobnr}/testReport/api/json?pretty=true
You can easily use the json-code from that url in your dashing.io-Dashboard if you adapt the dashing-jenkins-widget which you can find on github.

How to expose Openshift enviroment variables on a json

I have installed node-push-server. The configuration is loaded from a json like this:
{
"webPort": 8000,
"mongodbUrl": "mongodb://username:password#localhost/database",
"gcm": {
"apiKey": "YOUR_API_KEY_HERE"
},
"apn": {
"connection": {
"gateway": "gateway.sandbox.push.apple.com",
"cert": "/path/to/cert.pem",
"key": "/path/to/key.pem"
},
"feedback": {
"address": "feedback.sandbox.push.apple.com",
"cert": "/path/to/cert.pem",
"key": "/path/to/key.pem",
"interval": 43200,
"batchFeedback": true
}
}
}
How can I set the enviroment variables for my application in this json file?
I don't think it's possible. You should be able to change all these settings in the code though. For example in node you can do: process.env.OPENSHIFT_VARIABLENAME to read an environment variable.
Example for MongoDB connection string from docs:
//provide a sensible default for local development
mongodb_connection_string = 'mongodb://127.0.0.1:27017/' + db_name;
//take advantage of openshift env vars when available:
if(process.env.OPENSHIFT_MONGODB_DB_URL){
mongodb_connection_string = process.env.OPENSHIFT_MONGODB_DB_URL + db_name;
}
As an alternative, there is a quick and easy deployable gear called AeroGear Push that might serve your needs.
Config files can be awkward because including them in your source repo isn't always a good move.
OpenShift deployments are mostly git push-driven, so there are several options for helping you correctly resolve your configs on the server.
Configuring your service using ENV vars is the most common approach, but since this one requires a flat file, you'll need to find a way to update the file with the correct values.
If you know what keys and values are needed, you should be able to write a script that updates the example json, or merges two json objects to produce a flat config file including the strings node-pushserver will expect.
It looks like mongodbUrl, webPort, (and domain?) would need to be populated with OpenShift-provided values (when available). config-multipaas might be able to help with that.
I would probably implement the config bootstrapping / merging work as a build step, allowing you to prep the config file and start node-pushserver in it's usual way