I am trying to add an environment variable containing a json which needs to give out the deployed branch name like below:
stage('Build files') {
agent { label "${AGENT_NAME}"}
environment {
DEPLOYED_BRANCH = '{\"branch\":\"${env.BRANCH_NAME}\"}'
}
steps { //Do something}
ansiColor('xterm') {
sh """
echo \"${DEPLOYED_BRANCH}\" > deployed_branch.json
"""
}
}
While deploying, I am getting :
"{"branch":"${env.BRANCH_NAME}"}": bad substitution
I've tried a few different options with the single/double quotes but with no success. Any ideas?
you can do it like this, but it will be a JSON string in the deployed_branch.json file (JSON file with a string content)
stage('create a JSON ') {
environment {
SOME_ENV_VARS = '{\"branch\":\"$WORKSPACE\"}'
}
steps {
sh """
echo $SOME_ENV_VARS > deployed_branch.json
"""
}
}
It can be done better way like this:
scripted pipeline
node {
stage('any') {
def myJSON = readJSON text: '{}'
myJSON.branch = "${env.WORKSPACE}" as String
writeJSON file: "${WORKSPACE}/x.json", json: myJSON
}
}
declarative pipeline
pipeline {
agent any;
stages {
stage('create a JSON ') {
steps {
script {
def myJSON = readJSON text: '{}'
myJSON.branch = "${env.WORKSPACE}" as String
writeJSON file: "${WORKSPACE}/x.json", json: myJSON
}
}
}
}
}
Note - it need Jenkins Utility Pluigns
Related
Hope anybody could guide me here. I spend some hours on it and can't understand what's going on.
Mission: Replace a json element by a jsonpath search tag. (sort of $ref feature)
In my code example below i want to replace the value of DataReaderUser by a value found by the json path search $.UsersAndGroups.Users[?(#.Name == 'OMDASAccountUser')].Username . In this case it should result in the value "contoso\SVCSCOM-DO-OMDAS"
The code below works as expected.. the issue is below this code ..
https://dotnetfiddle.net/gEjggK
using System;
using Newtonsoft.Json.Linq;
public class Program
{
public static void Main()
{
string json = #"{
""SQLServer"": {
""SQLReportingServices"": {
""AccountSettings"": {
""DataReaderUser"": {""$JsonPath"": ""$.UsersAndGroups.Users[?(#.Name == 'OMDASAccountUser')].Username""},
}
}
},
""UsersAndGroups"": {
""Users"": [
{
""Name"": ""OMActionAccountUser"",
""Username"": ""contoso\\SVCSCOM-DO-OMDAS"",
},
{
""Name"": ""OMDASAccountUser"",
""Username"": ""contoso\\SVCSCOM-DO-OMDAS"",
}
]
}
}";
JObject jo = JObject.Parse(json);
var JsonPath = jo.SelectToken("..$JsonPath");
JsonPath.Parent.Parent.Replace(jo.SelectToken(JsonPath.ToString()));
Console.WriteLine(jo.ToString());
}
}
The output will be :
{
"SQLServer": {
"SQLReportingServices": {
"AccountSettings": {
"DataReaderUser": "contoso\\SVCSCOM-DO-OMDAS"
}
}
},
"UsersAndGroups": {
"Users": [
{
"Name": "OMActionAccountUser",
"Username": "contoso\\SVCSCOM-DO-OMDAS"
},
{
"Name": "OMDASAccountUser",
"Username": "contoso\\SVCSCOM-DO-OMDAS"
}
]
}
}
Now the issue:
I want to do the same for all possible jsonpaths refers. So i use the SelectTokens and an foreach . But it looks like the behavior is different , the parents are null.
https://dotnetfiddle.net/lZW3XP
using System;
using Newtonsoft.Json.Linq;
public class Program
{
public static void Main()
{
string json = #"{
""SQLServer"": {
""SQLReportingServices"": {
""AccountSettings"": {
""DataReaderUser"": {""$JsonPath"": ""$.UsersAndGroups.Users[?(#.Name == 'OMDASAccountUser')].Username""},
}
}
},
""UsersAndGroups"": {
""Users"": [
{
""Name"": ""OMActionAccountUser"",
""Username"": ""contoso\\SVCSCOM-DO-OMDAS"",
},
{
""Name"": ""OMDASAccountUser"",
""Username"": ""contoso\\SVCSCOM-DO-OMDAS"",
}
]
}
}";
JObject jo = JObject.Parse(json);
var JsonPaths = jo.SelectTokens("..$JsonPath");
foreach (var JsonPath in JsonPaths )
{
JsonPath.Parent.Parent.Replace(jo.SelectToken(JsonPath.ToString()));
}
Console.WriteLine(jo.ToString());
}
}
And the output:
Run-time exception (line 34): Object reference not set to an instance of an object.
Stack Trace:
[System.NullReferenceException: Object reference not set to an instance of an object.]
at Newtonsoft.Json.Linq.JsonPath.PathFilter.GetNextScanValue(JToken originalParent, JToken container, JToken value)
at Newtonsoft.Json.Linq.JsonPath.ScanFilter.<ExecuteFilter>d__4.MoveNext()
at Program.Main() :line 34
would be great to get some directions since i am spinning my head here.
michel
SelectTokens uses lazy evaluation and if you modify the token while enumerating all matches it can break in unexpected ways. A simple fix is to add ToArray() to force eager evaluation:
var JsonPaths = jo.SelectTokens("..$JsonPath").ToArray();
I have a function in a Jenkins shared library which builds and returns a map object (class java.util.LinkedHashMap):
#!/usr/bin/env groovy
def call(Map config) {
script {
echo "Building nested map"
def MAP_VAR = [:]
MAP_VAR.put("one", [:])
MAP_VAR.put("two", [:])
MAP_VAR.get("one").put("a", "b")
MAP_VAR.get("two").put("c", "d")
echo "Returning: ${MAP_VAR}"
echo "Type: ${MAP_VAR.getClass()}"
return MAP_VAR
}
}
When this function runs the log output shows:
Returning: [one:[a:b], two:[c:d]]
Type: class java.util.LinkedHashMap
But when I call the function and assign the return value to a variable, it ends up being a string (class java.lang.String):
#!/usr/bin/env groovy
library identifier: 'library#main',
changelog: false,
retriever: modernSCM([ $class: 'GitSCMSource',
remote: 'git#github.com:org/repo.git',
credentialsId: 'credentials' ])
pipeline {
agent none
stages {
stage('Get map') {
agent any
steps {
script {
env.MAP_VAR = getMapVar()
}
}
}
stage('Check map') {
agent any
steps {
script {
echo "Value: ${env.MAP_VAR}"
echo "Type: ${env.MAP_VAR.getClass()}"
}
}
}
}
}
The output shows:
Value: {one={a=b}, two={c=d}}
Type: class java.lang.String
Ultimately I'm trying to access the map's properties in multiple stages within the Jenkinsfile. If I try this:
echo "Value: ${env.MAP_VAR['one']}"
I get:
groovy.lang.MissingPropertyException: No such property: one for class: java.lang.String
I've tried:
def env.MAP_VAR = getMapVar()
But that results in:
WorkflowScript: 59: unexpected token: def # line 59, column 21.
def env.MAP_VAR = getMapVar()
^
I've also tried:
def Map env.MAP_VAR = getMapVar()
But that results in:
WorkflowScript: 60: unexpected token: Map # line 60, column 25.
def Map env.MAP_VAR = getMapVar()
^
How can I get the Map/LinkedHashMap from the function as a Map/LinkedHashMap (which would allow me to access the properties/values of the map contents) and assign it to a global variable which can be used in all stages?
def env.MAP_VAR = getMapVar()
This will not work as you are trying to define a key MAP_VAR in env map.
def Map env.MAP_VAR = getMapVar()
This also will not work as you are trying to define MAP_VAR key in env map as def and also MAP. This is similar to doing something like String Integer my_var_name = something()
env.MAP_VAR = getMapVar()
You are storing the return value as a string in env environment variable map. Jenkins output is accurate given the code.
To achieve what you are trying to do, you can store it as a variable and it will work fine. In the following example, the variable goes into Groovy's script binding and will be available in the next stage. You can also define def MAP_VAR outside the pipeline block to get the same result.
#!/usr/bin/env groovy
library identifier: 'library#main',
changelog: false,
retriever: modernSCM([ $class: 'GitSCMSource',
remote: 'git#github.com:org/repo.git',
credentialsId: 'credentials' ])
pipeline {
agent none
stages {
stage('Get map') {
agent any
steps {
script {
MAP_VAR = getMapVar()
}
}
}
stage('Check map') {
agent any
steps {
script {
echo "Value: ${MAP_VAR}"
echo "Type: ${MAP_VAR.getClass()}"
}
}
}
}
}
This will print
Value: [one:[a:b], two:[c:d]]
Type: class java.util.LinkedHashMap
In stage two,
echo "${MAP_VAR['one']}"
will output
[a:b]
How do you capture a JSON object as a prettified string using a Jenkins declarative-syntax pipeline?
pipeline {
agent any
stages {
stage( "Set up" ) {
steps {
script {
hostname = "bld-machine"
reply_email = "jenkins#${hostname}.company.com"
actor_email = "user#company.com"
status_json = initStatusJson()
}
}
}
/** Try figure out the difference between "global" and "env." variables. */
stage( "Capture variables" ) {
steps {
script {
status_json.env["var"] = "${env.var}" as String
status_json.env["var2"] = "${var}" as String
}
}
}
}
post {
always {
script {
def pretty_json = writeJSON( returnText: true, json: status_json )
}
emailext( subject: "CI/CD | ${currentBuild.currentResult}",
from: "${reply_email}",
to: "${actor_email}",
mimeType: "text/plain",
body: "${pretty_json}" )
}
}
}
def initStatusJson() {
def json_obj = readJSON text: '{}'
json_obj.job = readJSON text: '{}'
json_obj.env = [:]
json_obj.job.name = "${JOB_BASE_NAME}" as String
json_obj.job.number = "${BUILD_ID}" as String
json_obj.job.server = "${JENKINS_URL}" as String
json_obj.job.visualization = "${JENKINS_URL}/blue/organizations/jenkins/${JOB_BASE_NAME}/detail/${JOB_BASE_NAME}/${BUILD_ID}/pipeline" as String
return json_obj
}
The def pretty_json =... statement in the above Jenkinsfile triggers the following error:
WARNING: Unknown parameter(s) found for class type WARNING: Unknown parameter(s) found for class type 'org.jenkinsci.plugins.pipeline.utility.steps.json.WriteJSONStep': returnText
[Pipeline] }
[Pipeline] // script
Error when executing always post condition:
java.lang.IllegalArgumentException: You have to provided a file for writeJSON.
at org.jenkinsci.plugins.pipeline.utility.steps.json.WriteJSONStepExecution.run(WriteJSONStepExecution.java:61)
at org.jenkinsci.plugins.pipeline.utility.steps.json.WriteJSONStepExecution.run(WriteJSONStepExecution.java:43)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
What I have tried:
The def pretty_json = writeJSON( returnText: true, json: status_json ) statement is inspired by these resources:
Jenkinsfile pipeline construct JSON object and write to file
https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#writejson-write-json-to-a-file-in-the-workspace
I also tried def pretty_json = writeJSON returnText: true, json: status_json which resulted in an identical error.
status_json.toString() returns a valid, but non-prettified JSON string.
I tried def pretty_json = JsonOutput.toJson(status_json) based on Create JSON strings from Groovy variables in Jenkins Pipeline, and it generates this error:
Error when executing always post condition:
groovy.lang.MissingPropertyException: No such property: JsonOutput for class: groovy.lang.Binding
Tried def pretty_json = groovy.json.JsonOutput.prettyPrint(status_json) based on https://gist.github.com/osima/1161966, and it generated this error:
Error when executing always post condition:
groovy.lang.MissingMethodException: No signature of method: java.lang.Class.prettyPrint() is applicable for argument types: (net.sf.json.JSONObject)
Update: Attempted #daggett's solution as follows:
post {
always {
script {
def pretty_json = status_json.toString(2)
}
emailext( subject: "CI/CD | ${currentBuild.currentResult}",
from: "${reply_email}",
to: "${actor_email}",
mimeType: "text/plain",
body: "${pretty_json}" )
}
}
...and also tried some variations like pretty_json = ... (instead of def pretty_json = ...) and also moving the pretty_json assignment outside the script{...} scope...but none worked.
Inside the script{...} context, the .toString(2) generated this error:
Scripts not permitted to use method net.sf.json.JSON toString int.
Outside the script{...} context, it generated what I interpret to be a "syntax error":
WorkflowScript: 79: Expected a step # line 79, column 7.
pretty_json = status_json.toString(2)
According to last error message
groovy.lang.MissingMethodException:
No signature of method: java.lang.Class.prettyPrint()
is applicable for argument types: (net.sf.json.JSONObject)
You have net.sf.json.JSONObject in status_json variable.
that's really strange - seems you are getting status_json not in a standard way for jenkins
however according to documentation of this class
http://json-lib.sourceforge.net/apidocs/jdk15/net/sf/json/JSONObject.html#toString(int)
just do following to make pretty json:
def pretty_json = status_json.toString(2)
If you have Scripts not permitted to use method XYZ exception:
for security reasons a lot of non-standard methods are disabled in jenkins.
refer this answer to resolve this kind of issue: https://stackoverflow.com/a/39412951/1276664
and finally - almost every case from your question should work:
writeJSON( returnText: true, json: status_json ) :
update pipeline-utility-steps jenkins plugin to the latest version to support returnText parameter
the same as above
...
JsonOutput.toJson(status_json) : JsonOutput class located in groovy.json package. you could import this package at t
he beginning of the script import groovy.json or call it like this: groovy.json.JsonOutput.toJson(status_json). note that this method returns non-formatted json.
groovy.json.JsonOutput.prettyPrint(status_json) : check the documentation for JsonOutput.prettyPrint - it could be called for string and not for object. so this could work: groovy.json.JsonOutput.prettyPrint(status_json.toString()) but only in case when status_json.toString() returns a valid json and JsonOutput.prettyPrint allowed to be called in jenkins admin.
I just did a test and it gave results :
def pretty_json = writeJSON( returnText: true, json: status_json , pretty: 4)
Note : Ensure you have the plugin Pipeline Utility Steps installed. Or reinstall it again.
Below is the script example:
#!groovy
import hudson.model.Result
import groovy.json.*
pipeline
{
agent any
stages
{
stage ('Set up')
{
steps
{
script
{
hostname = "bld-machine"
reply_email = "jenkins#${hostname}.company.com"
actor_email = "user#company.com"
status_json = initStatusJson()
println (status_json)
}
}
}
stage ('Capture variables')
{
steps
{
script
{
// Added just for test
status_json.env["var"] = "Alt" as String
status_json.env["var2"] = "su" as String
println (status_json)
}
}
}
}
post {
always {
script {
def pretty_json = writeJSON( returnText: true, json: status_json , pretty: 4)
println (pretty_json)
emailext( subject: "CI/CD | ${currentBuild.currentResult}",
from: "${reply_email}",
to: "${actor_email}",
mimeType: "text/plain",
body: "${pretty_json}" )
}
}
}
}
def initStatusJson() {
def json_obj = readJSON text: '{}'
json_obj.job = readJSON text: '{}'
json_obj.env = [:]
json_obj.job.name = "${JOB_BASE_NAME}" as String
json_obj.job.number = "${BUILD_ID}" as String
json_obj.job.server = "${JENKINS_URL}" as String
json_obj.job.visualization = "${JENKINS_URL}/blue/organizations/jenkins/${JOB_BASE_NAME}/detail/${JOB_BASE_NAME}/${BUILD_ID}/pipeline" as String
return json_obj
}
Output log :
I have a json file. I want to encode into base64 string and add it in my main json using jsonnet.
datasources.json:
{
"datasources": [{
"id": 1,
"orgId": 1,
"name": "prometheus"
}]
}
grafana.jsonnet:
local getDataSources() = {
'datasources': (import 'datasources.json').datasources,
};
local kp = {
dashboardDatasources+: {
data: std.base64(getDataSources().datasources),
}
}
kp
Please suggest how to get this done. Struggling to convert json into a string in jsonnet.
This can be done using std.manifestJsonEx
std.manifestJsonEx(value, indent) Convert the given object to a JSON
form. indent is a string containing one or more whitespaces that are
used for indentation:
Here is the solution:
local getDataSources() = {
'datasources': (import 'datasources.json').datasources,
};
local dataSources = getDataSources().datasources;
local kp = {
dashboardDatasources+: {
data: std.base64(std.manifestJsonEx(dataSources, " ")),
}
}
kp
Given a JSON output of an api:
{
"id": 13,
"name": "horst",
"cars": [{
"brand": "VW",
"maxSpeed": 120,
"isWastingGazoline": true
}]
}
I would like to define interfaces for typescript:
export interface Car {
brand: string;
maxSpeed: number;
isWastingGazoline: boolean;
}
export interface RaceCarDriver {
id: number;
name: string;
cars: Car[];
}
Yet I don't want them to type them manually, I rather have a script generate them for me.
How can I convert json into typescript interfaces? I also don't want to use webservices like MakeTypes or json2ts.
You can write the script using typescript compiler API and its ability to infer types. I was really surprised how easy it was.
You have to wrap your sample data to make it compile-able as typescript code. The script will pick all variable declarations and try to print inferred types for them. It uses variable names and property names for assigning names to types, and if two objects have a property with the same name, it will pick the type from the first one. So it will not work in case these types are actually different (the fix is left as an exercise). For your JSON output, the data sample will look like
file sample.ts
let raceCarDriver = {
"id": 13,
"name": "horst",
"cars": [{
"brand": "VW",
"maxSpeed": 120,
"isWastingGazoline": true,
}]
};
The script was tested with Typescript 2.1 (just released):
npm i typescript
npm i #types/node
./node_modules/.bin/tsc --lib es6 print-inferred-types.ts
node print-inferred-types.js sample.ts
output:
export interface RaceCarDriver {
id: number;
name: string;
cars: Car[];
}
export interface Car {
brand: string;
maxSpeed: number;
isWastingGazoline: boolean;
}
Here is the script: print-inferred-types.ts:
import * as ts from "typescript";
let fileName = process.argv[2];
function printInferredTypes(fileNames: string[], options: ts.CompilerOptions): void {
let program = ts.createProgram(fileNames, options);
let checker = program.getTypeChecker();
let knownTypes: {[name: string]: boolean} = {};
let pendingTypes: {name: string, symbol: ts.Symbol}[] = [];
for (const sourceFile of program.getSourceFiles()) {
if (sourceFile.fileName == fileName) {
ts.forEachChild(sourceFile, visit);
}
}
while (pendingTypes.length > 0) {
let pendingType = pendingTypes.shift();
printJsonType(pendingType.name, pendingType.symbol);
}
function visit(node: ts.Node) {
if (node.kind == ts.SyntaxKind.VariableStatement) {
(<ts.VariableStatement>node).declarationList.declarations.forEach(declaration => {
if (declaration.name.kind == ts.SyntaxKind.Identifier) {
let identifier = <ts.Identifier>declaration.name;
let symbol = checker.getSymbolAtLocation(identifier);
if (symbol) {
let t = checker.getTypeOfSymbolAtLocation(symbol, identifier);
if (t && t.symbol) {
pendingTypes.push({name: identifier.text, symbol: t.symbol});
}
}
}
});
}
}
function printJsonType(name: string, symbol: ts.Symbol) {
if (symbol.members) {
console.log(`export interface ${capitalize(name)} {`);
Object.keys(symbol.members).forEach(k => {
let member = symbol.members[k];
let typeName = null;
if (member.declarations[0]) {
let memberType = checker.getTypeOfSymbolAtLocation(member, member.declarations[0]);
if (memberType) {
typeName = getMemberTypeName(k, memberType);
}
}
if (!typeName) {
console.log(`// Sorry, could not get type name for ${k}!`);
} else {
console.log(` ${k}: ${typeName};`);
}
});
console.log(`}`);
}
}
function getMemberTypeName(memberName: string, memberType: ts.Type): string | null {
if (memberType.flags == ts.TypeFlags.String) {
return 'string';
} else if (memberType.flags == ts.TypeFlags.Number) {
return 'number';
} else if (0 !== (memberType.flags & ts.TypeFlags.Boolean)) {
return 'boolean';
} else if (memberType.symbol) {
if (memberType.symbol.name == 'Array' && (<ts.TypeReference>memberType).typeArguments) {
let elementType = (<ts.TypeReference>memberType).typeArguments[0];
if (elementType && elementType.symbol) {
let elementTypeName = capitalize(stripS(memberName));
if (!knownTypes[elementTypeName]) {
knownTypes[elementTypeName] = true;
pendingTypes.push({name: elementTypeName, symbol: elementType.symbol});
}
return `${elementTypeName}[]`;
}
} else if (memberType.symbol.name == '__object') {
let typeName = capitalize(memberName);
if (!knownTypes[typeName]) {
knownTypes[typeName] = true;
pendingTypes.push({name: typeName, symbol: memberType.symbol});
}
return typeName;
} else {
return null;
}
} else {
return null;
}
}
function capitalize(n: string) {
return n.charAt(0).toUpperCase() + n.slice(1);
}
function stripS(n: string) {
return n.endsWith('s') ? n.substring(0, n.length - 1) : n;
}
}
printInferredTypes([fileName], {
noEmitOnError: true, noImplicitAny: true,
target: ts.ScriptTarget.ES5, module: ts.ModuleKind.CommonJS
});
Found a npm package that converts a arbitrary JSON file without a schema into a TS interface:
https://www.npmjs.com/package/json-to-ts
The author also provided a VSCode plugin.
You can use an npm module instead of the web hosted solution:
https://www.npmjs.com/package/json-schema-to-typescript
If your JSON comes from an HTTP API and the API has an swagger code definition, you can generate a TypeScript client:
https://github.com/swagger-api/swagger-codegen#api-clients
If you json comes from a Java ot .Net backened, you can generate TypeScript from Java or C# classes:
http://type.litesolutions.net
https://github.com/raphaeljolivet/java2typescript
Using only sed and tsc
sed '1s#^#const foo = #' sample.json > sample.$$.ts
tsc sample.$$.ts --emitDeclarationOnly --declaration
Append const foo = to beginning of file
Using sed to replace (s) nothing (#^#) at the beginning of the first line (1) with const foo =
output to sample.$$.ts
the extension is the required to be .ts
$$ expands to the shells process id, handy for a temp file that is unlikely to overwrite stuff you care about
ask tsc to only emit a .d.ts typings file
this file has pretty much everything you want for the interface. You might need to replace a few strings and customize it in a way you want but most of the leg work is done