i got stuck on splitting the values in json. I want to split the values for the key: ServerIPList. Here's the JSON format:
{
ServerIPList : [
<PrivateIP> <HOSTNAME> <REGION> <AWS ACCOUNT>
{ 172.00.00.00 ,ip-172-00-00-00.ec2.internal ,us-east-1,123456789123 } ,
] ,
Operation : start,
SNowTicket : RITM00001
}
Here's my code:
import groovy.json.*
def message
pipeline {
agent any
options {
buildDiscarder logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '', numToKeepStr: '5')
}
stages {
stage('Recieve SQS Message') { steps {
script {
def aws = new AwsUtil()
def success = aws.receiveSqsMessages(env.AWS_SQS_QUEUE, 1, { json ->
message = new Util().jsonSlurper(json.Body)
println message
println message.ServerIPList
println message.Operation
result = sh (
script: "python ec2InstanceState.py ${message.ServerIPList} ${message.Operation}",
returnStdout: true
).trim()
output of my code:
Body:{
"ServerIPList" : "172.24.8.73","ip-172-00-00-00.ec2.internal","us-east-1","123456789123"
"Operation" : "start",
"SNowTicket" : "RITM1062357"
},
"172.24.8.73","ip-172-00-00-00.ec2.internal","us-east-1","123456789123"
"start"
"python ec2InstanceState.py "172.24.8.73","ip-172-00-00-00.ec2.internal","us-east-1","123456789123" start,
Now, here i would like to grep only PrivateIp from the key: "ServerIPList". How to get it:
python ec2InstanceState.py 172.24.8.73 start
Please refer to below code
data "amazon-ami" "ubuntu" {
most_recent = true
filters = {
virtualization-type = "hvm"
name = "ubuntu/images/*ubuntu-focal-20.04-amd64-server-*"
root-device-type = "ebs"
}
owners = ["099720109477"]
}
source "null" "one" {
communicator = "none"
}
build {
name = "source-ouput"
sources = ["source.null.one"]
provisioner "shell-local" {
inline = [
"echo ${data.amazon-ami.ubuntu}", ##### <--- problem happens here
]
}
}
As you can see in the code, I want to print out the return value of data "amazon-ami" "ubuntu"
If I run this with packer build .. error occurs like below
Error: Failed preparing provisioner-block "shell-local" ""
on main.pkr.hcl line 20:
(source code not available)
main.pkr.hcl:22,15-37: Invalid template interpolation value; Cannot include the
given value in a string template: string required.
Is there any solution to print the object that returned by data?
I have a function in a Jenkins shared library which builds and returns a map object (class java.util.LinkedHashMap):
#!/usr/bin/env groovy
def call(Map config) {
script {
echo "Building nested map"
def MAP_VAR = [:]
MAP_VAR.put("one", [:])
MAP_VAR.put("two", [:])
MAP_VAR.get("one").put("a", "b")
MAP_VAR.get("two").put("c", "d")
echo "Returning: ${MAP_VAR}"
echo "Type: ${MAP_VAR.getClass()}"
return MAP_VAR
}
}
When this function runs the log output shows:
Returning: [one:[a:b], two:[c:d]]
Type: class java.util.LinkedHashMap
But when I call the function and assign the return value to a variable, it ends up being a string (class java.lang.String):
#!/usr/bin/env groovy
library identifier: 'library#main',
changelog: false,
retriever: modernSCM([ $class: 'GitSCMSource',
remote: 'git#github.com:org/repo.git',
credentialsId: 'credentials' ])
pipeline {
agent none
stages {
stage('Get map') {
agent any
steps {
script {
env.MAP_VAR = getMapVar()
}
}
}
stage('Check map') {
agent any
steps {
script {
echo "Value: ${env.MAP_VAR}"
echo "Type: ${env.MAP_VAR.getClass()}"
}
}
}
}
}
The output shows:
Value: {one={a=b}, two={c=d}}
Type: class java.lang.String
Ultimately I'm trying to access the map's properties in multiple stages within the Jenkinsfile. If I try this:
echo "Value: ${env.MAP_VAR['one']}"
I get:
groovy.lang.MissingPropertyException: No such property: one for class: java.lang.String
I've tried:
def env.MAP_VAR = getMapVar()
But that results in:
WorkflowScript: 59: unexpected token: def # line 59, column 21.
def env.MAP_VAR = getMapVar()
^
I've also tried:
def Map env.MAP_VAR = getMapVar()
But that results in:
WorkflowScript: 60: unexpected token: Map # line 60, column 25.
def Map env.MAP_VAR = getMapVar()
^
How can I get the Map/LinkedHashMap from the function as a Map/LinkedHashMap (which would allow me to access the properties/values of the map contents) and assign it to a global variable which can be used in all stages?
def env.MAP_VAR = getMapVar()
This will not work as you are trying to define a key MAP_VAR in env map.
def Map env.MAP_VAR = getMapVar()
This also will not work as you are trying to define MAP_VAR key in env map as def and also MAP. This is similar to doing something like String Integer my_var_name = something()
env.MAP_VAR = getMapVar()
You are storing the return value as a string in env environment variable map. Jenkins output is accurate given the code.
To achieve what you are trying to do, you can store it as a variable and it will work fine. In the following example, the variable goes into Groovy's script binding and will be available in the next stage. You can also define def MAP_VAR outside the pipeline block to get the same result.
#!/usr/bin/env groovy
library identifier: 'library#main',
changelog: false,
retriever: modernSCM([ $class: 'GitSCMSource',
remote: 'git#github.com:org/repo.git',
credentialsId: 'credentials' ])
pipeline {
agent none
stages {
stage('Get map') {
agent any
steps {
script {
MAP_VAR = getMapVar()
}
}
}
stage('Check map') {
agent any
steps {
script {
echo "Value: ${MAP_VAR}"
echo "Type: ${MAP_VAR.getClass()}"
}
}
}
}
}
This will print
Value: [one:[a:b], two:[c:d]]
Type: class java.util.LinkedHashMap
In stage two,
echo "${MAP_VAR['one']}"
will output
[a:b]
I'm trying to fetch and deserialize some data that is being hosted on github.
{
"Meals": [
{
"id": "1598044e-5259-11e9-8647-d663bd870b02",
"name": "Tomato pasta",
"quantity": [{
"quantity": 1 },
{
"quantity": 2
},
{
"quantity": 3
}],
"availableFromDate": "1605802429",
"expiryDate": "1905802429",
"info": "Vegetarian",
"hot": false,
"locationLat": 57.508865,
"locationLong": -6.292,
"distance": null
},
{
"id": "2be2d854-a067-43ec-a488-2e69f0f2a624",
"name": "Pizza",
"quantity": [{
"quantity": 1 },
{
"quantity": 2
},
{
"quantity": 3
}
],
"availableFromDate": "1605802429",
"expiryDate": "1905902429",
"info": "Meat",
"hot": false,
"locationLat": 51.509465,
"locationLong": -0.135392,
"distance": null
}
]
}
If I spin up a json-server locally then it works perfectly, so I know that my data class is not the problem. However when I try to do the same thing from that github link I get this error:
Error Domain=KotlinException Code=0 "No transformation found: class io.ktor.utils.io.ByteChannelNative -> class kotlin.collections.List
I have a feeling it might be to do with setting a ContentType or something along those lines but I haven't had any success specifying that so far.
Here is my code to make the request:
class MealApi {
private val httpClient = HttpClient {
install(JsonFeature) {
val json = Json { ignoreUnknownKeys = true }
serializer = KotlinxSerializer(json)
}
}
suspend fun getAllMeals(): List<Meal> {
return httpClient.get(endpoint)
}
}
and here is my data class just for completeness:
#Serializable
data class Meal(
#SerialName("id")
val id: String,
#SerialName("name")
val name: String,
#SerialName("quantity")
val quantity: List<Quantity>,
#SerialName("availableFromDate")
var availableFromDate: String,
#SerialName("expiryDate")
var expiryDate: String,
#SerialName("info")
val info: String,
#SerialName("hot")
val hot: Boolean,
#SerialName("locationLat")
val locationLat: Float,
#SerialName("locationLong")
val locationLong: Float,
#SerialName("distance")
var distance: Double? = null
)
#Serializable
data class Quantity(
#SerialName("quantity")
val quantity: Int
)
UPDATE
I've found that this server https://gitcdn.link/ allows you to serve your raw github files with the right Content-Type.
I've searched a lot how to change the server response headers (to change the plain/text one to application/json) but it seems that ktor actually doesn't allow to do that:
https://youtrack.jetbrains.com/issue/KTOR-617
https://youtrack.jetbrains.com/issue/KTOR-580
A nice way should be to allow the ResponseObserver to change the server response headers and pass through the modify response. But you can't actually.
Your problem depends, as you pointed out, from the fact that the raw github page provides an header Content-Type=plain/text instead of ContentType=application/json.
So IRL when you are running your API in a real server this won't occur as you'll take care to put the right content type at server level.
But if you want a workaround to this you could rewrite your api call in this way:
suspend fun getAllMealsWithFallback() {
var meals: Meals? = null
try {
meals = httpClient.get(endpoint)
} catch (e: NoTransformationFoundException) {
val mealsString: String = httpClient.get(endpoint)
val json = kotlinx.serialization.json.Json {
ignoreUnknownKeys = true
}
meals = json.decodeFromString(mealsString)
} finally {
println("Meals: ${meals?.meals}")
}
}
I had to add this class to conform to the json text you have provided in the github link.
#Serializable
data class Meals(
#SerialName("Meals")
val meals: List<Meal>,
)
Try this:
install(JsonFeature) {
serializer = KotlinxSerializer(KotlinJson { ignoreUnknownKeys = true })
acceptContentTypes = acceptContentTypes + ContentType.Any
}
If you'd like to accept all content types. Or Use ContentType.Text.Any, ContentType.Text.Html if you preferred.
In case the issue is the Content-Type:
You can alter the list of response content types, for which the KotlinxSerializer gets active, by extending the JsonFeature block to:
install(JsonFeature) {
val json = Json { ignoreUnknownKeys = true }
serializer = KotlinxSerializer(json)
receiveContentTypeMatchers += object : ContentTypeMatcher {
override fun contains(contentType: ContentType): Boolean =
contentType == ContentType("text", "plain")
}
}
If you're using Ktor 2.0, you would need ContentNegotiation plugin instead of JsonFeature.
For example if you use Gson:
install(io.ktor.client.plugins.contentnegotiation.ContentNegotiation) {
gson()
}
As a workaround for ktor version 2.0.3 you can create you own AppContentNegotiation class and in the scope.responsePipeline.intercept provide needed contentType
/**
* A plugin that serves two primary purposes:
* - Negotiating media types between the client and server. For this, it uses the `Accept` and `Content-Type` headers.
* - Serializing/deserializing the content in a specific format when sending requests and receiving responses.
* Ktor supports the following formats out-of-the-box: `JSON`, `XML`, and `CBOR`.
*
* You can learn more from [Content negotiation and serialization](https://ktor.io/docs/serialization-client.html).
*/
public class AppContentNegotiation internal constructor(
internal val registrations: List<Config.ConverterRegistration>
) {
/**
* A [ContentNegotiation] configuration that is used during installation.
*/
public class Config : Configuration {
internal class ConverterRegistration(
val converter: ContentConverter,
val contentTypeToSend: ContentType,
val contentTypeMatcher: ContentTypeMatcher,
)
internal val registrations = mutableListOf<ConverterRegistration>()
/**
* Registers a [contentType] to a specified [converter] with an optional [configuration] script for a converter.
*/
public override fun <T : ContentConverter> register(
contentType: ContentType,
converter: T,
configuration: T.() -> Unit
) {
val matcher = when (contentType) {
ContentType.Application.Json -> JsonContentTypeMatcher
else -> defaultMatcher(contentType)
}
register(contentType, converter, matcher, configuration)
}
/**
* Registers a [contentTypeToSend] and [contentTypeMatcher] to a specified [converter] with
* an optional [configuration] script for a converter.
*/
public fun <T : ContentConverter> register(
contentTypeToSend: ContentType,
converter: T,
contentTypeMatcher: ContentTypeMatcher,
configuration: T.() -> Unit
) {
val registration = ConverterRegistration(
converter.apply(configuration),
contentTypeToSend,
contentTypeMatcher
)
registrations.add(registration)
}
private fun defaultMatcher(pattern: ContentType): ContentTypeMatcher =
object : ContentTypeMatcher {
override fun contains(contentType: ContentType): Boolean =
contentType.match(pattern)
}
}
/**
* A companion object used to install a plugin.
*/
#KtorDsl
public companion object Plugin : HttpClientPlugin<Config, AppContentNegotiation > {
public override val key: AttributeKey<AppContentNegotiation> =
AttributeKey("ContentNegotiation")
override fun prepare(block: Config.() -> Unit): AppContentNegotiation {
val config = Config().apply(block)
return AppContentNegotiation(config.registrations)
}
override fun install(plugin: AppContentNegotiation, scope: HttpClient) {
scope.requestPipeline.intercept(HttpRequestPipeline.Transform) { payload ->
val registrations = plugin.registrations
registrations.forEach { context.accept(it.contentTypeToSend) }
if (subject is OutgoingContent || DefaultIgnoredTypes.any { it.isInstance(payload) }) {
return#intercept
}
val contentType = context.contentType() ?: return#intercept
if (payload is Unit) {
context.headers.remove(HttpHeaders.ContentType)
proceedWith(EmptyContent)
return#intercept
}
val matchingRegistrations =
registrations.filter { it.contentTypeMatcher.contains(contentType) }
.takeIf { it.isNotEmpty() } ?: return#intercept
if (context.bodyType == null) return#intercept
context.headers.remove(HttpHeaders.ContentType)
// Pick the first one that can convert the subject successfully
val serializedContent = matchingRegistrations.firstNotNullOfOrNull { registration ->
registration.converter.serialize(
contentType,
contentType.charset() ?: Charsets.UTF_8,
context.bodyType!!,
payload
)
} ?: throw ContentConverterException(
"Can't convert $payload with contentType $contentType using converters " +
matchingRegistrations.joinToString { it.converter.toString() }
)
proceedWith(serializedContent)
}
scope.responsePipeline.intercept(HttpResponsePipeline.Transform) { (info, body) ->
if (body !is ByteReadChannel) return#intercept
if (info.type == ByteReadChannel::class) return#intercept
// !!!!!!! Provide desired content type here
val contentType = <HERE> // default implementation is - context.response.contentType() ?: return#intercept
val registrations = plugin.registrations
val matchingRegistrations = registrations
.filter { it.contentTypeMatcher.contains(contentType) }
.takeIf { it.isNotEmpty() } ?: return#intercept
// Pick the first one that can convert the subject successfully
val parsedBody = matchingRegistrations.firstNotNullOfOrNull { registration ->
registration.converter
.deserialize(context.request.headers.suitableCharset(), info, body)
} ?: return#intercept
val response = HttpResponseContainer(info, parsedBody)
proceedWith(response)
}
}
}
}
And then install it for HttpClient
HttpClient {
install(AppContentNegotiation) {
json(json)
addDefaultResponseValidation()
}
}
In my case, this exception was thrown when I was attempting to access the body() potion of the HTTP response as such:
val httpResponse = httpClient.post(urlString) { ... }
val body = httpResponse.body<YourExpectedSerializableResponseType>
In the happy path scenario, the server would return a body that matched YourExpectedSerializableResponseType and everything would work as expected.
However, for a particular edge case that returned a different (still considered successful) status code, the server returned an empty response body. Since the client was expecting a response body and in this case, there wasn't any, this exception was thrown because it could not serialize an empty response body to the expected type YourExpectedSerializableResponseType.
My recommendation: In addition to ensuring your server is returning the type you expect to serialize/consume on your client, confirm your server is actually returning an object.
Internal Dialogue: I wonder if this exception could be more clear in this case as the issue is more so that the client expected a response body to exist and less so that an empty response body ("") couldn't be serialized into an expected type - especially given that an empty response body isn't even valid JSON. Hm. 🤔
I would like to know that how can I return json response data from Play(2.2.x) Scala controller class to display on my view page ? I have json objects in Postgresql database(table name: "test" and having: id and name). Please provide me any solutions for it.
I have tried the following cases(a and b), but I am not sure why I am not getting the response(like: names) on my controller, so I can show them on my view page ? since I am very new to Play/Scala and Postgresql.
case a. If I give like:
model:
def getTestValuesFromTable() = {
DB.withConnection { implicit connection =>
val selectJson =SQL("select name from test").on().apply().collect {
case Row(id:Long, Some(name:String)) =>
new TestContent(name)
}
//.head
//JsObject(selectJson().map { row => row[Long]("id").toString -> JsString(row[String]("name")) }.toSeq)
}
}
controller:
def getTest = Action {
val response = TestContent.getTestValuesFromTable()
Ok("Done")
//Ok(response)
}
Output is: Done(application is executing fine without any exceptions, of course json data is not coming since I am returning: Done only, so getting output: "Done")
case b. If I do like this: getting error: not enough arguments for method apply: (n: Int)models.Testname in trait LinearSeqOptimized. Unspecified value parameter n. I really not sure how can I get my response for it ?
controller:
def getTest = Action {
val response = TestContent.getTestValuesFromTable()
// Ok("Done")
Ok(response)
}
model:
def getTestValuesFromTable(): JsValue = {
DB.withConnection { implicit connection =>
val selectJson = SQL("select * from test")
JsObject(selectJson().map { row => row[Long]("id").toString -> JsString(row[String]("name")) }.toSeq)
// val selectJson =SQL("select name from test").on().apply().collect {
// case Row(id:Long, Some(name:String)) =>
// new TestContent(name)
// }
//.head
JsObject(selectJson().map { row => row[Long]("id").toString -> JsString(row[String]("name")) }.toSeq)//not enough arguments for method apply: (n: Int)models.Testname in trait LinearSeqOptimized. Unspecified value parameter n.
}
}
Please let me know how to get my response ?
getJsonValuesFromTable method return nothing (Unit). To fix it change definition of this method to
def getJsonValuesFromTable(testContent: TestContent) = {
or explicitly setting type:
def getJsonValuesFromTable(testContent: TestContent): Unit = {
Also as a next step to let client know that you are returning json, you should set content type:
Ok(Json.obj(response)).as("application/json")