what i want to achieve are : create an API, which looking for into ElasticSearch. my programming language is Scala.
//myRoute.scala
val pencarianES =
{
post
{
path("cariES")
{
parameters("xQuery", "xNilai")
{
(yQuery, yNilai) =>
val PR = new ProsesRekomendasi
respondWithMediaType(MediaTypes.`application/json`)
{
complete
{
PR.ambilDariES(yQuery, yNilai)
}
}
}
}
}
}
//prosesRekomendasi.scala
class ProsesRekomendasi
{
val ESM = new ESManager
val CLT = ESM.client
def ambilDariES(pQuery:String, pNilai:String) =
{
CLT.prepareSearch("app_lr_portal_01")
.setTypes("lr01")
.setQuery(QueryBuilders.termQuery(s"$pQuery",s"$pNilai"))
.execute()
.actionGet()
}
}
the error are :
could not find implicit value for parameter marshaller:
spray.httpx.marshalling.ToResponseMarshaller[org.eleasticsearch.action.search.SearchResponse]
PR.ambilDariES(yQuery, yNilai)
i was looking for at google, and founded
DefaultMarshallers missing with scala and spray-routing
and then, im follow the instructions :
def ambilDariES(pQuery:String, pNilai:String)(implicit ec:ExecutionContext) =
{
CLT.prepareSearch("app_lr_portal_01")
.setTypes("lr01")
.setQuery(QueryBuilders.termQuery(s"$pQuery",s"$pNilai"))
.execute()
.actionGet()
}
finally, i get another error which are :
Cannot find an implicit ExecutionContext, either import scala.concurrent.ExecutionContext.Implicits.global or use a custom one
PR.ambilDariES(yQuery, yNilai)
any idea, how to deal with that? thanks for your help!
Although building RootJsonFormat for java classes is very tedious, here one example for one result. Just import in scope:
object SearchResultProtocol {
implicit object SearchResulJsonFormatObject extends RootJsonFormat[SearchResponse] {
def read(e: JsValue) = null
/* {
"_shards":{
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits":{
"total" : 1,
"hits" : [
{
"_index" : "twitter",
"_type" : "tweet",
"_id" : "1",
"_source" : {
"user" : "kimchy",
"postDate" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
}
]
}
}*/
private def getHits(arr: Array[SearchHit]) : JsArray = {
JsArray(arr.map { x => marshallHit(x) }.toVector)
}
private def marshallHit(hit: SearchHit) : JsValue = {
JsObject(Map("_index" -> JsString(hit.index()),
"_type" -> JsString(hit.getType),
"_id" -> JsString(hit.getId),
"source" -> JsString(hit.getSourceAsString)))
}
def write(sr: SearchResponse) = {
JsObject(Map("_shards" ->
JsObject(Map("total" -> JsNumber(sr.totalShards()),
"successful" -> JsNumber(sr.getSuccessfulShards()),
"failed" -> JsNumber(sr.getFailedShards()))),
"hits" -> JsObject(Map("total" -> JsNumber(sr.getHits.totalHits()),
"" -> getHits(sr.getHits.getHits)
))))
}
}
}
Related
My requirement is to create a dynamic resource connector in Kafka cluster. Below is my "connector.tf" file:
resource "confluent_connector" "source" {
environment {
id = confluent_environment.staging.id
}
kafka_cluster {
id = confluent_kafka_cluster.dedicated.id
}
config_sensitive = {
"salesforce.password" : var.source_salesforce_password,
"salesforce.password.token" : var.source_salesforce_password_token,
"salesforce.consumer.key" : var.source_salesforce_consumer_key,
"salesforce.consumer.secret" : var.source_salesforce_consumer_secret
}
config_nonsensitive = {
"connector.class" : "SalesforceCdcSource",
"kafka.auth.mode" : "KAFKA_API_KEY",
"salesforce.cdc.name" : "AccountChangeEvent",
"kafka.api.key" : confluent_api_key.app-manager-kafka-api-key.id,
"kafka.api.secret" : confluent_api_key.app-manager-kafka-api-key.secret,
"salesforce.instance" : var.source_salesforce_url,
"salesforce.username" : var.source_salesforce_username,
for_each = { for s in var.source_salesforce_connector_name : s.source_salesforce_connector_name => s },
"name" : each.value["source_salesforce_connector_name"],
"kafka.topic" : each.value["source_salesforce_topic_name"],
"output.data.format" : each.value["source_salesforce_data_format"],
"tasks.max" : each.value["source_salesforce_max_task"]
}
depends_on = [
confluent_kafka_topic.topic
]
lifecycle {
prevent_destroy = false
}
}
Variable declaration as below: variable.tf file
variable "source_salesforce_connector_name" {
type = list(map(string))
default = [{
"source_salesforce_connector_name" = "SalesforceCdcSourceConnector_0_TF"
}]
}
And I am running this execution from .tfvars file:
source_salesforce_connector_name = [
{
source_salesforce_connector_name = "SalesforceCdcSourceConnector_1_TF"
source_salesforce_topic_name = "json-topic-1"
source_salesforce_data_format = "JSON"
source_salesforce_max_task = "1"
},
]
Getting below error with the execution, please suggest how to pass for_each condition into the JSON configuration as highlighted above.
I tried with above steps and execution, however getting below error:
terraform plan -var-file="DEV/DEV.tfvars"
Error: each.value cannot be used in this context
on modules\confluent_kafka_cluster_dedicated\source_connector_salesforce_cdc.tf line 27, in resource "confluent_connector" "source":
27: "name" : each.value["source_salesforce_connector_name"],
28: "kafka.topic" : each.value["source_salesforce_topic_name"],
29: "output.data.format" : each.value["source_salesforce_data_format"],
30: "tasks.max" : each.value["source_salesforce_max_task"],*
A reference to "each.value" has been used in a context in which it unavailable, such as when
the configuration no longer contains the value in its "for_each" expression. Remove this
reference to each.value in your configuration to work around this error.
If you want multiple confluent_connector resources based on var.source_salesforce_connector_name, then your for_each should be outside of config_nonsensitive:
resource "confluent_connector" "source" {
for_each = { for s in var.source_salesforce_connector_name : s.source_salesforce_connector_name => s },
environment {
id = confluent_environment.staging.id
}
kafka_cluster {
id = confluent_kafka_cluster.dedicated.id
}
config_sensitive = {
"salesforce.password" : var.source_salesforce_password,
"salesforce.password.token" : var.source_salesforce_password_token,
"salesforce.consumer.key" : var.source_salesforce_consumer_key,
"salesforce.consumer.secret" : var.source_salesforce_consumer_secret
}
config_nonsensitive = {
"connector.class" : "SalesforceCdcSource",
"kafka.auth.mode" : "KAFKA_API_KEY",
"salesforce.cdc.name" : "AccountChangeEvent",
"kafka.api.key" : confluent_api_key.app-manager-kafka-api-key.id,
"kafka.api.secret" : confluent_api_key.app-manager-kafka-api-key.secret,
"salesforce.instance" : var.source_salesforce_url,
"salesforce.username" : var.source_salesforce_username,
"name" : each.value["source_salesforce_connector_name"],
"kafka.topic" : each.value["source_salesforce_topic_name"],
"output.data.format" : each.value["source_salesforce_data_format"],
"tasks.max" : each.value["source_salesforce_max_task"]
}
depends_on = [
confluent_kafka_topic.topic
]
lifecycle {
prevent_destroy = false
}
}
Hi I'm facing problem with serialization of map where key is an custom class.
data class KeyClass(val id: Int, val name: String) {
fun toJSON() = "\"KeyClass\": {\"id\":$id,\"name\":\"$name\"}"
}
Invocation:
fun method(): Map<KeyClass, List<Something>> = ...
My jackson Serializer ofc I'm also adding this as module in objectMapper:
class KeyClassSerializer : JsonSerializer<KeyClass>() {
override fun serialize(value: KeyClass, gen: JsonGenerator, serializers: SerializerProvider) {
gen.writeRawValue(value.toJSON())
}
}
class KeyClassSerializerModule : SimpleModule() {
init {
addKeySerializer(KeyClass::class.java, KeyClassSerializer())
}
}
And JSON I'm receiving is:
"\"KeyClass\": {\"id\":1,\"name\":\"Thomas\"}" : [Something:...]
I mean the value of map is serialized correctly but key isn't.
I assume the expected result is :
"KeyClass": {
"id": 1,
"name":"Thomas"
} : [...]
But it's not valid Json. You can still do something like :
{
"key" : {
"id": 1,
"name":"Thomas"
},
"value" : [...]
}
The following code gets the request body and validates and creates a json:
object ValidateDBConfigJson {
implicit val reads: Reads[ValidateDetails] = (
(JsPath \ "name").read[String].filter(JsonValidationError("Invalid name"))(_.length > 0) and
(JsPath \ "email").read[String].filter(JsonValidationError("Invalid email"))(_.length > 0) and
)(ValidateDetails.apply _)
}
def index() = Action { implicit request =>
val bodyAsJson = request.body.asJson.get
bodyAsJson.validate[ValidateDetails] match {
case success: JsSuccess[ValidateDetails] => {
Ok(Json.parse("succeeded!"))
}
case JsError(error) =>
BadRequest(JsError.toJson(error))
}
}
The json looks like this:
{
"obj.name": [
{
"msg": [
"error.expected.jsstring"
],
"args": []
}
],
"obj.email": [
{
"msg": [
"Invalid email"
],
"args": []
}
]
}
I want structured in the following format:
{
"ErrorMessages" :
[
"error.expected.jsstring",
"Invalid email"
]
}
Preamble: when I am parsing JSON using Play I prefer to use case class/objects rather than implicit reads, so this answer will cover that method of doing this. There may be a simpler method of doing this with implicit reads but I am not as familiar with implicit reads.
Firstly, define a case class for everything you will be taking from JSON:
object Input {
case class Err(msg: Seq[String], args: Seq[String])
object Err {
implicit val format: OFormat[Err] = Json.format[Err]
}
case class ValidateDetails(`obj.name`: Seq[Err], `obj.email`: Seq[Err])
object ValidateDetails {
implicit val format: OFormat[ValidateDetails] = Json.format[ValidateDetails]
}
}
Note: Play won't know how to handle user-defined case classes so I've made one for Err as well. The implicit val format: OFormat[ValidateDetails] = Json.format[ValidateDetails] and implicit val format: OFormat[Err] = Json.format[Err] lines are magic and do all of the reads/writes for you.
Next, define a case class for your output JSON and define a function which will turn your input case class into your output one:
object Output {
case class OutputJson(`ErrorMessages`: Seq[String])
object OutputJson {
implicit val format: OFormat[OutputJson] = Json.format[OutputJson]
}
// take msg Seq from name & email and add together into single Seq
def inputToOutput(input: Input.ValidateDetails): OutputJson = {
OutputJson(input.`obj.name`.flatMap(_.msg) ++ input.`obj.email`.flatMap(_.msg))
}
}
Finally, put this into a method which maps to a POST route in your routes file:
def index() = Action { implicit request =>
val bodyAsJson = request.body.asJson.get
bodyAsJson.validate[Input.ValidateDetails] match {
case success: JsSuccess[Input.ValidateDetails] =>
// turn the JSON into the Output case class and parse that as JSON
val output: JsValue = Json.toJson(Output.inputToOutput(success.value))
Ok(output)
case JsError(error) =>
BadRequest(JsError.toJson(error))
}
}
Now, if you run the Play app on port 9000 and POST to http://localhost:9000/ with the below JSON body...
{
"obj.name": [
{
"msg": [
"error.expected.jsstring"
],
"args": []
}
],
"obj.email": [
{
"msg": [
"Invalid email"
],
"args": []
}
]
}
...the output will be:
{
"ErrorMessages": [
"error.expected.jsstring",
"Invalid email"
]
}
I hope this answers your question.
I'm trying to test bellow using Play 2.4.2 , Spec 2 ,
" test response Returns a json Array" in new WithApplication {
val response = route(FakeRequest(GET, "/myservice/xx")).get
// ??? test response is a json array
}
What would be the way to test this scenario ?
Here is a possibility
Controller
#Singleton
class BarryController extends Controller{
def barry = Action { implicit request =>
val json: JsValue = Json.parse("""
{
"residents" : [ {
"name" : "Fiver",
"age" : 4,
"role" : null
}, {
"name" : "Bigwig",
"age" : 6,
"role" : "Owsla"
} ]
}
""")
Ok(json)
}
}
Test
import org.specs2.mutable._
import play.api.mvc._
import play.api.test.FakeRequest
import play.api.test.Helpers._
import play.api.test.WithApplication
import controllers._
import play.api.libs.json._
class BarryControllerSpec extends Specification {
"controllers.BarryController" should {
val expectedJson: JsValue = Json.parse("""
{
"residents" : [ {
"name" : "Fiver",
"age" : 4,
"role" : null
}, {
"name" : "Bigwig",
"age" : 6,
"role" : "Owsla"
} ]
}
""")
"respond with JsArray for /barry" in new WithApplication {
val result = new controllers.BarryController().barry()(FakeRequest())
status(result) must equalTo(OK)
contentType(result) must equalTo(Some("application/json"))
//testing class is JsArray. The .get is necessary to get type out of JsLookupResult/JsDefined instance
(contentAsJson(result) \ "residents").get must haveClass[JsArray]
//testing JSON is equal to expected
contentAsJson(result) must equalTo(expectedJson)
//test an attribute in JSON
val residents = (contentAsJson(result) \ "residents").get
(residents(0) \ "age").get must equalTo(JsNumber(4))
}
}
}
Hopefully this gives you some ideas on what you can test or what you might want to do.
i'm writing a Play 2.3.2 application using Reactivemongo driver (with Scala).
I've the recommendation.user collection that store all the user data.
One document has the following form:
{
"_id" : ObjectId("542e67e07f724fc2af28ba75"),
"id" : "",
"email" : "luigi#gmail.com",
"tags" : [
{
"tag" : "Paper Goods:Liners - Baking Cups",
"weight" : 2,
"lastInsert" : 1412327492874
},
{
"tag" : "Vegetable:Carrots - Jumbo",
"weight" : 4,
"lastInsert" : 1412597883569
},
{
"tag" : "Paper Goods:Lialberto- Baking Cups",
"weight" : 1,
"lastInsert" : 1412327548205
},
{
"tag" : "Fish:Swordfish Loin Portions",
"weight" : 3,
"lastInsert" : 1412597939124
},
{
"tag" : "Vegetable:Carrots - alberto#gmail.com",
"weight" : 2,
"lastInsert" : 1412597939124
}
]
}
Now i'm writing a method that returns all the tags of a particular user.
I return a JSOn responce.
In play how can i create a dynamic json??
My return json has the following form:
{
"tags": [
{"tag": "tag1"},
{"tag": "tag2"}
]
}
This is my method implementation:
def userTag(user: String) = Action.async {
//obtain all the users saved in the db.
val allUser : Future[Option[User]] = Users.find(Json.obj("email" -> user)).one
val futureComputation = allUser map {
(user: Option[User]) =>
user match {
case Some(x) => x.tags match {
case Some(userTags) => val tags: List[Tag] = userTags.map{tag: (Tag, Double, Long) => tag._1} //obtain all the Tag objects
//here for every element in the tags variable i want to add it add the json.
case None => Ok(Json.obj()) //return an empty json.
}
case None => Ok(Json.obj()) //return an empty json
}
}
futureComputation
}
How can i solve my problem??
Solved using:
def userTag(user: String) = Action.async {
//obtain all the users saved in the db.
val allUser : Future[Option[User]] = Users.find(Json.obj("email" -> user)).one
val futureComputation = allUser map {
(user: Option[User]) =>
user match {
case Some(x) => x.tags match {
case Some(userTags) => val tags = userTags.map{tag: (Tag, Double, Long) => val t = tag._1; t.category + ":" + t.attr} //obtain all the Tag objects
val arrayTags = for(tag <- tags) yield{Json.obj("tag" -> tag)} //crete the array json
Ok(Json.obj("tags" -> arrayTags)) //return the json corresponding to the user.
case None => Ok(Json.obj()) //return an empty json.
}
case None => Ok(Json.obj()) //return an empty json
}
}
//return the Future computation JSON responce.
futureComputation
}