I am trying to parse and transform a json file so that it can fit a DB structure. I have the following json file.
{
"request": {
"Target": "AR",
"Format": "jsonp",
"Service": "HasOff",
"Version": "2",
"NetworkId": "xyz",
"Method": "getConversions",
"api_key": "xyz",
"fields": [
"Offer.name",
"Stat.datetime",
"Stat.conversion_status",
"Stat.approved_payout",
"Stat.ip",
"Stat.ad_id",
"Stat.affiliate_info1",
"Stat.offer_id"
],
"data_start": "2015-04-16 23:59:59",
"data_end": "2015-04-18 00:00:00",
"callback": "angular.callbacks._3",
"_ga": "GA1.2.1142892596.1429262595",
"_gat": "1"
},
"response": {
"status": 1,
"httpStatus": 200,
"data": {
"page": 1,
"current": 100,
"count": 521,
"pageCount": 6,
"data": [
{
"Offer": {
"name": "ABC Company"
},
"Stat": {
"datetime": "2015-04-19 12:09:01",
"conversion_status": "approved",
"approved_payout": "26.94000",
"ip": "111.11.11.1",
"ad_id": "123456",
"affiliate_info1": "123456",
"offer_id": "260"
}
},
{
"Offer": {
"name": "ABC Company"
},
"Stat": {
"datetime": "2015-04-11 01:01:38",
"conversion_status": "approved",
"approved_payout": "44.94000",
"ip": "49.204.222.117",
"ad_id": "123456",
"affiliate_info1": "123456",
"offer_id": "260"
}
},
To process it I wrote a case class, model the class and try to convert the json into the model as given in -- https://www.playframework.com/documentation/2.2.x/ScalaJson. The code snippet is given below. I have reduced the number of parameters in the case class for sake of brevity.
case class ConversionReport(offerName: String, date: DateTime)
implicit val readConversion: Reads[ConversionReport] = (
(__ \ "response" \ "data" \ "data" \ "Offer" \ "name").read[String] and
(__ \ "response" \ "data" \ "data" \ "Stat" \ "datetime").read[DateTime]
) (ConversionReport)
def getConversionReport(){
val conversionUrl: String = "http://some_link"
val holder = WS.url(conversionUrl).get()
val result = Await.result(holder, Duration(10, "second")).json
val conversionResult: JsResult[ConversionReport] = result.validate[ConversionReport]
println ("xxxx The conversion result is " +conversionResult)
}
When I execute the code, I am getting the error --
xxxx The conversion result is JsError(List((/offerName,List(ValidationError(error.path.missing,WrappedArray()))), (/date,List(ValidationError(error.path.missing,WrappedArray())))))
I guess, I am not able to properly model the json file.
Any help is appreciated.
In this case data is an array, so your approach will not work. You will have to change few things.
import play.api.libs.json._
import play.api.libs.json.Reads._
case class ConversionReport(offerName: String, date: DateTime)
implicit val readConversion: Reads[ConversionReport] = (
(__\ "Offer" \ "name").read[String] and
(__ \ "Stat" \ "datetime").read[DateTime]
) (ConversionReport)
case class FullReport( reportList: Seq[ ConversionReport ] )
implicit val readFullReport: Reads[ FullReport ] = (
(__ \ "response" \ "data" \ "data").lazyRead( Reads.seq[ ConversionReport ]( readConversion ) )
) ( FullReport )
def getConversionReport(){
val conversionUrl: String = "http://some_link"
val holder = WS.url(conversionUrl).get()
val result = Await.result(holder, Duration(10, "second")).json
val conversionResult: JsResult[FullReport] = result.validate[FullReport]
println ("xxxx The conversion result is " +conversionResult)
}
Related
This is the input json which I am getting which is nested json structure and I don't want to map directly to class, need custom parsing of some the objects as I have made the case classes
{
"uuid": "b547e13e-b32d-11ec-b909-0242ac120002",
"log": {
"Response": {
"info": {
"receivedTime": "2022-02-09T00:30:00Z",
"isSecure": "Yes",
"Data": [{
"id": "75641",
"type": "vendor",
"sourceId": "3",
"size": 53
}],
"Group": [{
"isActive": "yes",
"metadata": {
"owner": "owner1",
"compressionType": "gz",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "1"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner12",
"compressionType": "snappy",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "2"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner123",
"compressionType": "snappy",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "4"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner124",
"compressionType": "snappy",
"comments": "someComments",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "4"
}
]
}
}
}
}
Code that I am trying play json also tried circe . Please help ..New to scala world
below is object and case class
case class DataCatalog(uuid: String, data: Seq[Data], metadata: Seq[Metadata])
object DataCatalog {
case class Data(id: String,
type: String,
sourceId: Option[Int],
size: Int)
case class Metadata(isActive: String,
owner: String,
compressionType: String,
comments: String,
createdDate: String,
updatedDate: String
)
def convertJson(inputjsonLine: String): Option[DataCatalog] = {
val result = Try {
//val doc: Json = parse(line).getOrElse(Json.Null)
//val cursor: HCursor = doc.hcursor
//val uuid: Decoder.Result[String] = cursor.downField("uuid").as[String]
val lat = (inputjsonLine \ "uuid").get
DataCatalog(uuid, data, group)
}
//do pattern matching
result match {
case Success(dataCatalog) => Some(dataCatalog)
case Failure(exception) =>
}
}
}
Any parsing api is fine.
If you use Scala Play, for each case class you should have an companion object which will help you a lot with read/write object in/from json:
object Data {
import play.api.libs.json._
implicit val read = Json.reads[Data ]
implicit val write = Json.writes[Data ]
def tupled = (Data.apply _).tupled
}
object Metadata {
import play.api.libs.json._
implicit val read = Json.reads[Metadata ]
implicit val write = Json.writes[Metadata ]
def tupled = (Metadata.apply _).tupled
}
Is required as each companion object to be in same file as the case class. For your json example, you need more case classes because you have a lot of nested objects there (log, Response, info, each of it)
or, you can read the field which you're interested in as:
(jsonData \ "fieldName").as[CaseClassName]
You can try to access the Data value as:
(jsonData \ "log" \ "Response" \ "info" \ "Data").as[Data]
same for Metadata
I'm a newbie to scala/play and I've been trying to update (add a new id to) the query array in the sections[1] of the JSON but I've had no success traversing the JSON as I have little knowledge of transformers and how to use it.
"definitions": [
{
"sections": [
{
"priority": 1,
"content": {
"title": "Driver",
"links": [
{
"url": "https://blabla.com",
"text": "See all"
}
]
},
"SearchQuery": {
"options": {
"aggregate": true,
"size": 20,
},
"query": "{\"id\":{\"include\":[\"0wxZ4Nr2\", \"0wxZbNr2\", \"6WZOPMw1\"}}"
}
},
{
"priority": 2,
"content": {
"title": "Deliver",
"links": [
{
"url": "https://blabla.com",
"text": "See all"
}
]
},
"SearchQuery": {
"options": {
"aggregate": true,
"size": 20,
},
"query": "{\"id\":{\"include\":[\"2W12Q2wq\", \"Nwq09lW3\", \"QweNN2d9\"]}}"
}
}
]
}
Any suggestions on how I can achieve this. My goal is to put values inside the specific fields of JSON array. I am using play JSON library throughout my application?
As you noticed if you use PlayJSON you can use Json Transformers
Updating a field would work like this:
val queryUpdater = (__ \ "definitions" \ 1 \ "SearchQuery" \ "query").json.update(
of[JsString].map {
case JsString(value) =>
val newValue: String = ... // calculate new value
JsString(newValue)
}
)
json.transform(queryUpdater)
If you needed to update all queries it would be more like:
val updateQuery = (__ \ "SearchQuery" \ "query").json.update(
of[JsString].map {
case JsString(value) =>
val newValue: String = ... // calculate new value
JsString(newValue)
}
)
val updateQueries = (__ \ "definitions").json.update(
of[JsArray].map {
case JsArray(arr) =>
JsArray(arr.map(_.transform(updateQuery)))
}
)
json.transform(updateQueries)
I am new to Scala and please help me out.
I have 2 Json file. I want to sort first json with a key from the second json.
For eg:
First Json
{
"id": 1,
"response" : [{
"user_id" : 1,
"products" : [
{
"product_id": 10,
"price": 200
},
{
"product_id": 13,
"price": 210
},
{
"product_id": 9,
"price": 320
}
]
},{
"user_id" : 2,
"products" : [
{
"product_id": 15,
"price": 200
},
{
"product_id": 13,
"price": 210
},
{
"product_id": 8,
"price": 320
}
]
}
]
}
And My Second Json
{
"sort": [
{
"user_id": 1,
"products": [
{
"id": 8,
"rank": 5
},
{
"id": 9,
"rank": 1
},
{
"id": 10,
"rank": 3
},
{
"id": 13,
"rank": 2
},{
"id": 15,
"rank": 6
},{
"id": 17,
"rank": 4
},{
"id": 20,
"rank": 7
},{
"id": 21,
"rank": 8
},{
"id": 23,
"rank": 9
}
]
},{
"user_id": 2,
"products": [
{
"id": 8,
"rank": 5
},
{
"id": 9,
"rank": 1
},
{
"id": 10,
"rank": 3
},
{
"id": 13,
"rank": 2
},{
"id": 15,
"rank": 6
},{
"id": 17,
"rank": 4
},{
"id": 20,
"rank": 7
},{
"id": 21,
"rank": 8
},{
"id": 23,
"rank": 9
}
]
}
]
}
I want to sort my first json with respect to the rank I have in second Json.
Output should be like each user should have his products in sorted order based on the rank that is specified for each user on the second JSON.
This is so far what I have tried
def sortedRes() = Action.async {
val url = //1st json url
val sortUrl = //2nd json url
ws.url(url).get().map { response =>
val value: JsValue = Json.parse(response.body)
val result: Either[Exception, SampleResponses] = value.validate[SampleResponses] match {
case JsSuccess(searchResponse, _) =>
Right(searchResponse)
case JsError(errors) =>
Left(new Exception("Couldn't parse Search API response"))
}
val values: List[SampleResponse] = result.right.get.responses
ws.url(sortUrl).get().map { response =>
val sorted: JsValue = Json.parse(response.body)
val sortRespResult: Either[Exception, Sort] = sorted.validate[Sort] match {
case JsSuccess(sortResponse, _) =>
Right(sortResponse)
case JsError(errors) =>
Left(new Exception("Couldn't parse because of these errors : " + errors))
}
val prodRankDetails: List[SampleRank] = sortRespResult.right.get.sort
println("prod = " + prodRankDetails.head.products.sortWith(_.rank > _.rank))
}
Ok(Json.toJson(result.right.get))
}
}
In the last print statement I got the second json's first users product sorted. What I am trying to get is my first json sorted based on the second user.
Here is my model class
package models
import play.api.libs.functional.syntax._
import play.api.libs.json.Reads._
import play.api.libs.json._ // Combinator syntax
object sample {
case class SampleProduct(productId:Int, price: Int)
case class SampleResponse(userId: Int, products: List[SampleProduct])
case class SampleResponses(id: Int, responses: List[SampleResponse])
case class SampleRankedProduct(id: Int, rank: Int)
case class SampleRank(userId: Int, products: List[SampleRankedProduct])
case class Sort(sort: List[SampleRank])
implicit val productReads: Reads[SampleProduct] = (
(JsPath \ "product_id").read[Int] and
(JsPath \ "price").read[Int]
)(SampleProduct.apply _)
implicit val productWrites: Writes[SampleProduct] = (
(JsPath \ "product_id").write[Int] and
(JsPath \ "price ").write[Int]
)(unlift(SampleProduct.unapply))
implicit val responseReads: Reads[SampleResponse] = (
(JsPath \ "user_id").read[Int] and
(JsPath \ "products").read[List[SampleProduct]]
)(SampleResponse.apply _)
implicit val responseWrites: Writes[SampleResponse] = (
(JsPath \ "user_id").write[Int] and
(JsPath \ "products").write[List[SampleProduct]]
)(unlift(SampleResponse.unapply))
implicit val responsesReads: Reads[SampleResponses] = (
(JsPath \ "id").read[Int] and
(JsPath \ "response").read[List[SampleResponse]]
)(SampleResponses.apply _)
implicit val responsesWrites: Writes[SampleResponses] = (
(JsPath \ "id").write[Int] and
(JsPath \ "response").write[List[SampleResponse]]
)(unlift(SampleResponses.unapply))
implicit val rankedProductReads: Reads[SampleRankedProduct] = (
(JsPath \ "id").read[Int] and
(JsPath \ "rank").read[Int]
)(SampleRankedProduct.apply _)
implicit val rankedProductWrites: Writes[SampleRankedProduct] = (
(JsPath \ "id").write[Int] and
(JsPath \ "rank ").write[Int]
)(unlift(SampleRankedProduct.unapply))
implicit val rankReads: Reads[SampleRank] = (
(JsPath \ "user_id").read[Int] and
(JsPath \ "products").read[List[SampleRankedProduct]]
)(SampleRank.apply _)
implicit val rankWrites: Writes[SampleRank] = (
(JsPath \ "user_id").write[Int] and
(JsPath \ "products").write[List[SampleRankedProduct]]
)(unlift(SampleRank.unapply))
implicit val sortReads: Reads[Sort] = (JsPath \ "sort").read[List[SampleRank]].map(x ⇒ Sort(x))
implicit val sortWrites: Writes[Sort] = (__ \ "sort").write[List[SampleRank]].contramap { (person: Sort) => person.sort }
}
An approach for sorting the first JSON using the ranking from the second one, is something like that:
....
val prodRankDetails: List[SampleRank] = sortRespResult.right.get.sort
val sortedResults = values.copy(
responses = values.responses.map { resultForUser =>
val rankingForUser = prodRankDetails
.find(_.userId == resultForUser.userId)
.getOrElse(throw new RuntimeException(s"Rank not found for user ${resultForUser.userId}"))
.products
.map(product => (product.id, product.rank))
.toMap
resultForUser.copy(
products = resultForUser.products.sortWith((a, b) =>
rankingForUser.getOrElse(a.productId, Int.MaxValue) < rankingForUser.getOrElse(b.productId, Int.MaxValue))
)
}
)
Ok(Json.toJson(sortedResults))
that should be performed just after you got prodRankDetails. And also the response will be available inside the .map as you are managing a Future.
So you have to move also the Ok(...) inside that map and to flatMap the outer Future.
Hope this helps.
I have the following JSON being returned from a web service:
{
"hits": [
{
"created_at": "2016-02-01T15:01:03.000Z",
"title": "title",
"num_comments": 778,
"parent_id": null,
"_tags": [
"story",
"author",
"story_11012044"
],
"objectID": "11012044",
"_highlightResult": {
"title": {
"value": "title",
"matchLevel": "full",
"matchedWords": [
"title"
]
},
"author": {
"value": "author",
"matchLevel": "none",
"matchedWords": [
]
},
"story_text": {
"value": "Please lead",
"matchLevel": "none",
"matchedWords": [
]
}
}
}
]
}
and I am trying to parse it using the JSON parsing libs in Play Framework. I have the following code:
import play.api.libs.functional.syntax._
import play.api.libs.json._
case class Post(id: Long, date: String, count: Int)
object Post {
implicit val postFormat = Json.format[Post]
implicit val writes: Writes[Post] = (
(JsPath \ "id").write[Long] and
(JsPath \"date").write[String] and
(JsPath \ "count").write[Int]
)(unlift(Post.unapply))
implicit val reads: Reads[Post] = (
(JsPath \ "objectID").read[Long] and
(JsPath \ "created_at").read[String] and
(JsPath \ "num_comments").read[Int]
)(Post.apply _)
}
import play.api.libs.json._
class PostRepo {
val request: WSRequest = ws.url(MY_URL)
def getPosts: Future[Seq[Post]] =
val result: Future[JsValue] = request.get().map(response =>
response.status match {
case 200 => Json.parse(response.body)
case _ => throw new Exception("Web service call failed: " + response.body)
})
result.map( {
jsonvalue => println("JSARRAY: " + jsonvalue);
(jsonvalue \ "hits").as[Seq[Post]]
})
result
}
Now, when I run the code, I am getting the following error:
play.api.http.HttpErrorHandlerExceptions$$anon$1: Execution exception[[JsResultException:
JsResultException(errors:List(((0)/date,List(ValidationError(List(error.path.missing),WrappedArray()))),
((0)/count,List(ValidationError(List(error.path.missing),WrappedArray()))),
((0)/id,List(ValidationError(List(error.path.missing),WrappedArray()))),
((1)/date,List(ValidationError(List(error.path.missing),WrappedArray()))),
((1)/count,List(ValidationError(List(error.path.missing),WrappedArray()))),
((1)/id,List(ValidationError(List(error.path.missing),WrappedArray()))),
((2)/date,List(ValidationError(List(error.path.missing),WrappedArray()))),
((2)/count,List(ValidationError(List(error.path.missing),WrappedArray()))),
((2)/id,List(ValidationError(List(error.path.missing),WrappedArray()))),
((3)/date,List(ValidationError(List(error.path.missing),WrappedArray()))),
((3)/count,List(ValidationError(List(error.path.missing),WrappedArray()))),
((3)/id,List(ValidationError(List(error.path.missing),WrappedArray())))
Obviously something is wrong with the way I'm trying to parse the JSON but I have now spent a few hours trying to figure out the problem and I'm well and truly stuck.
Some refactoring code with Reads.seq
val r = (__ \ "hits").read[Seq[Post]](Reads.seq[Post])
def getPosts: Future[Seq[Post]] = {
WS.url(MY_URL).get().map(response =>
response.status match {
case 200 => r.reads(response.json) match {
case JsError(e) => throw new Exception("Json read fails. Response body:" + response.json.toString() + "\nRead error:" + e.toString())
case JsSuccess(x, _) => x
}
case _ => throw new Exception("Web service call failed: " + response.body)
})
}
I have a Json of the following format:
{
"user": {
"id": "1",
"name": "Some User",
"permGroups": [
{
"id": "group1",
"name": "Group 1",
"actions": [
{
"id": "action1",
"name": "Action 1"
}
]
},
{
"id": "group2",
"name": "Group 2",
"actions": [
{
"id": "action2",
"name": "Action 2"
},
{
"id": "action3",
"name": "Action 3"
}
]
}
]
},
"title": "New Role",
"role_id": "56fea66c"
}
How can I make a JSON Transformer in Play! 2.1, that will turn that into:
{
"name": "New Role",
"id" : "56fea66c",
"permGroupIds": ["group1","group2"]
"actions": ["action1", "action2", "action3"]
}
I have this working so far:
import play.api.libs.json._
import play.api.libs.functional.syntax._
import play.api.libs.json.Reads._
val jsonStr = """{"user":{"id":"1","name":"Some User","permGroups":[{"id":"group1","name":"Group 1","actions":[{"id":"action1","name":"Action 1"}]},{"id":"group2","name":"Group 2","actions":[{"id":"action3","name":"Action 3"},{"id":"action3","name":"Action 3"}]}]},"title":"New Role","role_id":"56fea66c"}"""
val jsonVal = Json.parse(jsonStr)
val jsonTransformer = (
(__ \ 'name).json.copyFrom((__ \ 'title).json.pick) and
(__ \ 'id).json.copyFrom((__ \ 'role_id).json.pick)
).reduce
jsonVal.transform(jsonTransformer)
this produces:
{
"name": "New Role",
"id" : "56fea66c",
}
Here is one way to do it. This will ignore any perm groups or actions that don't have an id.
import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.functional.syntax._
val jsonStr = """{"user":{"id":"1","name":"Some User","permGroups":[{"id":"group1","name":"Group 1","actions":[{"id":"action1","name":"Action 1"}]},{"id":"group2","name":"Group 2","actions":[{"id":"action3","name":"Action 3"},{"id":"action3","name":"Action 3"}]}]},"title":"New Role","role_id":"56fea66c"}"""
val jsonTransformer = (
(__ \ 'name).json.copyFrom((__ \ 'user \ 'name).json.pick) and
(__ \ 'id).json.copyFrom((__ \ 'role_id).json.pick) and
(__ \ 'permGroupIds).json.copyFrom((__ \ 'user \ 'permGroups).read[List[JsObject]].map(permGroups =>
JsArray(permGroups.flatMap(permGroup => (permGroup \ "id").toOption))
)) and
(__ \ 'actions).json.copyFrom((__ \ 'user \ 'permGroups).read[List[JsObject]].map(permGroups =>
JsArray(permGroups.flatMap(permGroup => (permGroup \ "actions").asOpt[JsArray]).flatMap(_ \\ "id"))
))
).reduce
val transformed = Json.parse(jsonStr).transform(jsonTransformer)
Json.prettyPrint(transformed.get)
This produces
{
"name" : "Some User",
"id" : "56fea66c",
"permGroupIds" : [ "group1", "group2" ],
"actions" : [ "action1", "action3", "action3" ]
}