Deep remove specific empty json array in circe scala - json

I want to remove deep empty json array from my json before/during processing by circe.
Incoming JSON
{
"config": {
"newFiles": [{
"type": "audio",
"value": "welcome1.mp3"
}],
"oldFiles": [],
"channel": "BC"
}
}
or
{
"config": {
"newFiles": [],
"oldFiles": [{
"type": "audio",
"value": "welcome1.mp3"
}],
"channel": "BC"
}
}
Resulted Json should look like
{
"config": {
"newFiles": [{
"type": "audio",
"value": "welcome1.mp3"
}],
"channel": "BC"
}
}
or
{
"config": {
"oldFiles": [{
"type": "audio",
"value": "welcome1.mp3"
}],
"channel": "BC"
}
}
What i understand that this can be done before decoding config as well as during decoding config.
The idea here is i want to handle only one of files (either new or old) at my case class level.
Method 1: Tried at config decoding level which works well.
case class File(`type`: String, value: String)
case class Config(files: List[File],
channel: String = "BC")
object Config{
implicit final val FileDecoder: Decoder[File] = deriveDecoder[File]
implicit val ConfigDecoder: Decoder[Config] = (h:HCursor) =>
for {
oldFiles <- h.get[List[File]]("oldFiles")
files <- if (oldFiles.isEmpty) h.get[List[File]]("newFiles") else h.get[List[File]]("oldFiles")
channel <- h.downField("channel").as[String]
}yield{
Config(files, channel)
}
}
case class Inventory(config: Config)
object Inventory {
implicit val InventoryDecoder: Decoder[Inventory] = deriveDecoder[Inventory]
}
Method 2: Tried before feeding into decoding which didn't worked out
Let me know what could be the elegant approach to handle it.
PS: I have multiple similar config decoders and if i handle this at config decoding level then there will be a lot of boiler plate code.

I have simplified the problem a little bit again, but combining this with the previous answer should be simple.
I also took advantage of cats.data.NonEmptyList
final case class Config(files: NonEmptyList[String], channel: String = "BC")
object Config {
implicit final val ConfigDecoder: Decoder[Config] =
(
(Decoder[NonEmptyList[String]].at(field = "newFiles") or Decoder[NonEmptyList[String]].at(field = "oldFiles")),
Decoder[String].at(field = "channel")
).mapN(Config.apply).at(field = "config")
}
This can be used like this:
val data =
"""[
{
"config": {
"newFiles": ["Foo", "Bar", "Baz"],
"oldFiles": [],
"channel": "BC"
}
},
{
"config": {
"newFiles": [],
"oldFiles": ["Quax"],
"channel": "BC"
}
}
]"""
parser.decode[List[Config]](data)
// res: Either[io.circe.Error, List[Config]] =
// Right(List(
// Config(NonEmptyList("Foo", "Bar", "Baz"), "BC"),
// Config(NonEmptyList("Quax"), "BC")
// ))
Note: I am assuming that at least one of the two lists must be non-empty and give priority to the new one.
You can see the code running here.

Related

Scala Json parsing

This is the input json which I am getting which is nested json structure and I don't want to map directly to class, need custom parsing of some the objects as I have made the case classes
{
"uuid": "b547e13e-b32d-11ec-b909-0242ac120002",
"log": {
"Response": {
"info": {
"receivedTime": "2022-02-09T00:30:00Z",
"isSecure": "Yes",
"Data": [{
"id": "75641",
"type": "vendor",
"sourceId": "3",
"size": 53
}],
"Group": [{
"isActive": "yes",
"metadata": {
"owner": "owner1",
"compressionType": "gz",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "1"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner12",
"compressionType": "snappy",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "2"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner123",
"compressionType": "snappy",
"comments": "someComment",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "4"
},
{
"isActive": "yes",
"metadata": {
"owner": "owner124",
"compressionType": "snappy",
"comments": "someComments",
"createdDate": "2022-01-11T11:00:00Z",
"updatedDate": "2022-01-12T14:17:55Z"
},
"setId": "4"
}
]
}
}
}
}
Code that I am trying play json also tried circe . Please help ..New to scala world
below is object and case class
case class DataCatalog(uuid: String, data: Seq[Data], metadata: Seq[Metadata])
object DataCatalog {
case class Data(id: String,
type: String,
sourceId: Option[Int],
size: Int)
case class Metadata(isActive: String,
owner: String,
compressionType: String,
comments: String,
createdDate: String,
updatedDate: String
)
def convertJson(inputjsonLine: String): Option[DataCatalog] = {
val result = Try {
//val doc: Json = parse(line).getOrElse(Json.Null)
//val cursor: HCursor = doc.hcursor
//val uuid: Decoder.Result[String] = cursor.downField("uuid").as[String]
val lat = (inputjsonLine \ "uuid").get
DataCatalog(uuid, data, group)
}
//do pattern matching
result match {
case Success(dataCatalog) => Some(dataCatalog)
case Failure(exception) =>
}
}
}
Any parsing api is fine.
If you use Scala Play, for each case class you should have an companion object which will help you a lot with read/write object in/from json:
object Data {
import play.api.libs.json._
implicit val read = Json.reads[Data ]
implicit val write = Json.writes[Data ]
def tupled = (Data.apply _).tupled
}
object Metadata {
import play.api.libs.json._
implicit val read = Json.reads[Metadata ]
implicit val write = Json.writes[Metadata ]
def tupled = (Metadata.apply _).tupled
}
Is required as each companion object to be in same file as the case class. For your json example, you need more case classes because you have a lot of nested objects there (log, Response, info, each of it)
or, you can read the field which you're interested in as:
(jsonData \ "fieldName").as[CaseClassName]
You can try to access the Data value as:
(jsonData \ "log" \ "Response" \ "info" \ "Data").as[Data]
same for Metadata

How to parse just part of JSON with Klaxon?

I'm trying to parse some JSON to kotlin objects. The JSON looks like:
{
data: [
{ "name": "aaa", "age": 11 },
{ "name": "bbb", "age": 22 },
],
otherdata : "don't need"
}
I just need to data part of the entire JSON, and parse each item to a User object:
data class User(name:String, age:Int)
But I can't find an easy way to do it.
Here's one way you can achieve this
import com.beust.klaxon.Klaxon
import java.io.StringReader
val json = """
{
"data": [
{ "name": "aaa", "age": 11 },
{ "name": "bbb", "age": 22 },
],
"otherdata" : "not needed"
}
""".trimIndent()
data class User(val name: String, val age: Int)
fun main(args: Array<String>) {
val klaxon = Klaxon()
val parsed = klaxon.parseJsonObject(StringReader(json))
val dataArray = parsed.array<Any>("data")
val users = dataArray?.let { klaxon.parseFromJsonArray<User>(it) }
println(users)
}
This will work as long as you can fit the whole json string in memory. Otherwise you may want to look into the streaming API: https://github.com/cbeust/klaxon#streaming-api

How to get json value from Angular2 app

I need to get values from a JSON file which is served from fake-json-server.
To be precise, I need an exact value, e.g. I need to get all "type": "values" where group is Air.
I'm using Angular2 with TypeScript and here is a part of the code where I'm doing a get request in the TransformerService file:
getVehicleGroups(groupName: string) {
return this.http.get(`${this.apiUrl}/vehicleTypes?group=${groupName}`)
.map((res: Response) => res.json() as VehicleTypes[]).catch(this.handleError);
}
Exported class:
export class VehicleTypes {
// vehicleGroup: string;
vehicleType: string;
vehicleModel: string;
}
And here I'm calling that method in the separate file:
getVehicleGroups() {
return this.transformersService.getVehicleGroups(this.vehicleGroup)
.subscribe((vehicleTypes => this.vehicleTypes = vehicleTypes));
}
The url of the fake-server "http://localhost:3000/vehicleTypes" and this is the code from db.json on that server (url):
[
{
"group": "Air",
"type": "Plane",
"model": "F-22"
},
{
"group": "Air",
"type": "Plane",
"model": "Sukhoi"
},
{
"group": "Air",
"type": "Plane",
"model": "MiG"
},
{
"group": "Air",
"type": "Helicopter",
"model": "Apache"
},
{
"group": "Air",
"type": "Helicopter",
"model": "Kamov"
}
{
"group": "Sea",
"type": "Boat",
"model": "Sailboat"
},
{
"group": "Sea",
"type": "Boat",
"model": "Jetboat"
},
{
"group": "Sea",
"type": "Submarine",
"model": "Standard"
},
{
"group": "Land",
"type": "Car",
"model": "Camaro"
},
{
"group": "Land",
"type": "Car",
"model": "AMG GT R"
},
{
"group": "Land",
"type": "Car",
"model": "Lamborghini"
},
{
"group": "Land",
"type": "Truck",
"model": "Unimog"
},
{
"group": "Land",
"type": "Truck",
"model": "Western Star 5700"
}
]
I need to mention, all my files are set well. I don't get any errors, I'm just not getting the right values..
I need to get all "type": "values" where group is Air
First you need do filter your json result to get Air group only.
You can apply observable filter
getVehicleGroups(groupName: string) {
return this.http.get(`${this.apiUrl}/vehicleTypes?group=${groupName}`)
.filter(data => data.group === "Air")
.map((res: Response) => res.json() as VehicleTypes[]).catch(this.handleError);
}
Second your VehicleTypes model variable names are different with json response so how will angular convert your json array into VehicleTypes array. you need change VehicleTypes class or your backend code send match variables name.
export interface VehicleTypes {
type: string;
model: string;
}
Adjust the brackets of your method.
From:
getVehicleGroups() {
return this.transformersService.getVehicleGroups(this.vehicleGroup)
.subscribe((vehicleTypes => this.vehicleTypes = vehicleTypes));
}
To:
getVehicleGroups() {
return this.transformersService.getVehicleGroups(this.vehicleGroup)
.subscribe((vehicleTypes) => this.vehicleTypes = vehicleTypes);
}
Map the data with your model:
Option 1: Change the model to match the json data.
export class VehicleTypes {
type: string;
model: string;
}
Option 2: Change the json properties at service level, right after converting to json.
getVehicleGroups(groupName: string) {
return this.http.get(`${this.apiUrl}/vehicleTypes?group=${groupName}`)
.map((res: Response) => res.json().map(res => new VehicleTypes(res.type, res.model)).catch(this.handleError);
}
and you would need to create a constructor for the VehicleTypes.
You do not need a class in this case, a type interface will suffice since you don't have (or seem to need) any methods in your vehicle, you're only using it for type assertion.
A class exists in the emitted JavaScript incurring unnecessary overhead, whereas an interface provides type safety without emitting classes to JavaScript: it's only used by the type-checker in tsc and then discarded.
export interface VehicleTypes {
// vehicleGroup: string;
vehicleType: string;
vehicleModel: string;
}
Declare the type that your service returns:
getVehicleGroups(groupName: string): Observable<VehicleTypes[]> {
return this.http.get(`${this.apiUrl}/vehicleTypes?group=${groupName}`)
.map(res.json)
.catch(this.handleError);
}
And consume it in your Component as:
// Assert the type vehicleTypes expects
vehicleTypes: <VehicleTypes>[];
getVehicleGroups() {
return this.transformersService.getVehicleGroups(this.vehicleGroup)
.subscribe(vehicleTypes => this.vehicleTypes = vehicleTypes);
}
Note that you don't need the (res: Response) assertion in the chain from http.get: that's what get is typed to return anyway so the type-checker already knows what to expect. Since you can remove the parameter, you can make the chain even shorter, as you did with .catch.

extract case classes from json file scala play

im trying to extract my data from json into a case class without success.
the Json file:
[
{
"name": "bb",
"loc": "sss",
"elements": [
{
"name": "name1",
"loc": "firstHere",
"elements": []
}
]
},
{
"name": "ca",
"loc": "sss",
"elements": []
}
]
my code :
case class ElementContainer(name : String, location : String,elements : Seq[ElementContainer])
object elementsFormatter {
implicit val elementFormatter = Json.format[ElementContainer]
}
object Applicationss extends App {
val el = new ElementContainer("name1", "firstHere", Seq.empty)
val el1Cont = new ElementContainer("bb","sss", Seq(el))
val source:String=Source.fromFile("src/bin/elementsTree.json").getLines.mkString
val jsonFormat = Json.parse(source)
val r1= Json.fromJson[ElementContainer](jsonFormat)
}
after running this im getting inside r1:
JsError(List((/elements,List(ValidationError(List(error.path.missing),WrappedArray()))), (/name,List(ValidationError(List(error.path.missing),WrappedArray()))), (/location,List(ValidationError(List(error.path.missing),WrappedArray())))))
been trying to extract this data forever, please advise
You have location instead loc and, you'll need to parse file into a Seq[ElementContainer], since it's an array, not a single ElementContainer:
Json.fromJson[Seq[ElementContainer]](jsonFormat)
Also, you have the validate method that will return you either errors or parsed json object..

Groovy JSONBuilder issues

I'm trying to use JsonBuilder with Groovy to dynamically generate JSON. I want to create a JSON block like:
{
"type": {
"__type": "urn",
"value": "myCustomValue1"
},
"urn": {
"__type": "urn",
"value": "myCustomValue2"
},
"date": {
"epoch": 1265662800000,
"str": "2010-02-08T21:00:00Z"
},
"metadata": [{
"ratings": [{
"rating": "NR",
"scheme": "eirin",
"_type": {
"__type": "urn",
"value": "myCustomValue3"
}
}],
"creators": [Jim, Bob, Joe]
}]
}
I've written:
def addUrn(parent, type, urnVal) {
parent."$type" {
__type "urn"
"value" urnVal
}
}
String getEpisode(String myCustomVal1, String myCustomVal2, String myCustomVal3) {
def builder = new groovy.json.JsonBuilder()
def root = builder {
addUrn(builder, "type", myCustomVal1)
addUrn(builder, "urn", "some:urn:$myCustomVal2")
"date" {
epoch 1265662800000
str "2010-02-08T21:00:00Z"
}
"metadata" ({
ratings ({
rating "G"
scheme "eirin"
addUrn(builder, "_type", "$myCustomVal3")
})
creators "Jim", "Bob", "Joe"
})
}
return root.toString();
}
But I've run into the following issues:
Whenever I call addUrn, nothing is returned in the string. Am I misunderstanding how to use methods in Groovy?
None of the values are encapsulated in double (or single) quotes in the returned string.
Anytime I use a {, I get a '_getEpisode_closure2_closure2#(insert hex)' in the returned value.
Is there something wrong with my syntax? Or can someone point me to some example/tutorial that uses methods and/or examples beyond simple values (e.g. nested values within arrays).
NOTE: This is a watered down example, but I tried to maintain the complexity around the areas that were giving me issues.
You have to use delegate in addUrn method instead of
passing the builder on which you are working.
It is because you are doing a toSting() or toPrettyString() on root instead of builder.
Solved if #2 is followed.
Sample:
def builder = new groovy.json.JsonBuilder()
def root = builder {
name "Devin"
data {
type "Test"
note "Dummy"
}
addUrn(delegate, "gender", "male")
addUrn(delegate, "zip", "43230")
}
def addUrn(parent, type, urnVal) {
parent."$type" {
__type "urn"
"value" urnVal
}
}
println builder.toPrettyString()
Output:-
{
"name": "Devin",
"data": {
"type": "Test",
"note": "Dummy"
},
"gender": {
"__type": "urn",
"value": "male"
},
"zip": {
"__type": "urn",
"value": "43230"
}
}