Find a JsValue in JsObject In Scala - json

How can I make a recursive function which finds a JsValue in JsObject, for example:
Function input: JsObject and path as String
Examples:
Input for JsObject:
{
"name": "Zvi"
parents: {
"father": {
"name": "myFatherName",
"dob": "10/10/70"
}
}
}
and for path Example:
path = notExistsField
the output will be None
path = name
the input will be Some("Zvi")
path = parents
the output will be
Some({
"father": {
"name": "myFatherName",
"dob": "10/10/70"
}
})
parents.father
the output will be:
Some({
"name": "myFatherName",
"dob": "10/10/70"
})
parents.father.name
the output will be "myFatherName"
Tests Example:
"deepSearch" - {
val payload: JsObject = Json.parse(
"""{
| "key1": 1,
| "key2": {
| "key1": "value21",
| "key2": {
| "key1": 221
| }
| },
|}""".stripMargin).as[JsObject]
"found" - {
"nested" in {
// In Nested Object
service.deepSearch(payload,"key2.key1").get shouldBe JsString("value21")
service.deepSearch(payload,"key2.key2.key1").get shouldBe JsNumber(221)
service.deepSearch(payload,"key2.key2").get shouldBe Json.parse("{\"key1\": 221}")
}
"top-level" in {
service.deepSearch(payload,"key1").get shouldBe JsNumber(1)
}
}
"not found" - {
"nested" in {
service.deepSearch(payload,"key2.key2.key1.notExists").isEmpty shouldBe true
service.deepSearch(payload,"key2.key2.notExists").isEmpty shouldBe true
}
"top-level" in {
service.deepSearch(payload,"Boom").isEmpty shouldBe true
}
}
}

I did the following implementation, Is there any suggestions to prettier code
def deepSearch(payload: JsValue, path: String): JsLookupResult = {
path indexOf (".") match {
// For the Base case:
case base if base < 0 => (payload \ path)
// For the Step case:
case found if found >= 0 =>
val untilFirstDot = path.substring(0, found)
(payload \ untilFirstDot) match {
case JsDefined(newPayload) => deepSearch(newPayload, path.substring(found + 1))
case undefined: JsUndefined => undefined
}
}
}

Related

Compare two Json having different arrangement of items using scala

I have two json string having different order of some of the items, so they are like equal, and I want to compare them for equality, but not sure how to achieve it.
JSON 1
{
"data": [
{
"configTemplateId": "44f11ed4-5b08-11ea-8e2d-0242ac132222",
"params": {
"keyOne": "valueOne",
"keyTwo": "valueTwo"
}
},
{
"configTemplateId": "44f11ed4-5b08-11ea-8e2d-0242ac131111",
"params": {
"keyOne": "valueOne",
"keyTwo": "valueTwo"
}
}
]}
JSON 2
{
"data": [
{
"configTemplateId": "44f11ed4-5b08-11ea-8e2d-0242ac131111",
"params": {
"keyOne": "valueOne",
"keyTwo": "valueTwo"
}
},
{
"configTemplateId": "44f11ed4-5b08-11ea-8e2d-0242ac132222",
"params": {
"keyOne": "valueOne",
"keyTwo": "valueTwo"
}
}
]}
I have tried using spray-json as mentioned in Compare json equality in Scala but that does not seem working.
You have to implement some kind of deepEquals. The idea is the following:
import play.api.libs.json.{JsArray, JsObject, JsValue, Json}
def deepEquals(json1: JsValue, json2: JsValue): Boolean = {
def sortArray(json: JsValue): JsValue = json match {
case JsArray(arr) => JsArray(arr.map(sortArray).sortBy(_.toString))
case other => other
}
if (json1.getClass != json2.getClass) {
false
} else {
(json1, json2) match {
case (j1: JsObject, j2: JsObject) =>
val fields1 = j1.value
val fields2 = j2.value
(fields1.keySet ++ fields2.keySet).foldLeft(true) { (acc, key) =>
val field1 = fields1.get(key)
val field2 = fields2.get(key)
acc && ((field1, field2) match {
case (Some(v1), Some(v2)) => deepEquals(v1, v2)
case _ => false
})
}
case (j1: JsArray, j2: JsArray) =>
sortArray(j1) == sortArray(j2)
case (other1, other2) => other1 == other2
}
}
}
I used json play library, but with spray the implementation won't be much different

Scala: Edit/Modify json string based on internal value

I have a json string structured similarly to the following:
val json : String =
{
"identifier":{
"id":"1234_567_910",
"timestamp":"12:34:56",
},
"information":[
{
"fieldName":"test_name",
"fieldId":"test_fieldId",
}
]
}
What I want to do is create a check that verifies the 'id' field matches the structure "Int_Int_Int" and if it doesn't I want to change the value to match this intended structure but I want to keep the rest of the information in the json string as is.
So if I received the following 'id' fields within a json string I would want to change them like so:
"id":"1234_567_910" -> do nothing
"id":"1234" -> "id":"1234_0_0"
"id":"1234_567" -> "id":"1234_567_0"
"id":"1234_???" -> "id":"1234_0_0"
"id":"1234_??_???" -> "id":"1234_0_0"
"id":"1234_foo" -> "id":"1234_0_0"
"id":"1234_567_foo" -> "id":"1234_567_0"
For Example:
If I receive json like this:
{
"identifier":{
"id":"1234",
"timestamp":"12:34:56",
},
"information":[
{
"fieldName":"test_name",
"fieldId":"test_fieldId",
}
]
}
I would want to modify it so I end up with a json like this:
{
"identifier":{
"id":"1234_0_0",
"timestamp":"12:34:56",
},
"information":[
{
"fieldName":"test_name",
"fieldId":"test_fieldId",
}
]
}
What would be the most effective/cleanest way to achieve this type of json modification in Scala?
Below is how it can be done with the Dijon library.
import com.github.pathikrit.dijon._
def normalize(id: String): String =
id.count(_ == '_') match {
case 0 => id + "_0_0"
case 1 => id + "_0"
case _ => id
}
val json =
json"""
{
"identifier":{
"id":"1234",
"timestamp":"12:34:56"
},
"information":[
{
"fieldName":"test_name",
"fieldId":"test_fieldId"
}
]
}"""
json.identifier.id = json.identifier.id.asString.fold("0_0_0")(normalize)
println(pretty(json))
It should print:
{
"identifier": {
"id": "1234_0_0",
"timestamp": "12:34:56"
},
"information": [
{
"fieldName": "test_name",
"fieldId": "test_fieldId"
}
]
}

How to read part of a nested json object in as a string in Scala

I want to read in a json object into a scala class that keeps part of the json object as a string without trying to parse it.
This is what the json looks like:
[
{
"contractType": "NullContract",
"contractDefinition": {
"column": "age",
"conditions": [
{
"conditionType": "equalsCondition",
"conditionDefinition": {
"column": "species",
"value": "person"
}
}
]
}
}
]
I am using the jackson library. This is my mapper:
val mapper = new ObjectMapper()
.registerModule(DefaultScalaModule)
.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
.setSerializationInclusion(Include.NON_ABSENT)
This is my class:
case class ContractJson(contractType: String, contractDefinition: String)
, which is how I want my resulting object to look.
This is the code parsing it:
val contractJson: Array[ContractJson] = mapper.readValue(contractsJsonString, classOf[Array[ContractJson]])
Error message I'm getting: Can not deserialize instance of java.lang.String out of START_OBJECT token when it starts parsing the contractDefinition
If you are allowed to use other libraries and type of the contractDefinition field is not restricted to String then try jsoniter-scala's feature of extraction raw JSON values to byte arrays.
You will need to add dependencies:
libraryDependencies ++= Seq(
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-core" % "1.1.0" % Compile,
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-macros" % "1.1.0" % Provided // required only in compile-time
)
Then define types with codecs and parse the input:
import java.nio.charset.StandardCharsets.UTF_8
import com.github.plokhotnyuk.jsoniter_scala.macros._
import com.github.plokhotnyuk.jsoniter_scala.core._
import scala.util.hashing.MurmurHash3
object RawVal {
def apply(s: String) = new RawVal(s.getBytes)
implicit val codec: JsonValueCodec[RawVal] = new JsonValueCodec[RawVal] {
override def decodeValue(in: JsonReader, default: RawVal): RawVal = new RawVal(in.readRawValAsBytes())
override def encodeValue(x: RawVal, out: JsonWriter): Unit = out.writeRawVal(x.bs)
override val nullValue: RawVal = new RawVal(new Array[Byte](0))
}
}
case class RawVal private(bs: Array[Byte]) {
def this(s: String) = this(s.getBytes(UTF_8))
override lazy val hashCode: Int = MurmurHash3.arrayHash(bs)
override def equals(obj: Any): Boolean = obj match {
case that: RawVal => java.util.Arrays.equals(bs, that.bs)
case _ => false
}
override def toString: String = new String(bs, UTF_8)
}
case class ContractJson(contractType: String, contractDefinition: RawVal)
implicit val codec: JsonValueCodec[List[ContractJson]] = JsonCodecMaker.make(CodecMakerConfig)
val jsonBytes =
"""[
| {
| "contractType": "NullContract",
| "contractDefinition": {
| "column": "age",
| "conditions": [
| {
| "conditionType": "equalsCondition",
| "conditionDefinition": {
| "column": "species",
| "value": "person"
| }
| }
| ]
| }
| }
|]
|""".stripMargin.getBytes("UTF-8")
val contractJsons = readFromArray(jsonBytes)
println(contractJsons)
Printed result will be:
List(ContractJson(NullContract, {
"column": "age",
"conditions": [
{
"conditionType": "equalsCondition",
"conditionDefinition": {
"column": "species",
"value": "person"
}
}
]
}))

Parsing a file having content as Json format in Scala

I want to parse a file having content as json format.
From the file I want to extract few properties (name, DataType, Nullable) to create some column names dynamically.
I have gone through some examples but most of them are using case class but my problem is every time I will receive a file may have different content.
I tried to use the ujson library to parse the file but I am unable to understand how to use it properly.
object JsonTest {
def main(args: Array[String]): Unit = {
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
println(source)
val input = try source.mkString finally source.close()
println(input)
val data = ujson.read(input)
data("name") = data("name").str.reverse
val updated = data.render()
}
}
Content of the file example:
{
"Organization": {
"project": {
"name": "POC 4PL",
"description": "Implementation of orderbook"
},
"Entities": [
{
"name": "Shipments",
"Type": "Fact",
"Attributes": [
{
"name": "Shipment_Details",
"DataType": "StringType",
"Nullable": "true"
},
{
"name": "Shipment_ID",
"DataType": "StringType",
"Nullable": "true"
},
{
"name": "View_Cost",
"DataType": "StringType",
"Nullable": "true"
}
],
"ADLS_Location": "/mnt/mns/adls/raw/poc/orderbook/"
}
]
}
}
Expected output:
StructType(
Array(StructField("Shipment_Details",StringType,true),
StructField("Shipment_ID",DateType,true),
StructField("View_Cost",DateType,true)))
StructType needs to be added to the expected output programatically.
Try Using Playframework's Json utils - https://www.playframework.com/documentation/2.7.x/ScalaJson
Here's the solution to your issue-
\ Placed your json in text file
val fil_path = "C:\\TestData\\Config\\Conf.txt"
val conf_source = scala.io.Source.fromFile(fil_path)
lazy val json_str = try conf_source.mkString finally conf_source.close()
val conf_json: JsValue = Json.parse(json_str)
val all_entities: JsArray = (conf_json \ "Organization" \ "Entities").get.asInstanceOf[JsArray]
val shipments: JsValue = all_entities.value.filter(e => e.\("name").as[String] == "Shipments").head
val shipments_attributes: IndexedSeq[JsValue] = shipments.\("Attributes").get.asInstanceOf[JsArray].value
val shipments_schema: StructType = StructType(shipments_attributes.map(a => Tuple3(a.\("name").as[String], a.\("DataType").as[String], a.\("Nullable").as[String]))
.map(x => StructField(x._1, StrtoDatatype(x._2), x._3.toBoolean)))
shipments_schema.fields.foreach(println)
Output is -
StructField(Shipment_Details,StringType,true)
StructField(Shipment_ID,StringType,true)
StructField(View_Cost,StringType,true)
It depends if you want it to be completely dynamic or not, here are some options:
If you just want to read one field you can do:
import upickle.default._
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
val json = ujson.read(input)
println(json("Organization")("project")("name"))
the output will be: "POC 4PL"
If you just want just the Attributes to be with types, you can do:
import upickle.default.{macroRW, ReadWriter => RW}
import upickle.default._
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
val json = ujson.read(input)
val entitiesArray = json("Organization")("Entities")(0)("Attributes")
println(read[Seq[StructField]](entitiesArray))
case class StructField(name: String, DataType: String, Nullable: String)
object StructField{
implicit val rw: RW[StructField] = macroRW
}
the output will be: List(StructField(Shipment_Details,StringType,true), StructField(Shipment_ID,StringType,true), StructField(View_Cost,StringType,true))
another option, is to use a different library to do the class mapping. If you use Google Protobuf Struct and JsonFormat it can be 2-liner:
import com.google.protobuf.Struct
import com.google.protobuf.util.JsonFormat
val source = scala.io.Source.fromFile("C:\\Users\\ktngme\\Desktop\\ass\\file.txt")
val input = try source.mkString finally source.close()
JsonFormat.parser().merge(input, builder)
println(builder.build())
the output will be: fields { key: "Organization" value { struct_value { fields { key: "project" value { struct_value { fields { key: "name" value { string_value: "POC 4PL" } } fields { key: "description" value { string_value: "Implementation of orderbook" } } } } } fields { key: "Entities" value { list_value { values { struct_value { fields { key: "name" value { string_value: "Shipments" } }...

How to get Keys and values, while parsing Json using Scala-Play Json Framework?

I have a json file , which has some keys and values. I need to parse the Json and print the keys and their values. For example, the json file is like below. I want to print this as Keys and values
{
"Parcer":[
{
"key":"0203",
"value":{
"Encryption":
{
"enabled":"yes",
"encryption_type":"base64",
"key":"334848484",
"return":"name"
}
}
},
{
"key":"0405",
"value":{
"Encryption":
{
"enabled":"yes",
"encryption_type":"base64",
"key":"334848484",
"return":"none"
},
"Parcer":[
{
"key":"0102",
"value":"humidity"
},
{
"key":"0304",
"value":{
"Encryption":{
"enabled":"yes",
"encryption_type":"SHA1",
"key":"1211212",
"return":"none"
}
}
}
]
}
}],
}```
The easiest way is to create a case class, like:
case class MyObj(header:String, value: Seq[Map[String, String]])
Then you just need to add one line for marshalling, like:
import play.api.libs.json._
object MyObj {
implicit val jsonFormat: OFormat[MyObj] = Json.format[MyObj]
}
Now you get a nice case class that you can work with:
val json =
Json.parse(
"""{
"header" : "header value",
"value" : [
{
"a" : "a_val",
"b" : "b_val",
"c" : "c_val"
},
{
"a" : "a_val",
"b" : "b_val",
"c" : "c_val"
}
]
}""")
Here an example how to retrieve all "a".
json.validate[MyObj] match {
case JsSuccess(myObj, _) =>
val allAs =myObj.value.flatMap(m => m.get("a").toSeq)
println(allAs) // >> Vector(a_val, a_val)
case e:JsError => // handle error
}
This gives you:
json.validate[MyObj] returns JsSuccess(MyObj(header value,Vector(Map(a -> a_val, b -> b_val, c -> c_val), Map(a -> a_val, b -> b_val, c -> c_val))),)
The println returns: Vector(a_val, a_val)
This is described here in the Documentation: JSON automated mapping