JsonNode with Array elements - json

The node fields contain an array. Currently there is only one element, So I can use get(0) to get the first element and parse the string to find the value of the valid.
The problem with this solution is if tomorrow there will be more k:v added to the array, this will fail. Also can I use any elegant way to parse the value of valid?
import com.fasterxml.jackson.databind.{JsonNode, ObjectMapper}
val response = """{"myTable":{"fields":["valid:true"]}}"""
val mapper = new ObjectMapper()
val node = mapper.readTree(response)
val result = node.get("myTable").get("fields").get(0).toString.contains("valid:true")
println(s"valid=$result")
result :
valid=true

You can iterate over array items using elements method. You just need to convert Iterator to Stream:
val fieldsValues = node.get("myTable").get("fields").elements()
val value = StreamSupport.stream(Spliterators.spliteratorUnknownSize(fieldsValues, 0), false)
.filter(item => item.isTextual)
.map[String]((item: JsonNode) => item.textValue())
.map[Boolean]((item: String) => item.equals("valid:true"))
.findAny()
.orElse(false)

Related

How to extract values from inner JSONObject from a key inside of entity response output-type of an API in KOTLIN?

I was trying to get the array of JSONObjects from the ResponseEntity< JSONObject > variable "result".
<200 OK,{"response":"success","count":0,"maintenanceDetails":[{"date":"2020\/08\/21","activity":"TestActivity","operatorName":"Operator","operator":"Mock for Party, hashCode: 552416003","status":"Open"}],"buildingId":Mock for UUID, hashCode: 2141932519},{}>
Output value of result is given above.
val result = dashBoardService!!.getMaintenanceStateByBuildingId(id)
println("Result "+result.body["maintenanceDetails"])
val a =result.body["maintenanceDetails"] as Array<JSONObject>
println("A "+ a.first().get("activity"))
I Use "as Array < JSONObject >" to convert, because result.body["maintenanceDetails"] normally comes as "Any" type, which I can't extract more. But this gives me below error.
java.lang.ClassCastException: java.util.ArrayList cannot be cast to [Lorg.json.simple.JSONObject;
What I need is to get the first JSONObject from Array corresponds to result.body["maintenanceDetails"] so that I can use it in assert checks. Kindly help.
Conversion is working to ArrayList since it is considered as util.ArrayList by Intelij, So after this I am able to get the data from inside.
val tempArr= result.body["maintenanceDetails"] as ArrayList<JSONObject>
assertEquals(tempArr[0]["activity"],maintenanceState.activity)
assertEquals(tempArr[0]["status"],maintenanceState.status)
Use:
val array = JSONArray(result.body["maintenanceDetails"])
val first = array.getJSONObject(0)
You can use JSONArray() to access the array inside a json object (Root) and then iterate the array to get all the json objects present in the array. Here is a sample,
import org.json.simple.JSONArray
import org.json.simple.JSONObject
val jsonObject = JSONObject()
val childObject = JSONObject()
childObject["date"]="10-10-2010"
childObject["operator"]="mock party"
val jsonArray = JSONArray()
jsonArray.add(0,childObject)
jsonObject["response"]="success"
jsonObject["count"]=0
jsonObject["maintenanceDetails"]=jsonArray
val json = jsonObject["maintenanceDetails"] as JSONArray
println(json[0])
val outputObject = json[0] as JSONObject
println(outputObject["operator"])
Output:
{"date":"10-10-2010","operator":"mock party"}
mock party

Klaxon Parsing of Nested Arrays

Im trying to parse this file with Klaxon, generally its going well, except I am totally not succeeding in parsing that subarray of features/[Number]/properties/
So my thought is to get the raw string of properties and to parse it seperately with Klaxon, though I dont succeed in that either. Apart from that I took many other approaches as well.
My code so far:
class Haltestelle(val type: String?, val totalFeatures: Int?, val features: Array<Any>?)
fun main(args: Array<String>) { // Main-Routine
val haltejsonurl = URL("http://online-service.kvb-koeln.de/geoserver/OPENDATA/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=ODENDATA%3Ahaltestellenbereiche&outputFormat=application/json")
val haltestringurl = haltejsonurl.readText()
val halteklx = Klaxon().parse<Haltestelle>(haltestringurl)
println(halteklx?.type)
println(halteklx?.totalFeatures)
println(halteklx?.features)
halteklx?.features!!.forEach {
println(it)
}
I am aware that I am invoking features as an Array of Any, so the Output is just printing me java.lang.Object#blabla everytime. Though, using Array failes either.
Really spend hours in this, how would you go on this?
Regards of newbie
Here's how I did something similar in Kotlin. You can parse the response as a Klaxon JsonObject, then access the "features" element to parse all the array objects into a JsonArray of JsonObjects. This can be iterated over and cast with parseFromJsonObject<Haltestelle> in your example:
import com.beust.klaxon.JsonArray
import com.beust.klaxon.JsonObject
import com.beust.klaxon.Parser
import com.github.aivancioglo.resttest.*
val response : Response = RestTest.get("http://anyurlwithJSONresponse")
val myParser = Parser()
val data : JsonObject = myParser.parse(response.getBody()) as JsonObject
val allFeatures : JsonArray<JsonObject>? = response["features"] as JsonArray<JsonObject>?
for((index,obj) in allFeatures.withIndex()) {
println("Loop Iteration $index on each object")
val yourObj = Klaxon().parseFromJsonObject<Haltestelle>(obj)
}

How to convert Row to json in Spark 2 Scala

Is there a simple way to converting a given Row object to json?
Found this about converting a whole Dataframe to json output:
Spark Row to JSON
But I just want to convert a one Row to json.
Here is pseudo code for what I am trying to do.
More precisely I am reading json as input in a Dataframe.
I am producing a new output that is mainly based on columns, but with one json field for all the info that does not fit into the columns.
My question what is the easiest way to write this function: convertRowToJson()
def convertRowToJson(row: Row): String = ???
def transformVenueTry(row: Row): Try[Venue] = {
Try({
val name = row.getString(row.fieldIndex("name"))
val metadataRow = row.getStruct(row.fieldIndex("meta"))
val score: Double = calcScore(row)
val combinedRow: Row = metadataRow ++ ("score" -> score)
val jsonString: String = convertRowToJson(combinedRow)
Venue(name = name, json = jsonString)
})
}
Psidom's Solutions:
def convertRowToJSON(row: Row): String = {
val m = row.getValuesMap(row.schema.fieldNames)
JSONObject(m).toString()
}
only works if the Row only has one level not with nested Row. This is the schema:
StructType(
StructField(indicator,StringType,true),
StructField(range,
StructType(
StructField(currency_code,StringType,true),
StructField(maxrate,LongType,true),
StructField(minrate,LongType,true)),true))
Also tried Artem suggestion, but that did not compile:
def row2DataFrame(row: Row, sqlContext: SQLContext): DataFrame = {
val sparkContext = sqlContext.sparkContext
import sparkContext._
import sqlContext.implicits._
import sqlContext._
val rowRDD: RDD[Row] = sqlContext.sparkContext.makeRDD(row :: Nil)
val dataFrame = rowRDD.toDF() //XXX does not compile
dataFrame
}
You can use getValuesMap to convert the row object to a Map and then convert it JSON:
import scala.util.parsing.json.JSONObject
import org.apache.spark.sql._
val df = Seq((1,2,3),(2,3,4)).toDF("A", "B", "C")
val row = df.first() // this is an example row object
def convertRowToJSON(row: Row): String = {
val m = row.getValuesMap(row.schema.fieldNames)
JSONObject(m).toString()
}
convertRowToJSON(row)
// res46: String = {"A" : 1, "B" : 2, "C" : 3}
I need to read json input and produce json output.
Most fields are handled individually, but a few json sub objects need to just be preserved.
When Spark reads a dataframe it turns a record into a Row. The Row is a json like structure. That can be transformed and written out to json.
But I need to take some sub json structures out to a string to use as a new field.
This can be done like this:
dataFrameWithJsonField = dataFrame.withColumn("address_json", to_json($"location.address"))
location.address is the path to the sub json object of the incoming json based dataframe. address_json is the column name of that object converted to a string version of the json.
to_json is implemented in Spark 2.1.
If generating it output json using json4s address_json should be parsed to an AST representation otherwise the output json will have the address_json part escaped.
Pay attention scala class scala.util.parsing.json.JSONObject is deprecated and not support null values.
#deprecated("This class will be removed.", "2.11.0")
"JSONFormat.defaultFormat doesn't handle null values"
https://issues.scala-lang.org/browse/SI-5092
JSon has schema but Row doesn't have a schema, so you need to apply schema on Row & convert to JSon. Here is how you can do it.
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
def convertRowToJson(row: Row): String = {
val schema = StructType(
StructField("name", StringType, true) ::
StructField("meta", StringType, false) :: Nil)
return sqlContext.applySchema(row, schema).toJSON
}
Essentially, you can have a dataframe which contains just one row. Thus, you can try to filter your initial dataframe and then parse it to json.
I had the same issue, I had parquet files with canonical schema (no arrays), and I only want to get json events. I did as follows, and it seems to work just fine (Spark 2.1):
import org.apache.spark.sql.types.StructType
import org.apache.spark.sql.{DataFrame, Dataset, Row}
import scala.util.parsing.json.JSONFormat.ValueFormatter
import scala.util.parsing.json.{JSONArray, JSONFormat, JSONObject}
def getValuesMap[T](row: Row, schema: StructType): Map[String,Any] = {
schema.fields.map {
field =>
try{
if (field.dataType.typeName.equals("struct")){
field.name -> getValuesMap(row.getAs[Row](field.name), field.dataType.asInstanceOf[StructType])
}else{
field.name -> row.getAs[T](field.name)
}
}catch {case e : Exception =>{field.name -> null.asInstanceOf[T]}}
}.filter(xy => xy._2 != null).toMap
}
def convertRowToJSON(row: Row, schema: StructType): JSONObject = {
val m: Map[String, Any] = getValuesMap(row, schema)
JSONObject(m)
}
//I guess since I am using Any and not nothing the regular ValueFormatter is not working, and I had to add case jmap : Map[String,Any] => JSONObject(jmap).toString(defaultFormatter)
val defaultFormatter : ValueFormatter = (x : Any) => x match {
case s : String => "\"" + JSONFormat.quoteString(s) + "\""
case jo : JSONObject => jo.toString(defaultFormatter)
case jmap : Map[String,Any] => JSONObject(jmap).toString(defaultFormatter)
case ja : JSONArray => ja.toString(defaultFormatter)
case other => other.toString
}
val someFile = "s3a://bucket/file"
val df: DataFrame = sqlContext.read.load(someFile)
val schema: StructType = df.schema
val jsons: Dataset[JSONObject] = df.map(row => convertRowToJSON(row, schema))
if you are iterating through an data frame , you can directly convert the data frame to a new dataframe with json object inside and iterate that
val df_json = df.toJSON
I combining the suggestion from: Artem, KiranM and Psidom. Did a lot of trails and error and came up with this solutions that I tested for nested structures:
def row2Json(row: Row, sqlContext: SQLContext): String = {
import sqlContext.implicits
val rowRDD: RDD[Row] = sqlContext.sparkContext.makeRDD(row :: Nil)
val dataframe = sqlContext.createDataFrame(rowRDD, row.schema)
dataframe.toJSON.first
}
This solution worked, but only while running in driver mode.

akka-stream - How to treat the last element of a stream differently in a Flow/Graph

I'm trying to implement an Akka Streams Flow that will convert a stream of JSON objects to a stream of a single array of JSON objects. I can use Concat to add an "[" before and "]" after, as well as Zip to insert commas in between elements, but I can't figure out how to not insert the final comma.
The code I have so far is:
trait JsonStreamSupport {
protected def toJsonArrayString[T : Writes] = Flow[T].map(Json.toJson(_)).map(_.toString()).via(jsonArrayWrapper)
private[this] val jsonArrayWrapper: Flow[String, String, NotUsed] = Flow.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val start = Source.single("[")
val comma = Source.repeat(",")
val end = Source.single("]")
val concat = b.add(Concat[String](3))
val zip = b.add(Zip[String,String])
comma ~> zip.in1
start ~> concat.in(0)
zip.out.map({case (msg,delim) => msg + delim}) ~> concat.in(1)
end ~> concat.in(2)
FlowShape(zip.in0, concat.out)
})
}
Currently the output is:
[{"key":"value},{"key","value"},]
but I need it to be
[{"key":"value},{"key","value"}] (without final comma), where each element of the array is still a distinct element of the stream so can be, for example, sent over chunked HTTP separately.
just found out about intersperse which is exactly what you need, and much simpler than what I suggested in the first place:
http://doc.akka.io/api/akka/2.4.4/index.html#akka.stream.scaladsl.Flow#intersperse[T%3E:Out]%28start:T,inject:T,end:T%29:FlowOps.this.Repr[T]

Play 2.1 Reading JSON Objects in order

JSON to Parse: http://www.dota2.com/jsfeed/heropickerdata?v=18874723138974056&l=english
Hero Class and JSON Serialization
case class Hero(
var id:Option[Int],
name: String,
bio: String,
var trueName:Option[String]
){}
implicit val modelReader: Reads[Hero] = Json.reads[Hero]
Reading Data
val future: Future[play.api.libs.ws.Response] = WS.url("http://www.dota2.com/jsfeed/heropickerdata?v=18874723138974056&l=english").get()
val json = Json.parse(Await.result(future,5 seconds).body).as[Map[String, Hero]]
var i = 1
json.foreach(p => {
p._2.trueName = Some(p._1)
p._2.id = Some(i)
p._2.commitToDatabase
i += 1
})
I need to get the id of each hero. The order of heros in the json matches their id. Obviously a map is unordered and wont work. Does anyone have any other ideas?
I have tried to use a LinkedHashMap. I even tried to make an implicit Reads for LinkedHashMap but I've failed. If anyone thinks that this is the answer then would you please give me some guidance?
It keeps just saying "No Json deserializer found for type scala.collection.mutable.LinkedHashMap[String,models.Hero]. Try to implement an implicit Reads or Format for this type.". I have the trait imported into the file i'm trying to read from. I have a funny feeling that the last line in my Reads is the problem. i think I can't just do the asInstanceOf, however I have no other ideas of how to do this reads.
LinkedHashMap Implicit Reads Code: http://pastebin.com/cf5NpSCX
You can try extracting data in order from the JsObject returned by Json.parse directly, possibly like this:
val json = Json.parse(Await.result(future,5 seconds).body)
val heroes: Map[String, Hero] = json match {
case obj: JsObject =>
obj.fields.zipWithIndex.map{ case ((name: String, heroJson: JsValue), id) =>
heroJson.asOpt[Hero].map{ _.copy(id = Some(id)) }
}.flatten.toMap
case _ = > Seq.empty
}
I don't believe you'll need an order-preserving map anymore since the ids are generated and fixed.