How do I de-serialize this json? - json

I am working on a project where I need to access currency rates once a day so I am trying to use this json.
To read this, I am simply getting the text of the URL and then trying to use the JSONReader to de-serialize.
val url = URL("https://www.floatrates.com/daily/usd.json")
val stream = url.openStream()
url.readText()
val jsonReader = JsonReader(InputStreamReader(stream))
jsonReader.isLenient = true
jsonReader.beginObject()
while (jsonReader.hasNext()) {
val codeName:String = jsonReader.nextName()
jsonReader.beginObject();
var code:String? = null
var rate = 0.0
while (jsonReader.hasNext()) {
val name:String = jsonReader.nextName()
when(name){
"code" -> {
code = jsonReader.nextString()
break
}
"rate" -> {
rate = jsonReader.nextDouble()
break
}
else -> {
jsonReader.skipValue()
}
}
code?.let {
rates?.set(it, rate)
}
}
}
jsonReader.endObject();
When I run the code , I get:
Expected BEGIN_OBJECT but was STRING
at
jsonReader.beginObject();
When I try using Gson, with the code below:
var url = URL("https://www.floatrates.com/daily/usd.json").readText()
//url = "[${url.substring(1, url.length - 1)}]"
val gson = Gson()
val currencies:Array<SpesaCurrency> = gson.fromJson(url, Array<SpesaCurrency>::class.java)
I get this error :
Expected BEGIN_OBJECT but was STRING at line 1 column 3 path $[0]
at:
gson.fromJson(url, Array<SpesaCurrency>::class.java)
SpesaCurrency.kt looks like this:
class SpesaCurrency(
val code:String,
val alphaCode:String,
val numericCode:String,
val name:String,
val rate:Float,
val date:String,
val inverseRate:Float
)
Any help is greatly appreciated.

I think there is one
jsonReader.endObject();
missing in your code. That imbalance causes the program to fail after the first object has been read. Add it after the inner
while (jsonReader.hasNext()) {
...
}
loop.

Related

JSONException Error upon parsing an array of values that I intend to handle in a RecyclerView

I am trying to parse a JSONArray using Volley and put it in a RecyclerView. I am stuck on this problem for days now and I really need some help. Everything seems to be fine but it just gives me a FATAL Exception regarding a JSONException. I have also used Postman and the Data Request seems to be fine. I just think there is a problem in what I am doing in my Kotlin code. Here is my code:
Get Data Function
val getLoanRecData = BASE_URL + "getLoanData"
val queue = Volley.newRequestQueue(this.activity)
val loanAppRequest = object : StringRequest(Method.POST,getLoanRecData,
Response.Listener { response ->
val jsonObject = JSONObject(response)
if(jsonObject.get("response").equals("Success"))
{
val jsonArray = jsonObject.getJSONArray("data")
for (i in 0..jsonArray.length()-1){
var jo = jsonArray.getJSONObject(i)
val id = jo.get("col_borrower_id").toString()
val loan_id = jo.get("col_loan_assignid").toString()
val loan_type = jo.get("col_loan_type").toString()
val due_date = jo.get("col_due_date").toString()
val loan_status = jo.get("col_status").toString()
val user = LoanRecordModel(id, loan_id, loan_type, loan_status, due_date)
list.add(user)
}
}else{
Toast.makeText(activity,jsonObject.get("response").toString(),Toast.LENGTH_SHORT).show()
}
}, Response.ErrorListener
{
error -> Toast.makeText(activity, error.toString(), Toast.LENGTH_SHORT).show()
}){
override fun getParams(): HashMap<String, String>
{
val sp = activity?.getSharedPreferences("user_data", Context.MODE_PRIVATE)
val emp_id = sp?.getString("user_id", "")
val map = HashMap<String,String>()
map["request"] = "SENT"
map["emp_id"] = emp_id.toString()
return map
}
}
queue.add(loanAppRequest)
I have also used serialized name on the data class for the RecyclerView because I thought the program only has problems regarding obtaining the data gathered from JSON, however that solved nothing and I still got the same error.
data class LoanRecordModel(
#SerialName("col_borrower_id")
val emp_id: String,
#SerialName("col_loan_assignid")
val loan_id: String,
#SerialName("col_loan_type")
val loan_type: String,
#SerialName("col_due_date")
val due_date: String,
#SerialName("col_status")
val loan_status: String)
These are the errors im getting:

Kotlin - parsing json string throws MalformedJsonException: Unterminated object

I am trying to save a value to a jsonb DB column:
val deliveryAddressAsJson = deliveryAddress?.toJson()
val lat = deliveryAddressAsJson?.get("latitude")
val lng = deliveryAddressAsJson?.get("longitude")
val dataJson = jsonObject("comment" to "KARTKOORD:#LE#$lat#$lng# #")
val values = mapOf(
"type" to EventType.RESOLVED.dbName,
"created_by" to ctx.userId,
"data" to dataJson.toPgObject(),
"package_id" to packageId
)
#Language("PostgreSQL")
val sql = """insert into package_event(type, created_by, data, package_id) values (:type, :created_by, :data, :package_id)""".trimMargin()
insert(ctx, sql, values).bind()
I can see that the data is saved like this:
data -> {Collections$SingletonMap#6348} size = 1
key = "data"
value = {Collections$SingletonMap#6348} size = 1
key = "comment"
value = "KARTKOORD:#LE#59.8098962#10.7809297# #"
But, if I try to parse it:
val resolvedPackageEvent = fetchRows(ctx, queryOf("select * from package_event where package_id = ? and type = 'resolved'", packageId)).first()
val data = parseJson(resolvedPackageEvent.string("data"))
val deliveryAddress = data.get("comment")
I get an exception thrown:
com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated object at line 1 column 20 path $.comment
If I try to get the value with jsonObject method like this:
fun Entity.jsonObject(key: String): JsonObject = when (val v = this[key]) {
is String -> parseJson(v)
else -> v as JsonObject
}
resolvedPackageEvent.jsonObject("data")
I get an exception:
java.lang.ClassCastException: class java.util.Collections$SingletonMap cannot be cast to class com.google.gson.JsonObject (java.util.Collections$SingletonMap is in module java.base of loader 'bootstrap'; com.google.gson.JsonObject is in unnamed module of loader 'app')
How should I parse this json string?
Your data is not an String. As you can see in the debugger, your data is something like
{
"data": {
"comment": "KARTKOORD:#LE#59.8098962#10.7809297# #"
}
}
I don't have the gson syntax at hand, so I can't provide precise code for retrieving the value. You can probably do something like
val result: JsonObject = parseJson(resolvedConsignmentEvent)
val data: JsonObject = result.get("data")
val deliveryAddress: String = data.get("comment")

How to convert Scala Document to JSON in Scala

I want to convert variable message which is of type scala.Seq[Scala.Document] to JSON format in following code:
path("getMessages"){
get {
parameters('roomname.as[String]) {
(roomname) =>
try {
val messagesByGroupName = MongoDatabase.collectionForChat.find(equal("groupChatName",roomname)).toFuture()
val messages = Await.result(messagesByGroupName,60.seconds)
println("Messages:"+messages)
complete(messages)
}
catch {
case e:TimeoutException =>
complete("Reading file timeout.")
}
}
}
But it is giving me error on complete(messages) line. It is not accepting message of that type.
I tried to convert it into JSON by using following :
import play.api.libs.json._
object MyJsonProtocol{
implicit object ChatFormat extends Format[Chat] {
def writes(c: Chat) : JsValue = {
val chatSeq = Seq (
"sender" -> JsString(c.sender),
"receiver" -> JsString(c.receiver),
"message" -> JsString(c.message),
"groupChatName" -> JsString(c.groupChatName),
)
JsObject(chatSeq)
}
def reads(value: JsValue) = {
JsSuccess(Chat("","","",""))
}
}
}
But it is not working.
My Chat.scala class is as follows:
import play.api.libs.json.{Json, Reads, Writes}
class Chat(var sender:String,var receiver:String,var message:String, var groupChatName:String){
def setSenderName(senderName:String) = {
sender = senderName
}
def setReceiverName(receiverName:String) = {
receiver = receiverName
}
def setMessage(getMessage:String) = {
message = getMessage
}
def setGroupChatName(chatName:String) = {
groupChatName = chatName
}
}
object Chat {
def apply(sender: String, receiver: String, message: String, groupname: String): Chat
= new Chat(sender, receiver, message,groupname)
def unapply(arg: Chat): Option[(String, String, String,String)] = ???
implicit val requestReads: Reads[Chat] = Json.reads[Chat]
implicit val requestWrites: Writes[Chat] = Json.writes[Chat]
}
I am also not able to figure out what to write in unapply method.
I am new to scala and akka.
EDIT:
My MongoDatabase.scala which has collection is as follows:
object MongoDatabase {
val chatCodecProvider = Macros.createCodecProvider[Chat]()
val codecRegistry = CodecRegistries.fromRegistries(
CodecRegistries.fromProviders(chatCodecProvider),
DEFAULT_CODEC_REGISTRY
)
implicit val system = ActorSystem("Scala_jwt-App")
implicit val executor: ExecutionContext = system.dispatcher
val mongoClient: MongoClient = MongoClient()
val databaseName = sys.env("database_name")
// Getting mongodb database
val database: MongoDatabase = mongoClient.getDatabase(databaseName).withCodecRegistry(codecRegistry)
val registrationCollection = sys.env("register_collection_name")
val chatCollection = sys.env("chat_collection")
// Getting mongodb collection
val collectionForUserRegistration: MongoCollection[Document] = database.getCollection(registrationCollection)
collectionForUserRegistration.drop()
val collectionForChat: MongoCollection[Document] = database.getCollection(chatCollection)
collectionForChat.drop()
}
And if try to change val collectionForChat: MongoCollection[Document] = database.getCollection(chatCollection)
to
val collectionForChat: MongoCollection[Chat] = database.getCollection[Chat](chatCollection)
then I get error on in saveChatMessage() method below:
def saveChatMessage(sendMessageRequest: Chat) : String = {
val senderToReceiverMessage : Document = Document(
"sender" -> sendMessageRequest.sender,
"receiver" -> sendMessageRequest.receiver,
"message" -> sendMessageRequest.message,
"groupChatName" -> sendMessageRequest.groupChatName)
val chatAddedFuture = MongoDatabase.collectionForChat.insertOne(senderToReceiverMessage).toFuture()
Await.result(chatAddedFuture,60.seconds)
"Message sent"
}
on val chatAddedFuture = MongoDatabase.collectionForChat.insertOne(senderToReceiverMessage).toFuture() this line since it accepts data of type Seq[Document] and I am trying to add data of type Seq[Chat]
I am going to assume that MongoDatabase.collectionForChat.find(equal("groupChatName",roomname)) returns either Seq[Chat], or Chat. Both of them are the same for play.
You have 2 options:
Adding the default format on the companion object:
object Chat {
implicit val format: Format[Chat] = Json.format[Chat]
}
In this case you can delete the object MyJsonProtocol which is not used.
In case you want to keep your own serializers(i.e. MyJsonProtocol), you need to rename MyJsonProtocol into Chat. This way the complete route will be able to find the implicit Format.
create case class for the message object you want to send
for example:
case class MyMessage(sender: String, receiver: String, message: String, groupChatName: String)
You should create Format for the type of case class
implicit val MessageTypeFormat = Json.format[MyMessage]
if complete should get JSON type - then call complete myMessage when myMessage is an instance of MyMessage.
complete(Json.toJson(myMessage))

Iterating through JSON data

I am generating a JSON output in php which i need to use to fill line chart in android application writen in kotlin.
The JSON data is:
[{"reading_temperature":"14","hour":"01"},{"reading_temperature":"14","hour":"02"},{"reading_temperature":"14","hour":"03"},{"reading_temperature":"14","hour":"04"},{"reading_temperature":"14","hour":"05"},{"reading_temperature":"14","hour":"06"},{"reading_temperature":"14","hour":"07"},{"reading_temperature":"14","hour":"08"},{"reading_temperature":"14","hour":"09"},{"reading_temperature":"14","hour":"10"},{"reading_temperature":"14","hour":"11"},{"reading_temperature":"14","hour":"12"},{"reading_temperature":"14","hour":"13"},{"reading_temperature":"14","hour":"14"},{"reading_temperature":"14","hour":"15"},{"reading_temperature":"14","hour":"16"},{"reading_temperature":"14","hour":"17"},{"reading_temperature":"14","hour":"18"},{"reading_temperature":"14","hour":"19"},{"reading_temperature":"14","hour":"20"},{"reading_temperature":"14","hour":"21"},{"reading_temperature":"14","hour":"22"},{"reading_temperature":"14","hour":"23"}]
This is the part of the code where i get the JSON:
override fun doInBackground(vararg params: String?): String? {
var response:String?
try{
response = URL("https://127.0.0.1/weatherStation/temperatureDaily.php").readText(
Charsets.UTF_8
)
}catch (e: Exception){
response = null
}
return response
}
override fun onPostExecute(result: String?) {
super.onPostExecute(result)
try {
val jsonObj = JSONObject(result)
} catch (e: Exception) {
}
}
Now i need to use that data to populate the line chart, this is hardcoded data which i need to exchange with JSON
private fun setupLineChartDataTemperatura() {
val yVals = ArrayList<Entry>()
//This next part i need to dynamically generate from JSON,
//first float value and last string value should be `"hour":"xx"`
// and second float value should be `[{"reading_temperature":"xx"`
yVals.add(Entry(0f, 4f, "0"))
yVals.add(Entry(1f, 5f, "1"))
yVals.add(Entry(1.5f, 4f, "1.5"))
yVals.add(Entry(2f, 5f, "2"))
yVals.add(Entry(3f, 3f, "3"))
yVals.add(Entry(4f, 2f, "4"))
val set1: LineDataSet
set1 = LineDataSet(yVals, "Temperatura")
val dataSets = ArrayList<ILineDataSet>()
dataSets.add(set1)
val data = LineData(dataSets)
chart1.setData(data)
chart1.description.isEnabled = false
chart1.legend.isEnabled = true
chart1.setPinchZoom(false)
chart1.xAxis.enableGridDashedLine(5f, 5f, 0f)
chart1.axisRight.enableGridDashedLine(5f, 5f, 0f)
chart1.axisLeft.enableGridDashedLine(5f, 5f, 0f)
chart1.setDrawGridBackground(true)
chart1.xAxis.labelCount = 11
chart1.xAxis.position = XAxis.XAxisPosition.BOTTOM
chart1.setTouchEnabled(true)
chart1.invalidate()
}
I have managed to do it:
val jsonTemperatureData = JSONArray(result)
for (i in 0 until jsonTemperatureData.length()) {
val item = jsonTemperatureData.getJSONObject(i)
val reading_temperature = item.getString("reading_temperature")
val hour = item.getString("hour")
yVals.add(Entry(hour.toFloat(), reading_temperature.toFloat(), hour.toString()))
}

Consuming Kafka DStream in Spark Streaming Procss

I'm consuming a Kafka topic inside a spark streaming program like this:
import ...
object KafkaStreaming {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("KafkaStreaming").setMaster("local[*]")
val sc = new SparkContext(conf)
val ssc = new StreamingContext(sc, Seconds(10))
val kafkaConf = Map(
...
)
val messages = KafkaUtils.createDirectStream[String, String](
ssc,
LocationStrategies.PreferConsistent,
ConsumerStrategies.Subscribe[String, String](Seq("topic"), kafkaConf)
)
val lines: DStream[String] = messages.map(_.value)
val line: DStream[String] = lines.flatMap(_.split("\n"))
process(line)
ssc.start()
ssc.awaitTermination()
}
def process(line: DStream[String]): Unit =
{
// here is where I want to convert the DStream to JSON
var json: Option[Any] = JSON.parseFull(line) // <--
println(json.getOrElse("json is NULL"))
if(json.isEmpty == false) {
println("NOT FALSE")
var map = json.get.asInstanceOf[Map[String, Any]]
// use every member of JSON document to access the value
map.get("any json element").toString
// do some other manipulation
}
}
}
Inside the process function I want to manipulate each line of string to extract a JSON object out of it and perform further processing and persisting. How can I do it?
Instead of taking a DStream[String], use can use DStream.map and then foreachRDD:
def process(line: String): Unit = ???
And then:
messages
.map(_.value)
.flatMap(_.split("\n"))
.map(process)
.foreachRDD { rdd =>
rdd.foreachPartition { itr =>
// Do stuff with `Iterator[String]` after JSON transformation
}
}