Circe decoder only array size - json

I have an Api that I need to do multiple concurrent call with cursor, but I'm interested in only knowing the size of an array in the Json only, I wonder if is it possible to make a custom decoder with Circe or Jsoniter for it?
The Json look simple only like this:
{
"myArray":[
{
"multipleJsonObjectsHere"
}
],
"cursor":"abc"
}
I just want to retrieve only the size of myArray to know how many items are there in it

Using jsoniter-scala you can write a custom codec that will parse/serialize JSON array to/from Int value:
case class Message(myArray: Int, cursor: String)
implicit val messageCodec: JsonValueCodec[Message] = {
implicit val intCodec: JsonValueCodec[Int] = new JsonValueCodec[Int] {
override def decodeValue(in: JsonReader, default: Int): Int =
if (in.isNextToken('[')) {
var i = 0
if (!in.isNextToken(']')) {
in.rollbackToken()
while ({
in.skip()
i += 1
in.isNextToken(',')
}) ()
if (!in.isCurrentToken(']')) in.arrayEndOrCommaError()
}
i
} else in.readNullOrTokenError(default, '[')
override def encodeValue(x: Int, out: JsonWriter): _root_.scala.Unit = {
out.writeArrayStart()
var i = x
while (i > 0) {
out.writeNull()
i -= 1
}
out.writeArrayEnd()
}
override def nullValue: Int = 0
}
JsonCodecMaker.make[Message]
}
val json1 =
"""{
| "myArray":[
| {
| "multipleJsonObjects": "Here"
| }
| ],
| "cursor":"abc"
|}""".stripMargin.getBytes(StandardCharsets.UTF_8)
assert(readFromArray[Message](json1) == Message(1, "abc"))
val json2 = """{"myArray":[null,null,null],"cursor":"www"}""".getBytes(StandardCharsets.UTF_8)
assert(readFromArray[Message](json2) == Message(3, "www"))

Related

JSON decode from object to string, and encode it back

I am trying to decode a JSON string to case class and also transform one of the field from Object to JSON String.
Currently, I have only finished the decoding part, but don't know how to implement the encoding part.
Here is my code:
case class Foo(objStr: String)
implicit val encoder: Encoder[Foo] = deriveEncoder[Foo]
implicit val decoder: Decoder[Foo] = deriveDecoder[Foo].prepare { cursor =>
cursor.withFocus { json =>
json.mapObject { o =>
o.add("objStr", o("objStr").map(_.noSpaces.asJson).getOrElse(Json.Null))
}
}
}
val json =
"""{
| "objStr": {
| "0": 1,
| "1": 3.4,
| "2": "abc"
| }
|}""".stripMargin
val foo = io.circe.parser.decode[Foo](json).right.get
println(foo)
// output `{"0":1,"1":3.4,"2":"abc"}` as expected
println(foo.asJson.noSpaces)
// output `{"objStr":"{\"0\":1,\"1\":3.4,\"2\":\"abc\"}"}`
// but I expect `{"objStr":{"0":1,"1":3.4,"2":"abc"}}`
Please help me to finish the encoding part.
Thank you.
Updated:
Found a workable solution:
implicit val encoder: Encoder[Foo] = deriveEncoder[Foo].mapJson { json =>
json.mapObject { o =>
o.add("objStr", o("objStr").flatMap(_.asString).flatMap(parse(_).toOption).getOrElse(Json.Null))
}
}

Skipping serialization of null values depending on the type

I started working with moshi a couple of weeks ago, so maybe I am missing something trivial, but I spent already quite a bit of time trying to fix this without success, so here is my question.
Having the following reproducible code:
fun main() {
val moshi = Moshi.Builder().add(OptionalAdapter).build()
val objectToSerialize = DummyObject()
val json = moshi.adapter(DummyObject::class.java).serializeNulls().toJson(objectToSerialize)
println(json)
}
#JsonClass(generateAdapter = true)
data class DummyObject(val value: Int=123, val someNullable: String? = null,
val someNotPresent: Optional<String> = Optional.NotPresent,
val somePresent: Optional<String> = Optional.Present("aaaa"))
class OptionalAdapter<T>(private val valueAdapter: JsonAdapter<T>) : JsonAdapter<Optional<T>>() {
#Suppress("UNCHECKED_CAST")
override fun fromJson(reader: JsonReader) = Optional.Present(valueAdapter.fromJson(reader) as T)
override fun toJson(writer: JsonWriter, value: Optional<T>?) {
when (value) {
is Optional.NotPresent -> writer.nullValue()
is Optional.Present -> valueAdapter.serializeNulls().toJson(writer, value.value)
}
}
companion object Factory : JsonAdapter.Factory {
override fun create(type: Type, annotations: Set<out Annotation>, moshi: Moshi): JsonAdapter<*>? {
return if (Types.getRawType(type) == Optional::class.java && annotations.isEmpty()) {
val valueType = if(type is ParameterizedType) {
type.actualTypeArguments.get(0)
} else {
//Should not happen
throw IllegalArgumentException()
}
return OptionalAdapter(moshi.adapter<Any>(valueType).nullSafe())
} else {
null
}
}
}
}
sealed class Optional<out T> {
val provided get() = this !is NotPresent
abstract val value: T
object NotPresent : Optional<Nothing>() {
// have the IDE raise an error if the user knows a type is missing but still tries to access a value
#Deprecated(
"Cannot access a missing value",
level = DeprecationLevel.ERROR,
replaceWith = ReplaceWith("TODO(\"value is missing\")")
)
override val value: Nothing
get() = error("cannot access provided field")
}
data class Present<out T>(override val value: T) : Optional<T>()
}
I would like to serialize as {"value":123,"someNullable":null,"somePresent":"aaaa"} instead of {"value":123,"someNullable":null,"someNotPresent":null,"somePresent":"aaaa"}, which is what is doing now.
Basically, I want to skip the serialization in case the type is Optional.NotPresent. Any suggestion?
The solution I ended up with:
override fun toJson(writer: JsonWriter, value: Optional<T>?) {
when (value) {
is Optional.NotPresent -> {
val wasSerializeNulls = writer.serializeNulls
writer.serializeNulls = false
try {
writer.nullValue()
} finally {
writer.serializeNulls = wasSerializeNulls
}
}
is Optional.Present -> valueAdapter.serializeNulls().toJson(writer, value.value)
}
}

"data to be connected 'chisel3.core.UInt#103' must be hardware, not a bare Chisel type" when rewrite OpenSoCFabric1.1.2 from Chisel2 to Chisel3

I am trying to rewrite OpenSoCFaric-1.1.2 from chisel2 to chisel3. But I encounter error messages "data to be connected 'chisel3.core.UInt#103' must be hardware, not a bare Chisel type" for below code:
File: packettoFlit.scala class: PacketToFlit
val flitWidth = Flit.fromBits(0.U, parms).getWidth
File:channel.scala object: Flit
object Flit {
def head(h: HeadFlit) : Flit = {
val f = new Flit(h.parms)
f.x := f.union.pack("Head", h)
f
}
def body(b: BodyFlit) : Flit = {
val f = new Flit(b.parms)
f.x := f.union.pack("Body", b)
f
}
def fromBits(n: UInt, parms: Parameters) : Flit = {
val f = new Flit(parms)
f.x := n
f
}
/*
def zeroHead(parms: Parameters) : HeadFlit = {
val x = new HeadFlit(parms)
x.
}
*/
}
And now I don't have good ideas about how to rewrite such code segments to fix the error. Could you give some help or suggestions ? Thanks a lot!
We would need to see more of the error message (line number) and more of the code to provide a definitive answer, but our guess is that somewhere a UInt(x.W) is being used where a literal (y.U) is expected, or a Wire()/WireInit() wrapper is missing.
NOTE: LBL is actively migrating this code to Chisel3. You might be better off waiting for their work to be published.
Below is source code for class Flit.
class Flit(parms: Parameters) extends Bundle {
val union = new BitUnion(Map("Head" -> new HeadFlit(parms), "Body" -> new BodyFlit(parms)))
val x = UInt(union.width.W)
val numVCs = parms.get[Int]("numVCs")
def asHead(dummy: Int = 0) : HeadFlit = union.unpack[HeadFlit]("Head", x)
def asBody(dummy: Int = 0) : BodyFlit = union.unpack[BodyFlit]("Body", x)
def whenHead(block: HeadFlit => Unit) { union.whenTag[HeadFlit]("Head", x)(block) }
def whenBody(block: BodyFlit => Unit) { union.whenTag[BodyFlit]("Body", x)(block) }
def isHead(dummy: Int = 0) : Bool = union.tagEquals("Head", x)
def isBody(dummy: Int = 0) : Bool = union.tagEquals("Body", x)
def isTail(dummy: Int = 0) : Bool = {
val tailBit = Bool()
when (isHead()) {
tailBit := union.unpack[HeadFlit]("Head", x).isTail
} .otherwise {
tailBit := union.unpack[BodyFlit]("Body", x).isTail
}
tailBit
}
def getVCPort(dummy: Int = 0) : UInt = {
val vcBits = UInt(log2Ceil(numVCs).W)
when (isHead()) {
vcBits := union.unpack[HeadFlit]("Head", x).vcPort
} .otherwise {
vcBits := union.unpack[BodyFlit]("Body", x).vcPort
}
vcBits
}
//override def clone = { new Flit(parms).asInstanceOf[this.type] }
override def cloneType: this.type = new Flit(parms).asInstanceOf[this.type]
// override def width : Int = {x.width}
}
Thanks a lot!
Bibo

Scala : Retry with try/catch for exception handling

I am trying to add a retry logic for JSON conversion. When converting an object to json, I am retrying for 3 times if there is any exception. I am doing :
var mapper = new ObjectMapper() with ScalaObjectMapper
intializeMapper( )
def intializeMapper() = {
// jackson library does not support seralization and deserialization of
// of scala classes like List and Map, this is needed to support it
mapper.registerModule( DefaultScalaModule )
// enables parsing of NaN. Enabling it here as JsonUtil class currently in
// use supports it.
mapper.configure(JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS, true )
mapper.setSerializationInclusion(Include.NON_NULL)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
}
def getPersonRDD(result: DataFrame): RDD[(String, String)] = {
val finalValue = result.rdd.map({
r =>
val customerId = r.getAs[String](CUSTOMER_ID)
val itemId = r.getAs[Map[String, Int]](ITEM_ID)
val itemName = r.getAs[Map[String, Int]](ITEM_NAME)
val person = Person(itemId, itemName)
val jsonString = toJson(person)
(customerId, jsonString)
})
return finalValue
}
def fromJson(json: String, clazz: Class[_]) = {
mapper.readValue(json, clazz)
}
def toJson(value: Any): String = {
var jsonString: String = " "
jsonString = mapper.writeValueAsString(value)
try {
fromJson(jsonString, clazz)
return jsonString
} catch {
case Exception => {
publishMetrics(PARSING_EXCEPTION, 1.0)
val result = util.Try(retry() {
jsonString = mapper.writeValueAsString(value)
val features = fromJson(jsonString, clazz)
})
result match {
case util.Success(value) => jsonString
case util.Failure(error) => {
log.error("Error while parsing JSON " + jsonString)
return jsonString
}
}
}
}
}
// Returning T, throwing the exception on failure
#annotation.tailrec
def retry[T](n: Int = 3)(fn: => T): T = {
util.Try {
fn
} match {
case util.Success(x) => x
case _ if n > 1 => retry(n - 1)(fn)
case util.Failure(e) => throw e
}
}
case class Person(itemId: Map[String, Int], itemName: Map[String, Int]) extends Serializable
Is this correct ? I am new to Scala. Can someone suggest me if there is any better way for achieving this ? Is there predefined retry logic available in Scala ? The reason I am trying to add retry logic for JSON conversion is due to Jackson version I use(which I can't change for now), sometimes my writeValueAsString results in incomplete JSON.
You retry function seems correct. The only flaw I can think of is that if you expect something would fail it's better just make the return type Try[T], so you can handle it outside in the scala way.
Here is one of my implementation:
def retry[T](n: Int)(block: => T): Try[T] = {
val stream = Stream.fill(n)(Try(block))
stream find (_.isSuccess) getOrElse stream.head
}

How to parse JSON in Scala using standard Scala classes?

I am using the build in JSON class in Scala 2.8 to parse JSON code. I don't want to use the Liftweb one or any other due to minimizing dependencies.
The way I am doing it seems too imperative, is there a better way to do it?
import scala.util.parsing.json._
...
val json:Option[Any] = JSON.parseFull(jsonString)
val map:Map[String,Any] = json.get.asInstanceOf[Map[String, Any]]
val languages:List[Any] = map.get("languages").get.asInstanceOf[List[Any]]
languages.foreach( langMap => {
val language:Map[String,Any] = langMap.asInstanceOf[Map[String,Any]]
val name:String = language.get("name").get.asInstanceOf[String]
val isActive:Boolean = language.get("is_active").get.asInstanceOf[Boolean]
val completeness:Double = language.get("completeness").get.asInstanceOf[Double]
}
This is a solution based on extractors which will do the class cast:
class CC[T] { def unapply(a:Any):Option[T] = Some(a.asInstanceOf[T]) }
object M extends CC[Map[String, Any]]
object L extends CC[List[Any]]
object S extends CC[String]
object D extends CC[Double]
object B extends CC[Boolean]
val jsonString =
"""
{
"languages": [{
"name": "English",
"is_active": true,
"completeness": 2.5
}, {
"name": "Latin",
"is_active": false,
"completeness": 0.9
}]
}
""".stripMargin
val result = for {
Some(M(map)) <- List(JSON.parseFull(jsonString))
L(languages) = map("languages")
M(language) <- languages
S(name) = language("name")
B(active) = language("is_active")
D(completeness) = language("completeness")
} yield {
(name, active, completeness)
}
assert( result == List(("English",true,2.5), ("Latin",false,0.9)))
At the start of the for loop I artificially wrap the result in a list so that it yields a list at the end. Then in the rest of the for loop I use the fact that generators (using <-) and value definitions (using =) will make use of the unapply methods.
(Older answer edited away - check edit history if you're curious)
This is the way I do the pattern match:
val result = JSON.parseFull(jsonStr)
result match {
// Matches if jsonStr is valid JSON and represents a Map of Strings to Any
case Some(map: Map[String, Any]) => println(map)
case None => println("Parsing failed")
case other => println("Unknown data structure: " + other)
}
I like #huynhjl's answer, it led me down the right path. However, it isn't great at handling error conditions. If the desired node does not exist, you get a cast exception. I've adapted this slightly to make use of Option to better handle this.
class CC[T] {
def unapply(a:Option[Any]):Option[T] = if (a.isEmpty) {
None
} else {
Some(a.get.asInstanceOf[T])
}
}
object M extends CC[Map[String, Any]]
object L extends CC[List[Any]]
object S extends CC[String]
object D extends CC[Double]
object B extends CC[Boolean]
for {
M(map) <- List(JSON.parseFull(jsonString))
L(languages) = map.get("languages")
language <- languages
M(lang) = Some(language)
S(name) = lang.get("name")
B(active) = lang.get("is_active")
D(completeness) = lang.get("completeness")
} yield {
(name, active, completeness)
}
Of course, this doesn't handle errors so much as avoid them. This will yield an empty list if any of the json nodes are missing. You can use a match to check for the presence of a node before acting...
for {
M(map) <- Some(JSON.parseFull(jsonString))
} yield {
map.get("languages") match {
case L(languages) => {
for {
language <- languages
M(lang) = Some(language)
S(name) = lang.get("name")
B(active) = lang.get("is_active")
D(completeness) = lang.get("completeness")
} yield {
(name, active, completeness)
}
}
case None => "bad json"
}
}
I tried a few things, favouring pattern matching as a way of avoiding casting but ran into trouble with type erasure on the collection types.
The main problem seems to be that the complete type of the parse result mirrors the structure of the JSON data and is either cumbersome or impossible to fully state. I guess that is why Any is used to truncate the type definitions. Using Any leads to the need for casting.
I've hacked something below which is concise but is extremely specific to the JSON data implied by the code in the question. Something more general would be more satisfactory but I'm not sure if it would be very elegant.
implicit def any2string(a: Any) = a.toString
implicit def any2boolean(a: Any) = a.asInstanceOf[Boolean]
implicit def any2double(a: Any) = a.asInstanceOf[Double]
case class Language(name: String, isActive: Boolean, completeness: Double)
val languages = JSON.parseFull(jstr) match {
case Some(x) => {
val m = x.asInstanceOf[Map[String, List[Map[String, Any]]]]
m("languages") map {l => Language(l("name"), l("isActive"), l("completeness"))}
}
case None => Nil
}
languages foreach {println}
val jsonString =
"""
|{
| "languages": [{
| "name": "English",
| "is_active": true,
| "completeness": 2.5
| }, {
| "name": "Latin",
| "is_active": false,
| "completeness": 0.9
| }]
|}
""".stripMargin
val result = JSON.parseFull(jsonString).map {
case json: Map[String, List[Map[String, Any]]] =>
json("languages").map(l => (l("name"), l("is_active"), l("completeness")))
}.get
println(result)
assert( result == List(("English", true, 2.5), ("Latin", false, 0.9)) )
You can do like this! Very easy to parse JSON code :P
package org.sqkb.service.common.bean
import java.text.SimpleDateFormat
import org.json4s
import org.json4s.JValue
import org.json4s.jackson.JsonMethods._
//import org.sqkb.service.common.kit.{IsvCode}
import scala.util.Try
/**
*
*/
case class Order(log: String) {
implicit lazy val formats = org.json4s.DefaultFormats
lazy val json: json4s.JValue = parse(log)
lazy val create_time: String = (json \ "create_time").extractOrElse("1970-01-01 00:00:00")
lazy val site_id: String = (json \ "site_id").extractOrElse("")
lazy val alipay_total_price: Double = (json \ "alipay_total_price").extractOpt[String].filter(_.nonEmpty).getOrElse("0").toDouble
lazy val gmv: Double = alipay_total_price
lazy val pub_share_pre_fee: Double = (json \ "pub_share_pre_fee").extractOpt[String].filter(_.nonEmpty).getOrElse("0").toDouble
lazy val profit: Double = pub_share_pre_fee
lazy val trade_id: String = (json \ "trade_id").extractOrElse("")
lazy val unid: Long = Try((json \ "unid").extractOpt[String].filter(_.nonEmpty).get.toLong).getOrElse(0L)
lazy val cate_id1: Int = (json \ "cate_id").extractOrElse(0)
lazy val cate_id2: Int = (json \ "subcate_id").extractOrElse(0)
lazy val cate_id3: Int = (json \ "cate_id3").extractOrElse(0)
lazy val cate_id4: Int = (json \ "cate_id4").extractOrElse(0)
lazy val coupon_id: Long = (json \ "coupon_id").extractOrElse(0)
lazy val platform: Option[String] = Order.siteMap.get(site_id)
def time_fmt(fmt: String = "yyyy-MM-dd HH:mm:ss"): String = {
val dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
val date = dateFormat.parse(this.create_time)
new SimpleDateFormat(fmt).format(date)
}
}
This is the way I do the Scala Parser Combinator Library:
import scala.util.parsing.combinator._
class ImprovedJsonParser extends JavaTokenParsers {
def obj: Parser[Map[String, Any]] =
"{" ~> repsep(member, ",") <~ "}" ^^ (Map() ++ _)
def array: Parser[List[Any]] =
"[" ~> repsep(value, ",") <~ "]"
def member: Parser[(String, Any)] =
stringLiteral ~ ":" ~ value ^^ { case name ~ ":" ~ value => (name, value) }
def value: Parser[Any] = (
obj
| array
| stringLiteral
| floatingPointNumber ^^ (_.toDouble)
|"true"
|"false"
)
}
object ImprovedJsonParserTest extends ImprovedJsonParser {
def main(args: Array[String]) {
val jsonString =
"""
{
"languages": [{
"name": "English",
"is_active": true,
"completeness": 2.5
}, {
"name": "Latin",
"is_active": false,
"completeness": 0.9
}]
}
""".stripMargin
val result = parseAll(value, jsonString)
println(result)
}
}
scala.util.parsing.json.JSON is deprecated.
Here is another approach with circe. FYI documentation: https://circe.github.io/circe/cursors.html
Add the dependency in build.sbt, I used scala 2.13.4, note the scala version must align with the library version.
val circeVersion = "0.14.0-M2"
libraryDependencies ++= Seq(
"io.circe" %% "circe-core" % circeVersion,
"io.circe" %% "circe-generic" % circeVersion,
"io.circe" %% "circe-parser" % circeVersion
)
Example 1:
case class Person(name: String, age: Int)
object Main {
def main(args: Array[String]): Unit = {
val input =
"""
|{
| "kind": "Listing",
| "data": [
| {
| "name": "Frodo",
| "age": 51
| },
| {
| "name": "Bilbo",
| "age": 60
| }
| ]
|}
|""".stripMargin
implicit val decoderPerson: Decoder[Person] = deriveDecoder[Person] // decoder required to parse to custom object
val parseResult: Json = circe.parser.parse(input).getOrElse(Json.Null)
val data: ACursor = parseResult.hcursor.downField("data") // get the data field
val personList: List[Person] = data.as[List[Person]].getOrElse(null) // parse the dataField to a list of Person
for {
person <- personList
} println(person.name + " is " + person.age)
}
}
Example 2, json has an object within an object:
case class Person(name: String, age: Int, position: Position)
case class Position(x: Int, y: Int)
object Main {
def main(args: Array[String]): Unit = {
val input =
"""
|{
| "kind": "Listing",
| "data": [
| {
| "name": "Frodo",
| "age": 51,
| "position": {
| "x": 10,
| "y": 20
| }
| },
| {
| "name": "Bilbo",
| "age": 60,
| "position": {
| "x": 75,
| "y": 85
| }
| }
| ]
|}
|""".stripMargin
implicit val decoderPosition: Decoder[Position] = deriveDecoder[Position] // must be defined before the Person decoder
implicit val decoderPerson: Decoder[Person] = deriveDecoder[Person]
val parseResult = circe.parser.parse(input).getOrElse(Json.Null)
val data = parseResult.hcursor.downField("data")
val personList = data.as[List[Person]].getOrElse(null)
for {
person <- personList
} println(person.name + " is " + person.age + " at " + person.position)
}
}