I'm trying to use circe to decode a JSON object into a list of objects. I only want to use some of the fields of the JSON response to create the object, so I feel like I have to create a custom decoder.
The class I want to make a sequence of is defined as follows:
case class Review(Id: String, ProductId: String, Rating: Int)
I tried creating a custom decoder like this:
implicit val reviewDecoder: Decoder[Review] = Decoder.instance { c =>
val resultsC = c.downField("Results")
for {
id <- resultsC.downArray.get[String]("Id")
productId <- resultsC.downArray.get[String]("ProductId")
rating <- resultsC.downArray.get[Int]("Rating")
} yield Review(id, productId, rating)
}
reviewDecoder.decodeJson(json) seems to result in only doing the first result and not all of them.
I have a JSON response like this:
{
"Limit":2,
"Offset":0,
"TotalResults":31,
"Locale":"en_US",
"Results":
[
{"Id":"14518388",
"CID":"21a9436b",
"ProductId":"Product11",
"AuthorId":"jcknwekjcnwjk",
"Rating":3
},
{"Id":"14518035",
"CID":"8d67b6f5",
"ProductId":"Product11",
"AuthorId":"fnkjwernfk",
"Rating":3
}
],
"Includes":{},
"HasErrors":false,
"Errors":[]}
I want to be able to parse this JSON object using circe to create a Seq[Review], but I'm stumped how.
****Edit** Luis' answer does answer this question but say I have a more complicated class I want to create a sequence of:
case class User(id: Int)
case class Review(user: User, ProductId: String, Rating: Int)
How would I be able to create a sequence of Reviews in this case?
I would just use the cursor to getting the Array and then use the generic decoder.
The following code was tested on ammonite, where json is a string containing your sample input.
import $ivy.`io.circe::circe-core:0.11.1`
import $ivy.`io.circe::circe-generic:0.11.1`
import $ivy.`io.circe::circe-parser:0.11.1`
import io.circe.{Decoder, Jsom}
import io.circe.parser.parse
final case class Review(Id: String, ProductId: String, Rating: Int)
implicit val reviewDecoder: Decoder[Review] = io.circe.generic.semiauto.deriveDecoder
parse(json).getOrElse(Json.Null).hcursor.downField("Results").as[List[Review]]
// res: io.circe.Decoder.Result[List[Review]] = Right(List(Review("14518388", "Product11", 3), Review("14518035", "Product11", 3)))
Related
my JSON is supposed to look like this.
{"zip":123, "people":[{"firstname":"Thomas", "lastname":"Tatum"},
{"firstname":"Drew", "lastname":"Uncle"}]}
(I am using import org.json.JSONObject)
I have a MutableList, in the List are Person (it’s a data class with firstname and lastname).
But I don’t know how to get my list items in a JSONObject to fit in json (see below).
val json = JSONObject(
mapOf(
"zip" to 123,
"people" to //I don't know how to get my values here
)
)
Maybe someone can help me.
You could do this
import org.json.JSONObject
data class Person(val firstname: String, val lastname: String)
fun main() {
val people = arrayOf(Person("Thomas", "Tatum"), Person("Drew", "Uncle")) //also works for Lists, doesn't need to be an array
val json = JSONObject(
mapOf(
"zip" to 123,
"people" to people,
)
)
println(json)
//prints: {"zip":123,"people":[{"firstname":"Thomas","lastname":"Tatum"},{"firstname":"Drew","lastname":"Uncle"}]}
}
The JSON output that I am looking for is
{[[1, 1.5, "String1"], [-2, 2.3, "String2"]]}
So I want to have an Array of Arrays and the inner array is storing different types.
How should I store my variables so I can create such JSON in Scala?
I thought of List of Tuples. However, all the available JSON libraries try to convert a Tuple to a map instead of an Array. I am using json4s library.
Here is a custom serializer for those inner arrays using json4s:
import org.json4s._
class MyTupleSerializer extends CustomSerializer[(Int, Double, String)](format => ({
case obj: JArray =>
implicit val formats: Formats = format
(obj(0).extract[Int], obj(1).extract[Double], obj(2).extract[String])
}, {
case (i: Int, d: Double, s: String) =>
JArray(List(JInt(i), JDouble(d), JString(s)))
}))
The custom serialiser converts JArray into a tuple and back again. This will be used wherever the Scala object being read or written has a value of the appropriate tuple type.
To test this against the sample input I have modified it to make it valid JSON by adding a field name:
{"data": [[1, 1.5, "String1"], [-2, 2.3, "String2"]]}
I have defined a container class to match this:
case class MyTupleData(data: Vector[(Int, Double, String)])
The name of the class is not relevant but the field name data must match the JSON field name. This uses Vector rather than Array because Array is really a Java type rather than a Scala type. You can use List if preferred.
import org.json4s.jackson.Serialization.{read, write}
case class MyTupleData(data: Vector[(Int, Double, String)])
object JsonTest extends App {
val data = """{"data": [[1, 1.5, "String1"], [-2, 2.3, "String2"]]}"""
implicit val formats: Formats = DefaultFormats + new MyTupleSerializer
val td: MyTupleData = read[MyTupleData](data)
println(td) // MyTupleData(Vector((1,1.5,String1), (-2,2.3,String2)))
println(write(td)) // {"data":[[1,1.5,"String1"],[-2,2.3,"String2"]]}
}
If you prefer to use a custom class for the data rather than a tuple, the code looks like this:
case class MyClass(i: Int, d: Double, s: String)
class MyClassSerializer extends CustomSerializer[MyClass](format => ({
case obj: JArray =>
implicit val formats: Formats = format
MyClass(obj(0).extract[Int], obj(1).extract[Double], obj(2).extract[String])
}, {
case MyClass(i, d, s) =>
JArray(List(JInt(i), JDouble(d), JString(s)))
}))
Use a List of List rather than List of Tuples.
an easy way to convert list of tuples to list of list is:
val listOfList: List[List[Any]] = listOfTuples.map(_.productIterator.toList)
I would use jackson, which is a java library and can deal with arbitrary datatypes inside collections of type Any/AnyRef, rather than trying to come up with a custom serializer in one of scala json libraries.
To convert scala List to java List use
import collection.JavaConverters._
So, in summary the end list would be:
val javaListOfList: java.util.List[java.util.List[Any]] = listOfTuples.map(_.productIterator.toList.asJava).asJava
Using this solution, you could have arbitrary length tuples in your list and it would work.
import com.fasterxml.jackson.databind.ObjectMapper
import collection.JavaConverters._
object TuplesCollectionToJson extends App {
val tuplesList = List(
(10, false, 43.6, "Text1"),
(84, true, 92.1, "Text2", 'X')
)
val javaList = tuplesList.map(_.productIterator.toList.asJava).asJava
val mapper = new ObjectMapper()
val json = mapper.writeValueAsString(javaList)
println(json)
}
Would produce:
[[10,false,43.6,"Text1"],[84,true,92.1,"Text2","X"]]
PS: Use this solution only when you absolutely have to work with variable types. If your tuple datatype is fixed, its better to create a json4s specific serializer/deserializer.
I am trying to familiarize myself with the PlayJSON library. I have a JSON formatted file like this:
{
"Person": [
{
"name": "Jonathon",
"age": 24,
"job": "Accountant"
}
]
}
However, I'm having difficulty with parsing it properly due to the file having different types (name is a String but age is an Int). I could technically make it so the age is a String and call .toInt on it later but for my purposes, it is by default an integer.
I know how to parse some of it:
import play.api.libs.json.{JsValue, Json}
val parsed: JsValue = Json.parse(jsonFile) //assuming jsonFile is that entire JSON String as shown above
val person: List[Map[String, String]] = (parsed \ "Person").as[List[Map[String, String]]]
Creating that person value throws an error. I know Scala is a strongly-typed language but I'm sure there is something I am missing here. I feel like this is an obvious fix too but I'm not quite sure.
The error produced is:
JsResultException(errors:List(((0)/age,List(JsonValidationError(List(error.expected.jsstring),WrappedArray())))
The error you are having, as explained in the error you are getting, is in casting to the map of string to string. The data you provided does not align with it, because the age is a string. If you want to keep in with this approach, you need to parse it into a type that will handle both strings and ints. For example:
(parsed \ "Person").validate[List[Map[String, Any]]]
Having said that, as #Luis wrote in a comment, you can just use case class to parse it. Lets declare 2 case classes:
case class JsonParsingExample(Person: Seq[Person])
case class Person(name: String, age: Int, job: String)
Now we will create a formatter for each of them on their corresponding companion object:
object Person {
implicit val format: OFormat[Person] = Json.format[Person]
}
object JsonParsingExample {
implicit val format: OFormat[JsonParsingExample] = Json.format[JsonParsingExample]
}
Now we can just do:
Json.parse(jsonFile).validate[JsonParsingExample]
Code run at Scastie.
I have a complicated JSON column whose structure is :
story{
cards: [{story-elements: [{...}{...}{...}}]}
The length of the story-elements is variable. I need to extract a particular JSON block from the story-elements array. For this, I first need to extract the story-elements.
Here is the code which I have tried, but it is giving error:
import org.json4s.{DefaultFormats, MappingException}
import org.json4s.jackson.JsonMethods._
import org.apache.spark.sql.functions._
def getJsonContent(jsonstring: String): (String) = {
implicit val formats = DefaultFormats
val parsedJson = parse(jsonstring)
val value1 = (parsedJson\"cards"\"story-elements").extract[String]
value1
}
val getJsonContentUDF = udf((jsonstring: String) =>
getJsonContent(jsonstring))
input.withColumn("cards",getJsonContentUDF(input("storyDataFrame")))
According to json you provided, story-elements is a an array of json objects, but you trying to extract array as a string ((parsedJson\"cards"\"story-elements").extract[String]).
You can create case class representing on story (like case class Story(description: String, pageUrl: String, ...)) and then instead of extract[String], try extract[List[Story]] or extract[Array[Story]]
If you need just one piece of data from story (e.g. descrition), then you can use xpath-like syntax to get that and then extract List[String]
I have a case class and companion object:
case class Person private(name: String, age: Int)
object Person {
def validAge(age: Int) = {
if (age > 18) age.successNel else "Age is under 18".failureNel
}
def validName(name: String) = {
name.successNel
}
def create(name: String, age: Int) = (validAge(age) |#| validName(name))(Person.apply)
}
I want to use Argonaut to parse some JSON and return a Person OR some errors, as a list. So I need to:
Read the JSON from a string, and validate the string is correctly formed
Decode the JSON into a Person, or List of error strings.
I want to return errors in the form of something I can turn into some more JSON like:
{
errors: ["Error1", "Error2"]
}
I first tried using Argonauts decodeValidation method, which returns a Validation[String, X]. Unfortunately, I need a List of errors.
Any suggestions?
I'm adding this as an answer because it's how I'd solve the problem off the top of my head, but I haven't been keeping up closely with Argonaut development for a while, and I'd love to hear that there's a better way. First for the setup, which fixes a few little issues in yours, and adds a condition for validity on names to make the examples later more interesting:
import scalaz._, Scalaz._
case class Person private(name: String, age: Int)
object Person {
def validAge(age: Int): ValidationNel[String, Int] =
if (age > 18) age.successNel else "Age is under 18".failureNel
def validName(name: String): ValidationNel[String, String] =
if (name.size >= 3) name.successNel else "Name too short".failureNel
def create(name: String, age: Int) =
(validName(name) |#| validAge(age))(Person.apply)
}
And then I'd decode the JSON into a (String, Int) pair before creating the Person:
import argonaut._, Argonaut._
def decodePerson(in: String): ValidationNel[String, Person] =
Parse.decodeValidation(in)(
jdecode2L((a: String, b: Int) => (a, b)
)("name", "age")).toValidationNel.flatMap {
case (name, age) => Person.create(name, age)
}
And then:
scala> println(decodePerson("""{ "name": "", "age": 1 }"""))
Failure(NonEmptyList(Name too short, Age is under 18))
Note that this doesn't accumulate errors in more complex cases—e.g. if the value of the name field is a number and age is 1, you'll only get a single error (the name one). Making error accumulation work in cases like that would be considerably more complex.
Relatedly, you'll also see a deprecation warning about the flatMap on Validation, which you can think of as a reminder that accumulation won't happen across the bind. You can tell the compiler that you understand by importing scalaz.Validation.FlatMap._.