I'm writing a simple scala application that opens a flat file of json data, parses it and finally prints it out to the screen.
The next step will require that I stop at each object and add another item (string) to the front of it. My question is how can I add a new string per object in this list?
The following is my JSON implementation (credit goes to the init author here)
import scala.util.parsing.combinator._
class JSON extends JavaTokenParsers {
def obj: Parser[Map[String, Any]] =
"{"~> repsep(member, ",") <~"}" ^^ (Map() ++ _)
def arr: Parser[List[Any]] =
"["~> repsep(value, ",") <~"]"
def member: Parser[(String, Any)] =
stringLiteral~":"~value ^^
{ case name~":"~value => (name, value) }
def value: Parser[Any] = (
obj
| arr
| stringLiteral
| floatingPointNumber ^^ (_.toInt)
| "null" ^^ (x => null)
| "true" ^^ (x => true)
| "false" ^^ (x => false)
)
}
Next I call this w/ a flat file like so
import java.io.FileReader
import scala23.JSON
class JSONTest extends JSON {
def main(args: String) {
val reader = new FileReader(args)
println(parseAll(value, reader))
}
}
Then I get a valid println of the json contents. Instead I would like to pass this parse method a String and have it append it or create a new json object that has the string at the front of each object inside
Update
My current attempt looks something like the below
class JSONTest extends JSON {
def main(args: String) {
val reader = new FileReader(args)
val header = ("abc", "def")
// println(parseAll(value, reader).map(addHeader(_, header)))
println(parseAll(value, reader).map(addHeader(_.asInstanceOf[Map[String, Any]], header)))
}
def addHeader(xyz:Map[String, Any], header:(String, Any)):Map[String, Any] = {
xyz.map {
case (k, m:Map[String, Any]) => (k, addHeader(m))
case e => e
} + header
}
}
But I'm currently getting a few errors in Intellij
error: missing parameter type for expanded function ((x$1) => x$1.asInstanceOf[Map[String, Any]])
println(parseAll(value, reader).map(addHeader(_.asInstanceOf[Map[String, Any]], header)))
AND
error: not enough arguments for method addHeader: (xyz: Map[String,Any],header: (String, Any))Map[String,Any].
Unspecified value parameter header.
case (k, m:Map[String, Any]) => (k, addHeader(m))
Any help would be much appreciated (thank you in advance!)
Have you tried using map on the parser output instead.
Edit: this compiles on my machine
import java.io.FileReader
import scala23.JSON
class JSONTest extends JSON {
def main(args: String) {
val reader = new FileReader(args)
val header = ("abc", "def")
// println(parseAll(value, reader).map(addHeader(_, header)))
println(parseAll(value, reader).map(addHeader(_, header)))
}
def addHeader(xyz:Any, header:(String, Any)):Any = xyz match {
case obj:Map[String, Any] => obj.map {
case (k, m:Map[String, Any]) => (k, addHeader(m, header))
case e => e
} + header
case arr:List[Any] => arr.map(addHeader(_, header))
case e => e
}
}
It should be handling the varied output of the parse better.
Related
I've got a List which holds some Personalization-objects. The latter is defined like this:
sealed case class Personalization(firstname: String, lastname: String, keycardId: String)
I need to map this list to a Json-Array structure which has to look like this:
"personalization": [
{
"id": "firstname",
"value": "John"
},
{
"id": "lastname",
"value": "Doe"
}...
I am struggling with the part of mapping the field information to id/value pairs. Normally, I would create a play.api.libs.json.Format out of the Personalization class and let it map automatically -> Json.format[Personalization] - but this time, I need to create an array where an entry can hold n attributes.
Therefore I am asking for advice, if there is a possibility to use the Scala Play-Framework?
Any input is much appreciated, thank you!
Writing as such JSON representation is not quite complex, using Writes.transform.
import play.api.libs.json._
case class Personalization(firstname: String, lastname: String, keycardId: String) // No need to seal a case class
implicit def writes: Writes[Personalization] = {
val tx: JsValue => JsValue = {
case JsObject(fields) => Json.toJson(fields.map {
case (k, v) => Json.obj("id" -> k, "value" -> v)
})
case jsUnexpected => jsUnexpected // doesn't happen with OWrites
}
Json.writes[Personalization].transform(tx)
}
Which can be tested as bellow.
val personalization = Personalization(
firstname = "First",
lastname = "Last",
keycardId = "Foo")
val jsonRepr = Json.toJson(personalization)
// => [{"id":"firstname","value":"First"},{"id":"lastname","value":"Last"},{"id":"keycardId","value":"Foo"}]
Reading is a little bit tricky:
implicit def reads: Reads[Personalization] = {
type Field = (String, Json.JsValueWrapper)
val fieldReads = Reads.seq(Reads[Field] { js =>
for {
id <- (js \ "id").validate[String]
v <- (js \ "value").validate[JsValue]
} yield id -> v
})
val underlying = Json.reads[Personalization]
Reads[Personalization] { js =>
js.validate(fieldReads).flatMap { fields =>
Json.obj(fields: _*).validate(underlying)
}
}
}
Which can be tested as bellow.
Json.parse("""[
{"id":"firstname","value":"First"},
{"id":"lastname","value":"Last"},
{"id":"keycardId","value":"Foo"}
]""").validate[Personalization]
// => JsSuccess(Personalization(First,Last,Foo),)
Note that is approach can be used for any case class format.
Probably it is possible to do it a more elegant way I did, but you can use the following snippet:
case class Field(id: String, value: String)
object Field {
implicit val fieldFormatter: Format[Field] = Json.format[Field]
}
sealed case class Personalization(firstname: String, lastname: String, keycardId: String)
object Personalization {
implicit val personalizationFormatter: Format[Personalization] = new Format[Personalization] {
override def reads(json: JsValue): JsResult[Personalization] =
Try {
val data = (json \ "personalization").as[JsValue]
data match {
case JsArray(value) =>
val fields = value.map(_.as[Field]).map(f => f.id -> f.value).toMap
val firstname = fields.getOrElse("firstname", throw new IllegalArgumentException("Mandatory field firstname is absent."))
val lastname = fields.getOrElse("lastname", throw new IllegalArgumentException("Mandatory field lastname is absent."))
val keycardId = fields.getOrElse("keycardId", throw new IllegalArgumentException("Mandatory field keycardId is absent."))
Personalization(firstname, lastname, keycardId)
case _ => throw new IllegalArgumentException("Incorrect json format for Personalization.")
}
}.toEither.fold(e => JsError(e.getMessage), JsSuccess(_))
override def writes(o: Personalization): JsValue = {
val fields = List(Field("firstname", o.firstname), Field("lastname", o.lastname), Field("keycardId", o.keycardId))
JsObject(List("personalization" -> Json.toJson(fields)))
}
}
}
It converts {"personalization":[{"id":"firstname","value":"John"},{"id":"lastname","value":"Doe"},{"id":"keycardId","value":"1234"}]} to Personalization(John,Doe,1234) and vice versa
I have a Scala Map keyed by a type that itself needs serialising to JSON. Because of the nature of JSON that requires key names for objects to be strings, a simple mapping is not directly possible.
The work around I wish to implement is to convert the Map to a Set before serialising to JSON, and then from a Set back to a Map after deserialising.
I am aware of other methods using key serialisers on specific types, e.g. Serialising Map with Jackson, however, I require a solution that applies to arbitrary key types and in this regard, conversion to Set and back again looks to me like the best option.
I've had some success serialising to a Set with a wrapper object by modifying MapSerializerModule.scala from jackson-module-scala below, but I'm not familiar enough with the Jackson internals to get that JSON to deserialise back into the Map I started with.
I should add that I control both the serialisation and deserialisation side, so what the JSON looks like is not significant.
case class Wrapper[K, V](
value: Set[(K, V)]
)
class MapConverter[K, V](inputType: JavaType, config: SerializationConfig)
extends StdConverter[Map[K, V], Wrapper[K, V]] {
def convert(value: Map[K, V]): Wrapper[K, V] = {
val set = value.toSet
Wrapper(set)
}
override def getInputType(factory: TypeFactory) = inputType
override def getOutputType(factory: TypeFactory) =
factory.constructReferenceType(classOf[Wrapper[_, _]], inputType.getContentType)
.withTypeHandler(inputType.getTypeHandler)
.withValueHandler(inputType.getValueHandler)
}
object MapSerializerResolver extends Serializers.Base {
val MAP = classOf[Map[_, _]]
override def findMapLikeSerializer(
config: SerializationConfig,
typ: MapLikeType,
beanDesc: BeanDescription,
keySerializer: JsonSerializer[AnyRef],
elementTypeSerializer: TypeSerializer,
elementValueSerializer: JsonSerializer[AnyRef]): JsonSerializer[_] = {
val rawClass = typ.getRawClass
if (!MAP.isAssignableFrom(rawClass)) null
else new StdDelegatingSerializer(new MapConverter(typ, config))
}
}
object Main {
def main(args: Array[String]): Unit = {
val objMap = Map(
new Key("k1", "k2") -> "k1k2",
new Key("k2", "k3") -> "k2k3")
val om = new ObjectMapper()
om.registerModule(DefaultScalaModule)
om.registerModule(ConverterModule)
val res = om.writeValueAsString(objMap)
println(res)
}
}
I managed to find the solution:
case class MapWrapper[K, V](
wrappedMap: Set[MapEntry[K, V]]
)
case class MapEntry[K, V](
key: K,
value: V
)
object MapConverter extends SimpleModule {
addSerializer(classOf[Map[_, _]], new StdDelegatingSerializer(new StdConverter[Map[_, _], MapWrapper[_, _]] {
def convert(inMap: Map[_, _]): MapWrapper[_, _] = MapWrapper(inMap map { case (k, v) => MapEntry(k, v) } toSet)
}))
addDeserializer(classOf[Map[_, _]], new StdDelegatingDeserializer(new StdConverter[MapWrapper[_, _], Map[_, _]] {
def convert(mapWrapper: MapWrapper[_, _]): Map[_, _] = mapWrapper.wrappedMap map { case MapEntry(k, v) => (k, v) } toMap
}))
}
class MapKey(
val k1: String,
val k2: String
) {
override def toString: String = s"MapKey($k1, $k2) (str)"
}
object Main {
def main(args: Array[String]): Unit = {
val objMap = Map(
new MapKey("k1", "k2") -> "k1k2",
new MapKey("k2", "k3") -> "k2k3")
val om = setupObjectMapper
val jsonMap = om.writeValueAsString(objMap)
val deserMap = om.readValue(jsonMap, classOf[Map[_, _]])
}
private def setupObjectMapper = {
val typeResolverBuilder =
new DefaultTypeResolverBuilder(ObjectMapper.DefaultTyping.NON_FINAL) {
init(JsonTypeInfo.Id.CLASS, null)
inclusion(JsonTypeInfo.As.WRAPPER_OBJECT)
typeProperty("#CLASS")
override def useForType(t: JavaType): Boolean = !t.isContainerType && super.useForType(t)
}
val om = new ObjectMapper()
om.registerModule(DefaultScalaModule)
om.registerModule(MapConverter)
om.setDefaultTyping(typeResolverBuilder)
om
}
}
Interestingly, if the key type is a case class, the MapConverter is not necessary since the case class can be reconstituted from the string representation.
In my case, I had a nested Map. This required a small addition to the conversion back into a map:
addDeserializer(classOf[Map[_, _]], new StdDelegatingDeserializer(new StdConverter[MapWrapper[_, _], Map[_, _]] {
def convert(mapWrapper: MapWrapper[_, _]): Map[_, _] = {
mapWrapper.wrappedMap.map { case MapEntry(k, v) => {
v match {
case wm: MapWrapper[_, _] => (k, convert(wm))
case _ => (k, v)
}
}}.toMap
}
}))
I have a case class that only is a wrapper of a collection like this:
case class MyClass(list: List[String])
If I now try do deserialize some arbitrary json into this case class it doesn't fail if the list field is missing.
Is it possible to force it to fail during extraction?
Code example:
import org.json4s.jackson.JsonMethods.parse
implicit val formats = org.json4s.DefaultFormats
val json = parse("""{ "name": "joe" }""")
case class MyClass(list: List[String])
val myClass = json.extract[MyClass] // Works!
assert(myClass.list.isEmpty)
case class MyClass2(test: String)
val myClass2 = json.extract[MyClass2] // Fails!
I need it to fail for the missing list field as it does for the string field.
Please help. Thx!
Solution:
You can create a MyClass deserializer to deserialize MyClass with the List field, like:
class MyClassSerializer extends Serializer[MyClass] {
private val MyClassClass = classOf[MyClass]
override def deserialize(implicit format: Formats): PartialFunction[(TypeInfo, JValue), MyClass] = {
case (TypeInfo(MyClassClass, _), json) => json match {
case JObject(JField("list", JArray(l)) :: _) =>
MyClass(l.map(i => i.extract[String]))
case x => throw new MappingException("Can't convert " + x + " to MyClass")
}
}
override def serialize(implicit format: Formats): PartialFunction[Any, JValue] = ???
}
The cause missing List field can't parse with Exception is in Extraction.scala:
private class CollectionBuilder(json: JValue, tpe: ScalaType)(implicit formats: Formats) {
private[this] val typeArg = tpe.typeArgs.head
private[this] def mkCollection(constructor: Array[_] => Any) = {
val array: Array[_] = json match {
case JArray(arr) => arr.map(extract(_, typeArg)).toArray
case JNothing | JNull => Array[AnyRef]()
case x => fail("Expected collection but got " + x + " for root " + json + " and mapping " + tpe)
}
constructor(array)
}
we can see the above code snippet, when we meet a collection in the case class, Json4S will retrieve the fields in the Json AST by the constructor parameter name(list), if it can't find the parameter name(list), it will return JNothing, but for JNothing in the above code snippet, it will create a new collection(case JNothing | JNull => Array[AnyRef]()) without throwing errors.
I am using Scala with Play. I have a JSON file with all the countries in the world and their respective cities. The JSON looks like this:
{
"CountryA": ["City1","city2"],
"CountryB": ["City1"]
}
I parse it accordingly:
val source: String = Source.fromFile("app/assets/jsons/countriesToCities.json").getLines.mkString
val json: JsValue = Json.parse(source)
My ultimate goal is to convert the json contents into a Scala MultiMap where the key is a String - the country, and the value is a Set[String] - the cities.
Thanks in advance!
Here's a long-winded solution that won't throw if the JSON structure doesn't match what you're expecting:
val source: String =
"""
|{
| "CountryA": ["City1","city2"],
| "CountryB": ["City1"]
|}
""".stripMargin
val json: JsValue = Json.parse(source)
import scala.collection.breakOut
val map: Map[String, Set[String]] = json.asOpt[JsObject] match {
case Some(obj) =>
obj.fields.toMap.mapValues { v =>
v.asOpt[JsArray] match {
case Some(JsArray(cities)) => cities.flatMap(_.asOpt[String])(breakOut)
case _ => Set.empty[String]
}
}
case _ => Map.empty[String, Set[String]]
}
map must beEqualTo(Map("CountryA" -> Set("City1", "city2"), "CountryB" -> Set("City1")))
If you're confident about the structure of the JSON and don't mind using as (which could throw) instead of asOpt (which won't):
val map2: Map[String, Set[String]] = {
json.as[JsObject].fields.toMap.mapValues {
_.as[JsArray].value.map(_.as[String])(breakOut)
}
}
map2 must beEqualTo(Map("CountryA" -> Set("City1", "city2"), "CountryB" -> Set("City1")))
I am using anorm to query and save elements into my postgres database.
I have a json column which I want to read as class of my own.
So for example if I have the following class
case class Table(id: Long, name:String, myJsonColumn:Option[MyClass])
case class MyClass(site: Option[String], user:Option[String])
I am trying to write the following update:
DB.withConnection { implicit conn =>
val updated = SQL(
"""UPDATE employee
|SET name = {name}, my_json_column = {myClass}
|WHERE id = {id}
""".stripMargin)
.on(
'name -> name,
'myClass -> myClass,
'custom -> id
).executeUpdate()
}
}
I also defined a implicit convertor from json to my object
implicit def columnToSocialData: Column[MyClass] = anorm.Column.nonNull[MyClass] { (value, meta) =>
val MetaDataItem(qualified, nullable, clazz) = meta
value match {
case json: org.postgresql.util.PGobject => {
val result = Json.fromJson[MyClass](Json.parse(json.getValue))
result.fold(
errors => Left(TypeDoesNotMatch(s"Cannot convert $value: ${value.asInstanceOf[AnyRef].getClass} to Json for column $qualified")),
valid => Right(valid)
)
}
case _ => Left(TypeDoesNotMatch(s"Cannot convert $value: ${value.asInstanceOf[AnyRef].getClass} to Json for column $qualified"))
}
And the error I get is:
type mismatch;
found : (Symbol, Option[com.MyClass])
required: anorm.NamedParameter
'myClass -> myClass,
^
The solution is just to add the following:
implicit val socialDataToStatement = new ToStatement[MyClass] {
def set(s: PreparedStatement, i: Int, myClass: MyClass): Unit = {
val jsonObject = new org.postgresql.util.PGobject()
jsonObject.setType("json")
jsonObject.setValue(Json.stringify(Json.toJson(myClass)))
s.setObject(i, jsonObject)
}
}
and:
implicit object MyClassMetaData extends ParameterMetaData[MyClass] {
val sqlType = "OTHER"
val jdbcType = Types.OTHER
}