Modify json values from list of maps and save output - json

I have the following example json that I read in using the play framework.
{
"field_a": "dummy",
"field_b": "dummy",
"nest": {
"nest_a": "dummy",
"nest_b": 87
},
"field_c": null,
"field_d": null,
"field_e": "chocolate",
"field_f": "sugar",
"array": [
"dummy entry"
],
"id": "Anything"
}
I Then have the following List of Maps that I want to swap out data with which is my Input:
val substitutionsList: List[mutable.Map[String, String]] = List(
mutable.Map("field_b" -> "dummy string", "field_d" -> "2016-01-01", "field_f" -> "2011-01-01"),
mutable.Map("field_b" -> "dummy string", "field_d" -> "2018-01-01", "field_f" -> "2018-01-01"),
mutable.Map("field_b" -> "dummy string", "field_d" -> "2018-04-01", "field_f" -> "2018-04-01"),
mutable.Map("field_b" -> "dummy string", "field_d" -> "2016-01-01", "field_f" -> "2016-01-01")
)
I am reading in the json as follows:
def parseSchemaJson(schemaContent: String) = Json.parse(schemaContent).as[JsObject]
val baseSchemaInput = parseSchemaJson(Source.fromFile("/dummy.json").mkString)
I want to iterate over my Input and swap out the values in the json for the values in my map and after each one is done, create a new .json file.
private def replaceField(json: JsObject, fieldToReplace: String): Option[String] = (json \ fieldToReplace).asOpt[String]
println(replaceField(baseSchemaInput, "field_a")) //prints dummy
I can list out the value in my json using something like this but I have no idea how to swap the value from my list into each respective bit and write out a json file.
First occurance of the expected output
{
"field_a": "dummy",
"field_b": "dummy string",
"nest": {
"nest_a": "dummy",
"nest_b": 87
},
"field_c": null,
"field_d": "2016-01-01",
"field_e": "chocolate",
"field_f": "2011-01-01",
"array": [
"dummy entry"
],
"id": "Anything"
}

Given the following substitution list:
val substitutionsList: List[mutable.Map[String, Any]] = List(
mutable.Map("field_b" -> "dummy string 1", "field_d" -> "2016-01-01", "field_f" -> "2011-01-01"),
mutable.Map("field_b" -> "dummy string 2", "nest" -> mutable.Map("nest_b" -> "90"))
)
Note that the second one updates a nested value. You need to create a nested Map to define nested values.
You can define a function to transform a Map[String, Any] (Any because we can have either a String or a Map as value) into a JsObject. This is a recursive function and will call itself in case the Value is a Map
def mapToJsObject(map: mutable.Map[String, Any]): JsObject =
JsObject(map.mapValues {
case v:String => JsString(v)
case v:mutable.Map[String, mutable.Map[String, Any]] => mapToJsObject(v.asInstanceOf[mutable.Map[String, Any]])
})
Then go through your list of substitutions and deep merge the JsObject made from each Map with the original JsObject using the deepMerge function defined on the JsObject class. It will merge the nested objects too. See API here
val substitutedJsObjects: List[JsObject] = substitutionsList
.map(mapToJsObject)
.map(baseSchemaInput.deepMerge)
This should give you a list of JsObjects, one per Map in your List
You can then write them to file. Here is an example to write one file per json string. Files will be named 0.json, 1.json, etc ..
def writeToFile(jsObject: JsObject, fileName: String): Unit = {
println("writing "+fileName)
val pw = new PrintWriter(new File(fileName))
pw.write(jsObject.toString())
pw.close()
}
substitutedJsObjects.zipWithIndex.foreach {
case (jsObject, index) => {
val fileName = index.toString + ".json"
writeToFile(jsObject, fileName)
}
}

Related

Scala parses non-canonical JSON

The following data can be seen with different value types. How can I get the desired output?
package ceshi
import scala.util.parsing.json.JSON
object ceshi1212 {
def main(args: Array[String]): Unit = {
class CC[T] {
def unapply(a: Any): Option[T] = Some(a.asInstanceOf[T])
}
object M extends CC[Map[String, Any]]
object L extends CC[List[Any]]
object S extends CC[String]
val jsonString =
"""
{
"languages": [
{
"name": "English",
"is_active": "true",
"completeness": "asdf"
},
{
"name": "Latin",
"is_active": "asdf",
"completeness": "232"
}
,{
"name": "Latin",
"is_active": "0009"
}
,
"error"
]
}
""".stripMargin
// 不规则json error和并列的数组类型不同 怎么解析自动跳过?
val result = for {
Some(M(map)) <- List(JSON.parseFull(jsonString))
L(languages) = map("languages")
M(language) <- languages
S(name) = language("name")
S(active) = language("is_active")
S(completeness) = language.getOrElse("completeness","--")
} yield {
(name, active, completeness)
}
println(result)
//i want result is: List((English,true,asdf), (Latin,asdf,232),(Latain,0009,""))
}
}
i want get result is List((English,true,asdf), (Latin,asdf,232),(Latain,0009,""))
note: 1 The string is not always at the end of the array, and the position is indeterminate
2 The three keys I need may not be complete
As said in the comments there are other libraries to be recommended for working with json have a look at this post to get an overview: What JSON library to use in Scala?
Answer to your question with specific framework (play-json)
Personally I can recommend to use the play json framework.
To obtain the result you have described with play json, your code might look like this:
import play.api.libs.json._
val json: JsValue = Json.parse(jsonString)
val list = (json \ "languages").as[Seq[JsValue]]
val names = list.map(x => ((x\"name").validate[String] match {
case JsSuccess(v, p ) => v
case _ => ""
}
))
val isActives = list.map(x => ((x\"is_active").validate[String] match {
case JsSuccess(v, p ) => v
case _ => ""
}
))
val completeness = list.map(x => ((x\"completeness").validate[String] match {
case JsSuccess(v, p ) => v
case _ => ""
}
))
// need to know in advance what is your max length of your tuple (tmax)
// since 3rd value "completeness" can be missing, so we just take "" instead
val tmax = 3
val res = for(idx <-0 to tmax-1) yield (names(idx),isActives(idx),completeness(idx))
res.toList
// List[(String, String, String)] = List((English,true,asdf), (Latin,asdf,232), (Latin,0009,""))
There's also a very good documentation for the play json framework, just check it out yourself: https://www.playframework.com/documentation/2.8.x/ScalaJson
If you can switch parser library to circe, you can deal with this types of bad data.
Given you have data model
import io.circe.generic.semiauto._
import io.circe.parser.decode
import io.circe.{Decoder, Json}
case class Languages(languages: Seq[Language])
case class Language(name: String, is_active: String, completeness: Option[String])
You can define a fault-tolerant seq decoder that would skip bad data rather than crash whole parse
def tolerantSeqDecoder[A: Decoder]: Decoder[Seq[A]] = Decoder.decodeSeq(Decoder[A]
.either(Decoder[Json])).map(_.flatMap(_.left.toOption))
and the rest...
val jsonString = """
{
"languages": [
{
"name": "English",
"is_active": "true",
"completeness": "asdf"
},
{
"name": "Latin",
"is_active": "asdf",
"completeness": "232"
},
{
"name": "Latin",
"is_active": "0009"
},
"error"
]
}
"""
val languageDecoder = deriveDecoder[Language]
implicit val tolerantDecoder = tolerantSeqDecoder[Language](languageDecoder)
implicit val languagesDecoder = deriveDecoder[Languages]
val parsed = decode[Languages](jsonString)
println(parsed)
out:
Right(Languages(List(Language(English,true,Some(asdf)), Language(Latin,asdf,Some(232)), Language(Latin,0009,None))))
This approach was suggested by one of circe developers: How do I ignore decoding failures in a JSON array?

Elm: decoding json from http response and showing it

I'm kind of new to Elm and I find it very hard to decode a json from a http response.
The app I'm making is doing a call to gravatar and receives a profile.
I'd like to extract some fields from the response and put in in a record, which in turn in shown in the view.
This is my code:
-- MODEL
type alias MentorRecord =
{ displayName : String
, aboutMe : String
, currentLocation : String
, thumbnailUrl : String
}
type alias Model =
{ newMentorEmail : String
, newMentor : MentorRecord
, mentors : List MentorRecord
}
init : ( Model, Cmd Msg )
init =
( Model "" (MentorRecord "" "" "" "") [], Cmd.none )
-- UPDATE
type Msg
= MentorEmail String
| AddMentor
| GravatarMentor (Result Http.Error MentorRecord)
| RemoveMentor
update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
case msg of
MentorEmail newEmail ->
( { model | newMentorEmail = newEmail }, Cmd.none )
AddMentor ->
( model, getGravatarMentor model.newMentorEmail )
GravatarMentor (Ok addedMentor) ->
( Model "" addedMentor (addedMentor :: model.mentors)
, Cmd.none
)
GravatarMentor (Err _) ->
( model, Cmd.none )
RemoveMentor ->
( model, Cmd.none )
-- VIEW
view : Model -> Html Msg
view model =
div []
[ input [ placeholder "Email adress mentor", onInput MentorEmail ] []
, button [ onClick AddMentor ] [ text "Add Mentor" ]
, br [] []
, img [ src (createIconUrl model.newMentorEmail) ] []
, div [] [ text model.newMentor.displayName ]
, div [] [ toHtmlImgList model.mentors ]
]
toHtmlImgList : List MentorRecord -> Html Msg
toHtmlImgList mentors =
ul [] (List.map toLiImg mentors)
toLiImg : MentorRecord -> Html Msg
toLiImg mentor =
li [] [ img [ src mentor.thumbnailUrl ] [] ]
-- HTTP
getGravatarMentor : String -> Cmd Msg
getGravatarMentor newMentorEmail =
Http.send GravatarMentor
(Http.get (createProfileUrl newMentorEmail) decodeGravatarResponse)
createProfileUrl : String -> String
createProfileUrl email =
"https://en.gravatar.com/" ++ MD5.hex email ++ ".json"
createIconUrl : String -> String
createIconUrl email =
"https://www.gravatar.com/avatar/" ++ MD5.hex email
decodeGravatarResponse : Decoder MentorRecord
decodeGravatarResponse =
let
mentorDecoder =
Json.Decode.Pipeline.decode MentorRecord
|> Json.Decode.Pipeline.required "displayName" string
|> Json.Decode.Pipeline.required "aboutMe" string
|> Json.Decode.Pipeline.required "currentLocation" string
|> Json.Decode.Pipeline.required "thumbnailUrl" string
in
at [ "entry" ] mentorDecoder
If a valid email address if filled in (i.e. one with a gravatar profile), you see the icon. But what this code also should do is extract name, location, about me info, thumbnailUrl from another http response, put it in a list, and show it in the view. And that's not happening if you click on 'Add mentor'
So I guess the decoding part isn't going very well, but I'm not sure (maybe because the nested element is in a list?).
A response from gravatar looks like this (removed some fields in entry):
{ "entry": [
{
"preferredUsername": "bla",
"thumbnailUrl": "https://secure.gravatar.com/avatar/hashinghere",
"displayName": "anne",
"aboutMe": "Something...",
"currentLocation": "Somewhere",
}
]}
Code in Ellie app: https://ellie-app.com/n5dxHhvQPa1/1
entry is an array. To decode the contents of the first element of the array, you need to use Json.Decode.index.
Change:
(at [ "entry" ]) mentorDecoder
to
(at [ "entry" ] << index 0) mentorDecoder
But the bigger problem here is that Gravatar does not support cross origin requests (CORS) but only JSONP. elm-http doesn't support JSONP. You can either use ports for that or use a third party service which enables you to make CORS requests to arbitrary sites. I've used the latter in the ellie link below but you should use ports or your own CORS proxy in a real production application.
I also made aboutMe and currentLocation optional as they weren't present in the profile I checked. Here's the link: https://ellie-app.com/pS2WKpJrFa1/0
The original functions:
createProfileUrl : String -> String
createProfileUrl email =
"https://en.gravatar.com/" ++ MD5.hex email ++ ".json"
decodeGravatarResponse : Decoder MentorRecord
decodeGravatarResponse =
let
mentorDecoder =
Json.Decode.Pipeline.decode MentorRecord
|> Json.Decode.Pipeline.required "displayName" string
|> Json.Decode.Pipeline.required "aboutMe" string
|> Json.Decode.Pipeline.required "currentLocation" string
|> Json.Decode.Pipeline.required "thumbnailUrl" string
in
at [ "entry" ] mentorDecoder
The changed functions:
createProfileUrl : String -> String
createProfileUrl email =
"https://crossorigin.me/https://en.gravatar.com/" ++ MD5.hex email ++ ".json"
decodeGravatarResponse : Decoder MentorRecord
decodeGravatarResponse =
let
mentorDecoder =
Json.Decode.Pipeline.decode MentorRecord
|> Json.Decode.Pipeline.required "displayName" string
|> Json.Decode.Pipeline.optional "aboutMe" string ""
|> Json.Decode.Pipeline.optional "currentLocation" string ""
|> Json.Decode.Pipeline.required "thumbnailUrl" string
in
(at [ "entry" ] << index 0) mentorDecoder

Parse a JSON in Scala and create variables for each key

I am new to scala and I am trying to parse a JSON shown below
val result = JSON.parseFull("""
{"name": "Naoki", "lang": ["Java", "Scala"] , "positionandvalue": ["5:21", "6:24", "7:6"]}
""")
result: Option[Any] = Some(Map(name -> Naoki, lang -> List(Java, Scala), positionandvalue -> List(5:21, 6:24, 7:6)))
And get the parsed values in a Map
val myMap = result match {
case Some(e) => e
case None => None
}
myMap: Any = Map(name -> Naoki, lang -> List(Java, Scala), positionandvalue -> List(5:21, 6:24, 7:6))
What I need is
1. To get the key as a new variable (to be used as metadata to validate the file) with its corresponding value assigned to it. Something like,
val name = "Naoki"
positionandvalue -> List(5:21, 6:24, 7:6). This variable indicates the List of(Position of string delimited a in file:length of string in position). How can I use this variable to satisfy the requirement.
you cannot dynamically create the variables name and positionandvalue from the Map key. However they can be statically created using the below approach.
val result: Option[Any] = Some(Map("name" -> "Naoki", "lang" -> List("Java", "Scala"), "positionandvalue" -> List("5:21", "6:24", "7:6")))
val myMap: Map[String, Any] = result match {
case Some(e: Map[String, Any] #unchecked) => e
case _ => Map()
}
val name = myMap.get("name") match {
case Some(x: String) => x
case _ => throw new RuntimeException("failure retrieving name key")
}
val positionandvalue = myMap.get("positionandvalue") match {
case Some(x: List[String] #unchecked) => x.map(y => (y.split(":") match {case Array(x1,x2) => x1 -> x2})).toMap
case _ => throw new RuntimeException("failure retrieving positionandvalue key")
}
positionandvalue: scala.collection.immutable.Map[String,String] = Map(5 -> 21, 6 -> 24, 7 -> 6)

How do I avoid nested Array when serializing a Map with Play Writes?

I am trying to serialize a map using the Json library from Play. I wrote my own Writes since there is none for Maps.
import play.api.libs.json.Json._
import play.api.libs.json._
object NestedArray extends App {
val m: Map[Int, String] = Map(1 -> "one", 2 -> "two")
implicit val mWrites = new Writes[Map[Int, String]] {
def writes(m: Map[Int, String]) = arr(
m.keys.map(k => {
obj("key" -> k.toString,
"value" -> m(k)
)
})
)
}
val j = toJson[Map[Int, String]](m)
println(prettyPrint(j))
}
The output is this:
[ [ {
"key" : "1",
"value" : "one"
}, {
"key" : "2",
"value" : "two"
} ] ]
As you can see there are two pairs of [ ] around the items. When I use a Wrapper class around the map I only get one pair of [ ].
case class Wrap(m: Map[Int, String])
val w = new Wrap(m)
implicit val wrapWrites = new Writes[Wrap] {
def writes(w: Wrap) = obj(
"m" -> w.m.keys.map(k => {
obj("key" -> k.toString,
"value" -> w.m(k)
)
})
)
}
val j2 = toJson[Wrap](w)
println(prettyPrint(j2))
Output:
{
"m" : [ {
"key" : "1",
"value" : "one"
}, {
"key" : "2",
"value" : "two"
} ]
}
Is there a way to achieve that without a wrapper class?
Json.arr makes a JSON array from it's argument list. Since the first argument is itself a sequence, the result is a sequence of a sequence.
E.g.
scala> Json.arr(1,2,3)
res1: play.api.libs.json.JsArray = [1,2,3]
scala> Json.arr(List(1,2,3))
res2: play.api.libs.json.JsArray = [[1,2,3]]
Removing the call to arr and converting the Iterable directly to JSON using toJson removes the nested array
import play.api.libs.json.Json._
import play.api.libs.json._
object NestedArray extends App {
val m: Map[Int, String] = Map(1 -> "one", 2 -> "two")
implicit val mWrites = new Writes[Map[Int, String]] {
def writes(m: Map[Int, String]): JsValue =
Json.toJson(m.keys.map(k => {
obj("key" -> k.toString,
"value" -> m(k)
)
}))
}
val j = toJson[Map[Int, String]](m)
println(prettyPrint(j))
}
Play does provide a Writes[Map[String, _]], which you can probably adapt for your uses. It seems unnecessary to create structure to represent a sequence of key-value pairs, when that's what a JSON object is already. Please see my answer to a similar question: How to serialize a Map[CustomType, String] to JSON

Play Framework: How to replace all the occurrence of a value in a JSON tree

Given the following JSON...
{ "id":"1234",
"name" -> "joe",
"tokens: [{
"id":"1234",
"id":"2345"
}]
}
... I need to replace the value of all the ids by xxxx like this:
{ "id":"xxxx",
"name" -> "joe",
"tokens: [{
"id":"xxxx",
"id":"xxxx"
}]
}
Let's start create the JSON tree:
val json = Json.obj(
"id" -> "1234",
"name" -> "joe",
"tokens" -> Json.arr(
Json.obj("id" -> "1234"),
Json.obj("id" -> "2345")
)
)
json: play.api.libs.json.JsObject = {"id":"1234","name":"joe","tokens":[{"id":"1234"},{"id":"2345"}]}
Then, getting all the ids is very simple:
json \\ "id"
res64: Seq[play.api.libs.json.JsValue] = List("1234", "1234", "2345")
Now, how do I replace the value of all the ids by xxxx?
There doesn't appear to be a nice way to do this with the standard Play JSON library, although I'd be happy to be proved wrong in that regard. You can however do it easily using the play-json-zipper extensions:
import play.api.libs.json._
import play.api.libs.json.extensions._
val json = Json.obj(
"id" -> "1234",
"name" -> "joe",
"tokens" -> Json.arr(
Json.obj("id" -> "1234"),
Json.obj("id" -> "2345")
)
)
// Using `updateAll` we pattern match on a path (ignoring
// the existing value, as long as it's a string) and replace it
val transformed = json.updateAll {
case (__ \ "id", JsString(_)) => JsString("xxxx")
}
// play.api.libs.json.JsValue = {"id":"xxxx","name":"joe","tokens":[{"id":"xxxx"},{"id":"xxxx"}]}
To make that a re-usable function:
def replaceValue(json: JsValue, key: String, replacement: String) = json.updateAll {
case (__ \ path, JsString(_)) if path == key => JsString(replacement)
}
The json-zipper extensions are still "experimental", but if you want to add them to your project add the following to your project/Build.scala appDependencies:
"play-json-zipper" %% "play-json-zipper" % "1.0"
and the following resolver:
"Mandubian repository releases" at "https://github.com/mandubian/mandubian-mvn/raw/master/releases/"
Probably it isn't most efficient way to do it, but you can try to convert your JSON to an object copy it with new fields and then convert it back to json. Unfortunately currently I don't have environment to check the code, but it should be something like this:
case class MyId(id: String)
case class MyObject(id: String, name: String, tokens: List[MyId])
implicit val idFormat = Json.format[MyId]
implicit val objectFormat = Json.format[MyObject]
val json = Json.parse(jsonString)
val jsResult = Json.fromJson[MyObject](json)
val obj = jsResult match {
case JsSuccess(s, _) => s
case _ => throw new IllegalStateException("Unexpected")
}
val newObj = obj.copy(id = "xxxx")
val result = Json.toJson(newObj)