I have a large (>8 elements) object that contains a potentially empty list of strings ids.
I'd like to read the data in and parse in elm. Initially my attempts returned <internal structure> or undefined.
I found the problem to be not handling nulls. It took me a long time to diagnose and testing each of the elements is tedious. I don't know where to look for compiler or run time hints. How does one approach this in elm?
Are there established ways to quickly validate an Elm model against a javascript object? Are there good places to put Debug.* statements? Ways to test with the repl?
The Object
# curl '0.0.0.0:3003/person_search_view?select=ids'|jq -c
[{"ids":["11488"]},{"ids":["11489"]},{"ids":[null]}]
The code
-- testing what part is failing
-- *tedious* made a function to test reading in each type [List String, String, Int]
justID : (List String) -> Person
justID ids =
Person 0 "" "" "" "" "" "" 0 "" 0 0 0 ids [""] [""] ""
stringlist : Decode.Decoder (List String)
stringlist = list (oneOf [string, null "N/A"] )
memberDecoderID : Decode.Decoder Person
memberDecoderID =
Decode.succeed
justID
|: ("ids" := stringlist ) -- here was failing (List string)
fetchAllUrl = "http://0.0.0.0:3003/person_search_view?order=curage"
fetchAll : Effects Action
fetchAll =
Http.get (Decode.list memberDecoderID ) fetchAllUrl
|> Task.toResult
|> Task.map FetchAllDone
|> Effects.task
I would recommend building unit tests for your custom decoders. Not only that, but I would recommend starting with unit tests when building any decoders. If you start building unit tests around the smallest decoders possible using JSON values appropriate to what you'd see in the real world, you will probably catch errors much faster, and you'll be guaranteed to future-proof your code because each error you've caught along the way will have to be fixed via a backwards-compatible unit test.
For example, you could use the entertainingly-named deadfoxygrandpa/elm-test library and write tests like the following:
-- in your CustomDecoders.elm file...
stringlist : Decode.Decoder (List String)
stringlist = list (oneOf [string, null "N/A"] )
-- in your CustomDecoders-Tests.elm file...
tests : Test
tests =
suite "A Test Suite"
[ test "stringlist decodes string lists"
<| assertEqual (decodeString stringlist "[\"1\",\"2\",\"3\"]") (Ok ["1","2","3"])
, test "stringlist decodes null"
<| assertEqual (decodeString stringlist "[null]") (Ok ["N/A"])
]
You still may have to deal with cryptic JSON parsing error messages, but now you'll have a much easier time finding the problem since you'll know exactly which tests fail.
Related
had a look for something like this, but can't find the exact issue.
I have a JSON back from server side validation that looks like:
{
"field": ["field-name"],
"messages":["message","message"]
}
What I would like to do is decode it into an elm record like
{ field: String, messages: List String }
However, I'm having trouble with the err, field field. I'm having trouble turning a single element JSON array into just a string of that element.
Is it even possible with Decode, or am I better of Decoding it into a List and then just grabbing the head from the list.
This is what I have for the decode:
valErrorDecoder : Decode.Decoder ValError
valErrorDecoder =
decode ValError
|> required "field" (Decode.list Decode.string)
|> required "messages" (Decode.list Decode.string)
Thanks for any help!
Try Decode.index, that should do the trick.
valErrorDecoder : Decode.Decoder ValError
valErrorDecoder =
decode ValError
|> required "field" (Decode.index 0 Decode.string)
|> required "messages" (Decode.list Decode.string)
You mention in a comment that a colleague suggested Decode.map. In case you're curious, here is what that (more complex) solution might look like:
firstElementDecoder : Decode.Decoder a -> Decode.Decoder a
firstElementDecoder baseDecoder = Decode.list baseDecoder
|> Decode.map List.head
|> Decode.andThen (Maybe.map Decode.succeed >> Maybe.withDefault (Decode.fail "Empty list"))
What's happening here? We begin by decoding a list of strings, then map the List.head function onto that list, giving a Decoder (Maybe String). The function
Maybe.map Decode.succeed
>> Maybe.withDefault (Decode.fail "Empty list")
takes in a Maybe and turns it into a decoder, which either succeeds (with the value of the maybe) or fails (with an "Empty list" error message). We use this function as the argument to Decode.andThen, which:
Passes the Maybe from the list decoder into the function above, getting either a Decode.succeed or a Decode.fail
Runs the returned decoder, succeeding or failing with the appropriate value.
So, yes, Decode.index 0 is easier! But it may be of interest to see the longer solution, too :-)
I've been having a bit of difficulty wrapping my head around parsing JSON with Elm. I seem to have the basics down, but for some reason this small piece of JSON has left me stumped.
Here's the scenario: I am posting to a JSON API which will return with one of the two formats:
If the request is successful:
{
"status": "success",
"post": { ... }
}
If it fails:
{
"status": "error",
"message": "Some error message"
}
I'm trying to encode this JSON response into these data types:
type RequestStatus = Success | Error String
type alias CreatePostResponse =
{ status : RequestStatus
, post : Maybe Post }
So far I've had no luck. I've been looking through the JSON.Decode tutorial and this Thoughtbot article as guides, but neither seem to gotten me to the right place. Here's the code I have so far:
createPostResponse : Decoder CreatePostResponse
createPostResponse =
succeed CreatePostResponse
|: (("status" := string) `andThen` (("error" := string) `andThen` decodeStatus))
|: maybe ("post" := post)
decodeStatus : String -> String -> Decoder RequestStatus
decodeStatus status errorMessage =
succeed (case status of
"success" -> Success
"error" -> Error errorMessage
_ -> Error "unknown")
Obviously this produces all sorts of type errors and doesn't compile, but I haven't been able to come up with a good way to get the string from the "message" field into the RequestStatus type.
Does anyone have any ideas?
Here's a little working piece of code which should do what you are after. I've taken the liberty of assuming a basic shape for the Post type, for concreteness.
import Json.Decode exposing (..)
type alias Post = { title: String, body: String }
type RequestStatus = Success Post | Error String
post: Decoder Post
post = object2 Post ("title" := string) ("body" := string)
requestStatusData: String -> Decoder RequestStatus
requestStatusData status =
case status of
"success" -> object1 Success ("post" := post)
"error" -> object1 Error ("message" := string)
_ -> fail <| status ++ " is not a valid value for request status"
decodeStatus : Decoder RequestStatus
decodeStatus =
("status" := string) `andThen` requestStatusData
First, I turned the RequestStatus and CreatePostResponse types inside-out: instead of CreatePostResponse having a RequestStatus and a Maybe Post, which need to be kept in-sync with one another, the RequestStatus type itself models the fact that a success has a post, while an error does not. The new RequestStatus reads more like idiomatic Elm.
I then approached the decoding from the top-down:
To decode some JSON into a RequestStatus, we first deserialize the "status" property in the JSON object as a string, then we deserialize the rest based on what "status" was. The idiomatic way of doing this is with andThen (it looks like you knew this, but hadn't quite perfected the way it fit in with the rest). This translates to:
decodeStatus : Decoder RequestStatus
decodeStatus =
("status" := string) `andThen` requestStatusData
andThen decodes one record, then passes that record into a function which does the rest of the decoding, so requestStatusData needs to look like:
requestStatusData: String -> Decoder RequestStatus
requestStatusData is either a Success Post or a Error String. So we need two branches, then, associated with statuses "success" and "error" (plus a default to catch malformed statuses):
requestStatusData status =
case status of
"success" -> makeSuccessWithPost
"error" -> makeErrorWithString
_ -> fail <| status ++ " is not a valid value for request status"
We fill out the makeSuccessWithPost and makeErrorWithString implementations. The objectN functions in Json.Decode provide a facility to decode components, then feed them to a constructor (or other function):
object1 Success ("post" := post)
first decodes the "post" property using the decoder defined as post, then invokes the first argument of object1 (Success) on the result. Likewise,
object2 Post ("title" := string) ("body" := string)
decodes "title" using the string decoder, then decodes "body" using the string decoder, then invokes the Post function with the two decoded strings as its arguments. So we end up with:
requestStatusData status =
case status of
"success" -> object1 Success ("post" := post)
"error" -> object1 Error ("message" := string)
_ -> fail <| status ++ " is not a valid value for request status"
The last step is filling out the post decoder, which as mentioned above is a standard application of object2.
Overall, I think I've gone about this in fairly standard, idiomatic Elm style, but I'm quite new and I might have made some gaffes. It does work though! One last note: I would argue that the RequestStatus type is not actually necessary; the Result type in the Elm core library captures the idea of a type with a success and failure mode. You could use
Result String Post
instead of RequestStatus without losing any functionality (you'd need to change requestStatusData slightly but I'll leave that as an exercise for the reader...).
I have reached a point where my code compiles successfully, yet I have doubts about my solution and am posting this question for that reason.
I have a Node class defined as:
case class Node(id: Long, label: String, parent_id: Option[Long])
The reason I quote/unquote recursive is because technically, I do not store a Node within a Node. Rather, each node has a pointer to its parent and I can say: give me all children of Node id=X.
Here is an example tree, for the sake of visualization. I would like to give the root_node's ID, and obtain the tree's conversion to a Json String:
root_node
|_ node_1
| |_ node_11
| |_ node_111
|_ node_2
|_ node_3
The Json would look like:
{"title": "root_node", "children": [...]}
with the children array containing node_1, 2 and 3 etc... recursively
Here is the Writes Converter for Node:
/** json converter of Node to JSON */
implicit val NodeWrites = new Writes[Node] {
def writes(node: Node) = Json.obj(
"title" -> node.label,
"children" -> Node.getChildrenOf(node.id)
)
}
Quoting Play docs:
The Play JSON API provides implicit Writes for most basic types, such
as Int, Double, String, and Boolean. It also supports Writes for
collections of any type T that a Writes[T] exists.
I need to point out that Node.getChildrenOf(node.id) returns a List of Nodes from the DB. So according to Play's docs, I should be able to convert a List[Node] to Json. It seems that doing this within the Writes converter itself is a bit more troublesome.
Here is the resulting error from running this code:
type mismatch;
found : List[models.Node]
required: play.api.libs.json.Json.JsValueWrapper
Note: implicit value NodeWrites is not applicable here because it comes after the application point and it lacks an explicit result type
I added the "explicit result type" to my Writes converter, here is the result:
/** json converter of Node to JSON */
implicit val NodeWrites: Writes[Node] = new Writes[Node] {
def writes(node: Node) = Json.obj(
"title" -> node.label,
"children" -> Node.getChildrenOf(node.id)
)
}
The code now executes properly, I can visualize the tree on the browser.
Even though this looks to me like the cleanest working solution, IntelliJ still complains about the line:
"children" -> Node.getChildrenOf(node.id)
saying:
Type mismatch: found(String, List[Node]), required (String, Json.JsValueWrapper)
Could it be that IntelliJ's error reporting is not based off of the Scala compiler?
Finally, is the overall approach of the JSON converter terrible?
Thanks and sorry for the long post.
The problem lies in "children" -> Node.getChildrenOf(node.id). Node.getChildrenOf(node.id) returns a List[Node]. Whereas any attribute in the Json.obj expects JsValueWrappers. In this case a JsArray.
Something like this should work:
implicit val writes = new Writes[Node] {
def writes(node: Node) = Json.obj(
"title" -> node.label,
// Note that we have to pass this since the writes hasn't been defined just yet.
"children" -> JsArray(Node.getChildrenOf(node).map(child => Json.toJson(child)(this)))
)
}
This at least compiles, I haven't tested it with any data though.
I have a huge json object and I need to parse it and then write some tests to see if everything goes as expected.
case class User(id: Identification, age:Int, name: String ...)
case class Identification(id: Int, hash: String ....)
... a lot more classes
Now I'm trying to write the tests
val json = parse(Source.fromFile(/path).getLines.mkString("\n"))
import org.json4s.DefaultFormats
implicit val formats = DefaultFormats
So my question is how can i test if the case classes are ok?
I thought maybe I should try to extract for ex. the users and then to check parameter by parameter if they are correct, but I don't thing that is a good way because it is not me who created the json so I'm not interested about the content.
Thanks
This is what I found working with JSON and case classes overt the years the minimum to test.
This three things should be tested always
Serialization with deserialiaztion combined
val example = MyCaseClass()
read[MyCaseClass](write(example)) should Equal example
Checks if a class can be transformed to JSON, read back and still has the same values. This one breaks more often than one would think.
Deserialization: JSON String -> CaseClasses
val exampleAsJSON : String
val exampleAsCaseClass : MyCaseClass
read(exampleAsJSON) shouldEqual exampleAsCaseClass
Checks if JSON still can be deserialized.
Serialization: CaseClasses -> JSON String
val exampleAsJSON : String
val exampleAsCaseClass : MyCaseClass
write(exampleAsCaseClass) shouldEqual exampleAsJSON
Checks if String/ JSON Representation stays stable. Here it is hard to keep the data up to date and often some not nice whitespace changes lead to false alarms.
Additional things to test
Are there optional parameters present? If yes all tests should be done with and without the optional parameters.
I have a complex JSON that it's persisted in a database. It's complexity is "segregated" in "blocks", as follows:
Entire JSON:
{
"block1" : {
"param1" : "val1",
"param2" : "val2"
},
"block2" : {
"param3" : "val3",
"param4" : "val4"
},
...
}
In the database, every block is stored and treated individually:
Persisted Blocks
"block1" : {
"param1" : "val1",
"param2" : "val2"
}
"block2" : {
"param3" : "val3",
"param4" : "val4"
}
Every block has a business meaning, so, every one it's mapped to a case-class.
I'm building a Play API that stores, updates and retrieves this JSON Structure and i want to validate if someone altered it's data for the sake of integrity.
I'm doing the retrieval (parsing and validation) of every block as follows:
val block1 = Json.parse(block1).validate[Block1].get
val block2 = Json.parse(block2).validate[Block2].get
...
The case-classes:
trait Block
sealed case class Block1 (param1: String, param2: String, ...) extends Block
sealed case class Block2 (param3: String, param4: String, ...) extends Block
sealed case class Request (block1: Block1, block2: Block2, ...)
With the current structure, if some field is altered and doesn't match with the defined type for it Play throws this exception:
[NoSuchElementException: JsError.get]
So, i want to build an accumulative error structure with Scalaz and Validation that catch all possible parsing and validation errors. I have seen this and this so i've coded the validation this way:
def build(block1: String, block2: String, ...): Validation[NonEmptyList[String], Request] = {
val block1 = Option(Json.parse(block1).validate[Block1].get).toSuccess("Error").toValidationNel
val block2 = Option(Json.parse(block2).validate[Block2].get).toSuccess("Error").toValidationNel
...
val request = (Request.apply _).curried
blockn <*> (... <*> (... <*> (...<*> (block2 <*> (block1 map request)))))
}
Note that i'm using the applicative functor <*> because Request has 20 fields (building that is a parentheses-mess with that syntax), |#| applicative functor only applies for case classes up to 12 parameters.
That code works for the happy path, but, when i modify some field Play throws the Execution Exception later described.
Question: I want to accumulate all possible structure faults that Play can detect while parsing every block. How can i do that?
Note: If in some way Shapeless has something to do with this i'm open to use it (i'm already using it).
If multi-validation is the only reason for adding Scalaz to your project then why not consider an alternative which will not require you adapt your Play project to the monadic way in which Scalaz forces you to solve problems (which might be a good thing, but not necessarily if the only reason for that is multi-validation).
Instead consider using a standard Scala approach with scala.util.Try:
val block1 = Try(Json.parse(block1).validate[Block1].get)
val block2 = Try(Json.parse(block2).validate[Block2].get)
...
val allBlocks : List[Try[Block]] = List(block1, block2, ...)
val failures : List[Failure[Block]] = allBlocks.collect { case f : Failure[Block] => f }
This way you can still operate on standard scala collections to retrieve the list of failures to be processed further. Also this approach does not constrain you in terms of number of blocks.