Parsing nested JSON into a list of Tuples with Aeson - json

Say I have the following structure:
data AddressDto = AddressDto
{ addressDtoId :: UUID
, addressDtoCode :: Text
, addressDtoCity :: Maybe Text
, addressDtoStreet :: Text
, addressDtoPostCode :: Text
} deriving (Eq, Show, Generic)
instance FromJSON AddressDto where
parseJSON = genericParseJSON $ apiOptions "addressDto"
instance ToJSON AddressDto where
toJSON = genericToJSON $ apiOptions "addressDto"
toEncoding = genericToEncoding $ apiOptions "addressDto"
This works as you would expect.
Now say I want to parse a JSON structure of the format:
{ UUID: AddressDto, UUID: AddressDto, UUID: AddressDto }
A reasonable Haskell representation would seem to be:
data AddressListDto = AddressListDto [(UUID, AddressDto)]
Creating a helper function like so:
keyAndValueToList :: Either ServiceError AddressListDto -> Text -> DiscountDto -> Either ServiceError AddressListDto
keyAndValueToList (Left err) _ _ = Left err
keyAndValueToList (Right (AddressListDto ald)) k v = do
let maybeUUID = fromString $ toS k
case maybeUUID of
Nothing -> Left $ ParseError $ InvalidDiscountId k
Just validUUID -> Right $ AddressListDto $ (validUUID, v) : ald
and finally an instance of FromJSON:
instance FromJSON AddressListDto where
parseJSON = withObject "tuple" $ \o -> do
let result = foldlWithKey' keyAndValueToList (Right $ AddressListDto []) o
case result of
Right res -> pure res
Left err -> throwError err
This fails to compile with:
Couldn't match type ‘aeson-1.4.5.0:Data.Aeson.Types.Internal.Value’
with ‘AddressDto’
Expected type: unordered-containers-0.2.10.0:Data.HashMap.Base.HashMap
Text AddressDto
Two questions:
1) How do I make sure the nested values in the hashmap get parsed correctly to an AddressDto.
2) How do I avoid forcing the initial value into an either? Is there a function I could use instead of foldlWithKey' that didn't make me wrap the initial value like this?

Funny answer. To implement parseJSON, you can use
parseJSON :: Value -> Parser (HashMap Text AddressDto)
...but even better, why not just use a type alias instead of a fresh data type, so:
type AddressListDto = HashMap UUID AddressDto
This already has a suitable FromJSON instance, so then you don't even need to write any code yourself.

Related

Functionally changing key names in serialization to aeson with Text keys

I have a json object with a manually crafted ToJSON instance. I would like to replace this with a function that does not require my explicit enumeration of the key names.
I am using "rec*" as a prefix I would like to strip, and my fields start out as Text rather than string.
Starting with minimal data:
data R3 = R3 { recCode :: Code
, recValue :: Value} deriving (Show, Generic)
And smart constructor function:
makeR3 rawcode rawval = R3 code value where
code = rawcode
value = rawval
This implementation works fine:
instance ToJSON R3 where
toJSON (R3 recCode recValue) = object [ "code" .= recCode, "value" .= recValue]
But as you can imagine, typing out every key name by hand from "code" to "recCode" is not something I want to do.
tmp_r3 = makeR3 "TD" "100.42"
as_json = encode tmp_r3
main = do
let out = encodeToLazyText tmp_r3
I.putStrLn out
I.writeFile "./so.json" out
return ()
Output is correct:
{"value":100.42,"code":"TD"}
-- not recValue and recCode, correct!
However, when I try this function, it becomes unable to convert the text to string as it had automatically before.
instance ToJSON R3 where
toJSON = genericToJSON defaultOptions {
fieldLabelModifier = T.toLower . IHaskellPrelude.drop 3 }
Output:
<interactive>:8:35: error:
• Couldn't match type ‘Text’ with ‘String’
Expected type: String -> String
Actual type: String -> Text
• In the ‘fieldLabelModifier’ field of a record
In the first argument of ‘genericToJSON’, namely ‘defaultOptions {fieldLabelModifier = toLower . IHaskellPrelude.drop 3}’
In the expression: genericToJSON defaultOptions {fieldLabelModifier = toLower . IHaskellPrelude.drop 3}
<interactive>:8:47: error:
• Couldn't match type ‘String’ with ‘Text’
Expected type: String -> Text
Actual type: String -> String
• In the second argument of ‘(.)’, namely ‘IHaskellPrelude.drop 3’
In the ‘fieldLabelModifier’ field of a record
In the first argument of ‘genericToJSON’, namely ‘defaultOptions {fieldLabelModifier = toLower . IHaskellPrelude.drop 3}’
The error itself is clear enough that Text doesn't work, but what should I change to strip my prefixes from keynames functionally in json output
and also correctly convert text to string?
I am also a little confused that I didn't change my input, it was Text type in both instances, but the first implementation was OK to work with it, while the second was not.
I am working in an ihaskell jupyter notebook.
Update
When I use the Data.Char recommended in answers below:
import Data.Char(toLower)
In:
instance ToJSON R3 where
toJSON = genericToJSON defaultOptions {
fieldLabelModifier = Data.Char.toLower . IHaskellPrelude.drop 3 }
I get:
<interactive>:8:35: error:
• Couldn't match type ‘Char’ with ‘String’
Expected type: String -> String
Actual type: String -> Char
• In the ‘fieldLabelModifier’ field of a record
In the first argument of ‘genericToJSON’, namely ‘defaultOptions {fieldLabelModifier = Data.Char.toLower . IHaskellPrelude.drop 3}’
In the expression: genericToJSON defaultOptions {fieldLabelModifier = Data.Char.toLower . IHaskellPrelude.drop 3}
<interactive>:8:55: error:
• Couldn't match type ‘String’ with ‘Char’
Expected type: String -> Char
Actual type: String -> String
• In the second argument of ‘(.)’, namely ‘IHaskellPrelude.drop 3’
In the ‘fieldLabelModifier’ field of a record
In the first argument of ‘genericToJSON’, namely ‘defaultOptions {fieldLabelModifier = Data.Char.toLower . IHaskellPrelude.drop 3}’
And when I try a naked "drop" rather than an IHaskellPrelude drop, I get:
instance ToJSON R3 where
toJSON = genericToJSON defaultOptions {
fieldLabelModifier = Data.Char.toLower . drop 3 }
<interactive>:8:55: error:
Ambiguous occurrence ‘drop’
It could refer to either ‘BS.drop’, imported from ‘Data.ByteString’
or ‘IHaskellPrelude.drop’, imported from ‘Prelude’ (and originally defined in ‘GHC.List’)
or ‘T.drop’, imported from ‘Data.Text’
You seem to be using toLower from Data.Text, which works with Text, not with String, so quite naturally, it doesn't fit there.
Instead, you could use toLower from Data.Char and map it over the String:
fieldLabelModifier = map toLower . drop 3
You compose two function T.toLower and drop 3, but the types do not match. Indeed, if we lookup the types, we see toLower :: Text -> Text and drop :: Int -> [a] -> [a]. A String is a list of Chars, but Text is not: a Text can be seen as a packed "block" of characters.
We can however compose a function of type String -> String, the type of the field fieldLabelModifier :: String -> String:
import Data.Char(toLower)
instance ToJSON R3 where
toJSON = genericToJSON defaultOptions {
fieldLabelModifier = map toLower . drop 3
}
We thus use the toLower :: Char -> Char function of the Data.Char module, and perform a mapping, such that all characters in the string are mapped.
Note that if you simply want to derive FromJson and ToJSON with different options, you can make use of template Haskell, like:
{-# LANGUAGE DeriveGeneric, TemplateHaskell #-}
import Data.Char(toUpper)
import Data.Aeson.TH(deriveJSON, defaultOptions, Options(fieldLabelModifier))
data Test = Test { attribute :: String } deriving Show
$(deriveJSON defaultOptions {fieldLabelModifier = map toUpper . drop 3} ''Test)
In that case the template Haskell part will implement the FromJSON and ToJSON instances.
Note: We can use qualified imports in order to make it more clear what function we use, for example:
import qualified Data.List as L
import qualified Data.Char as C
instance ToJSON R3 where
toJSON = genericToJSON defaultOptions {
fieldLabelModifier = map C.toLower . L.drop 3
}
Note: As for the smart constructor, you can simplify this expression to:
makeR3 = R3

Parsing an Options Object to a List of Options

I have been working on a small library against an JSON-based API. This library makes use of an "options" object, a series of key-value pairs that specify advanced behaviour:
{
"id": 1234
...
"options": {
"notify_no_data": True,
"no_data_timeframe": 20,
"notify_audit": False,
"silenced": {"*" :1428937807}
}
}
I've represented the options concept in Haskell using a list of options:
data Option = NotifyNoData NominalDiffTime -- notify after xyz
| NotifyAudit -- notify on changes
| Silenced (Maybe UTCTime) -- silence all notifications ("*") until xyz (or indefinitely)
newtype Options = Options [Option]
Implementing toJSON was simple enough; I instantiated toJSON for the Option type, and then used those as helpers for the Options type.
instance ToJSON Option where
toJSON (Silenced mtime) =
Object $ Data.HashMap.fromList [("silenced", mapping)]
where stamp = maybe Null (jsonTime . floor . utcTimeToPOSIXSeconds) mtime
mapping = Object $ Data.HashMap.singleton "*" stamp
toJSON (NotifyNoData difftime) =
Object $ Data.HashMap.fromList [("notify_no_data", Bool True)
,("no_data_timeframe", stamp)]
where stamp = jsonTime $ floor (difftime / 60)
toJSON NotifyAudit =
Object $ Data.HashMap.fromList [("notify_audit", Bool True)]
instance ToJSON Options where
toJSON (Options options) = Object $ Data.HashMap.unions $ reverse $ (opts:) $ map ((\(Object o) -> o) . toJSON) options
where opts = Data.HashMap.fromList [("silenced", Object Data.HashMap.empty)
,("notify_no_data", Bool False)
,("notify_audit", Bool False)]
The problem I am running into is in the fromJSON implementation for Options. All the use cases I've seen before supply a fairly simple json-object-representation to data-representation mapping. What I need to do is convert an object to an object of options to a list of data (Option) representations. For example, the sample JSON under "options" that I gave at the start would have to become:
Options [NotifyNoData 20, Silenced (Just (posixSecondsToUTCTime 1428937807))]
FromJSON requires an implementation of parseJSON :: FromJSON a => Value -> Parser a. I am having trouble understanding how to build a Parser using this optional object structure given by the API. Is there a standard approach to parsing a JSON object to a list like this? It may be that I am failing to fully understand the Parser typeclass.
You can maybe use the parser monad to extract all the information, something like:
parseJSON (Object v) = do
maybeNotify <- v .:? "notify_no_data"
maybeTimeFrame <- v .:? "no_data_timeframe"
let nots = case (maybeNotify,maybeTimeFrame) of
(Just True,Just stamp) -> [NotifyNoData $ fromJsontime stamp]
_ -> []
return $ Options nots
(fromJsontime is just a helper function to convert from the JSON value to your NominalDiffTime, I suppose you have something for that).
Do the same for the other types of options, concatenating the result.
Using the json-stream parser (which I am the author of), it would look like this (from my head, untested):
option = NotifyNoData <$> "no_data_timeframe" .: value
<* filterI id ("notify_no_data" .: bool)
<|> const NotifyAudit <$> filterI id ("notify_audit" .: bool)
<|> Silenced <$> "silenced" .: "*" .:? value
optionList = Options <$> toList option
The code expects you have FromJSON instance for UTCTime (or you could just use 'integer' and some kind of fromposixtime etc.).

Creating a list when parsing a file using Cassava in Haskell

I'm able to parse my csv file using the following code from Data.Csv:
valuesToList :: Foo -> (Int, Int)
valuesToList (Foo a b) = (a,b)
loadMyData :: IO ()
loadMyData = do
csvData <- BL.readFile "mydata.csv"
case decodeByName csvData of
Left err -> putStrLn err
Right (_, v) -> print $ V.toList $ V.map valuesToList v
When I run this I get the correct output on the screen. The problem I have is I'm not clear how to create a pure function so I can consume the contents of the list, as in:
let l = loadMyData
Where l is of type [Int,Int] I'm guessing it's because I'm in the IO Monad and I'm doing something hopelessly silly...
I'm doing something hopelessly silly...
Yes, but worry not!
loadMyData = BL.readFile "mydata.csv"
processMyData :: String -> String
processMyData csvData =
case decodeByName csvData of
Left err -> err
Right (_, v) -> show $ V.toList $ V.map valuesToList v
main = do
csv <- loadMyData
let output = processMyData csv
print output
This way you separate the pure, "let" part from the impure loading part. This isn't Code Review (if you asked there, I could probably elaborate more), but I'd type the processing as String -> Either String [Int, Int] or something and keep the failure information in the type system.
processMyData csvData =
case decodeByName csvData of
Left err -> Left err
Right (_, v) -> Right $ V.toList $ V.map valuesToList v
And that in turn could simply be (pardon me if I make a mistake somewhere, I'm doing it live):
processMyData = fmap (V.toList . V.map valuesToList) . decodeByName
That should work because of how the Functor instance for Either is constructed.
Oh, and also use Control.Applicative for bonus style points:
main = do
output <- processMyData <$> loadMyData
print output
(in case you don't understand that example, (<$>) is infix fmap, but that's not stricly necessary; hence bonus)

Haskell :: Aeson :: parse ADT based on field value

I'm using an external API which returns JSON responses. One of the responses is an array of objects and these objects are identified by the field value inside them. I'm having some trouble understanding how the parsing of such JSON response could be done with Aeson.
Here is a simplified version of my problem:
newtype Content = Content { content :: [Media] } deriving (Generic)
instance FromJSON Content
data Media =
Video { objectClass :: Text
, title :: Text } |
AudioBook { objectClass :: Text
, title :: Text }
In API documentation it is said that the object can be identified by the field objectClass which has value "video" for our Video object and "audiobook" for our AudioBook and so on. Example JSON:
[{objectClass: "video", title: "Some title"}
,{objectClass: "audiobook", title: "Other title"}]
The question is how can this type of JSON be approached using Aeson?
instance FromJSON Media where
parseJSON (Object x) = ???
You basically need a function Text -> Text -> Media:
toMedia :: Text -> Text -> Media
toMedia "video" = Video "video"
toMedia "audiobook" = AudioBook "audiobook"
The FromJSON instance is now really simple (using <$> and <*> from Control.Applicative):
instance FromJSON Media where
parseJSON (Object x) = toMedia <$> x .: "objectClass" <*> x .: "title"
However, at this point you're redundant: the objectClass field in Video or Audio doesn't give you more information than the actual type, so you might remove it:
data Media = Video { title :: Text }
| AudioBook { title :: Text }
toMedia :: Text -> Text -> Media
toMedia "video" = Video
toMedia "audiobook" = AudioBook
Also note that toMedia is partial. You probably want to catch invalid "objectClass" values:
instance FromJSON Media where
parseJSON (Object x) =
do oc <- x .: "objectClass"
case oc of
String "video" -> Video <$> x .: "title"
String "audiobook" -> AudioBook <$> x .: "title"
_ -> empty
{- an alternative using a proper toMedia
toMedia :: Alternative f => Text -> f (Text -> Media)
toMedia "video" = pure Video
toMedia "audiobook" = pure AudioBook
toMedia _ = empty
instance FromJSON Media where
parseJSON (Object x) = (x .: "objectClass" >>= toMedia) <*> x .: "title"
-}
And last, but not least, remember that valid JSON uses strings for the name.
The default translation for a data type like:
data Media = Video { title :: Text }
| AudioBook { title :: Text }
deriving Generic
is actually very close to what you want. (For the simplicity of my examples, I define ToJSON instances and encode the examples to see what kind of JSON we get.)
aeson, default
So, with the default instance we have (view the complete source file which produces this output):
[{"tag":"Video","title":"Some title"},{"tag":"AudioBook","title":"Other title"}]
Let's see whether we can get even closer with custom options...
aeson, custom tagFieldName
With custom options:
mediaJSONOptions :: Options
mediaJSONOptions =
defaultOptions{ sumEncoding =
TaggedObject{ tagFieldName = "objectClass"
-- , contentsFieldName = undefined
}
}
instance ToJSON Media
where toJSON = genericToJSON mediaJSONOptions
we get:
[{"objectClass":"Video","title":"Some title"},{"objectClass":"AudioBook","title":"Other title"}]
(Think yourself what you want to do with an undefined field in the real code.)
aeson, custom constructorTagModifier
Adding
, constructorTagModifier = fmap Char.toLower
to mediaJSONOptions gives:
[{"objectClass":"video","title":"Some title"},{"objectClass":"audiobook","title":"Other title"}]
Great! Exactly what you specified!
decoding
Simply add an instance with the same options to be able to decode from this format:
instance FromJSON Media
where parseJSON = genericParseJSON mediaJSONOptions
Example:
*Main> encode example
"[{\"objectClass\":\"video\",\"title\":\"Some title\"},{\"objectClass\":\"audiobook\",\"title\":\"Other title\"}]"
*Main> decode $ fromString "[{\"objectClass\":\"video\",\"title\":\"Some title\"},{\"objectClass\":\"audiobook\",\"title\":\"Other title\"}]" :: Maybe [Media]
Just [Video {title = "Some title"},AudioBook {title = "Other title"}]
*Main>
Complete source file.
generic-aeson, default
To get a more complete picture, let's also look at what generic-aeson package would give (at hackage). It has also nice default translations, different in some respects from those from aeson.
Doing
import Generics.Generic.Aeson -- from generic-aeson package
and defining:
instance ToJSON Media
where toJSON = gtoJson
gives the result:
[{"video":{"title":"Some title"}},{"audioBook":{"title":"Other title"}}]
So, it's different from all what we've seen when using aeson.
generic-aeson's options (Settings) are not interesting for us (they allow only to strip a prefix).
(The complete source file.)
aeson, ObjectWithSingleField
Apart from lower-casing the first letter of the constructor names, generic-aeson's translation seems similar to an option available in aeson:
Let's try this:
mediaJSONOptions =
defaultOptions{ sumEncoding = ObjectWithSingleField
, constructorTagModifier = fmap Char.toLower
}
and yes, the result is:
[{"video":{"title":"Some title"}},{"audiobook":{"title":"Other title"}}]
the rest of options: (aeson, TwoElemArray)
One available option for sumEncoding has been left out from consideration above, because it gives an array which is not quite similar to the JSON representation asked about. It's TwoElemArray. Example:
[["video",{"title":"Some title"}],["audiobook",{"title":"Other title"}]]
is given by:
mediaJSONOptions =
defaultOptions{ sumEncoding = TwoElemArray
, constructorTagModifier = fmap Char.toLower
}

JSON parsing befuddlement

I'm working on the I/O aspect of my json server, and there's a method I just can't get right.
First, I'll give the error, then the code and datatypes involved and some commentary about the problem afterwards.
("X-Response-Body-Start","<!DOCTYPE html>\n<html><head><title>Invalid Arguments</title></head><body><h1>Invalid Arguments</h1><ul><li>when expecting a unit constructor (U1), encountered String instead</li></ul></body></html>")
Expecting unit contructor?
Okay here's some relevant code. Let's see if we can see where I go wrong
from Datatypes.hs
data JobID = JobID Project Int deriving Generic
data Project = BNAP deriving (Show,Generic) -- one day to be an ADT
instance ToJSON Project where
toJSON = toJSON . show
instance FromJSON Project
instance FromJSON JobID
instance ToJSON JobID
The test code
testReadR :: IO Value
testReadR = do
req <- parseUrl readURI
manager <- newManager def
pBody <- runResourceT $ do
reqBody <- readObject
liftIO $ print reqBody
Response _ _ _ body <- http (buildReq req reqBody) manager
pBody <- body $$+- sinkParser json
return pBody -- (return wraps it up)
closeManager manager
return pBody
buildReq :: forall a (m :: * -> *) (t :: * -> *).
ToJSON a =>
Request t -> a -> Request m
buildReq req reqBody =
let reqBodyBS = Data.Aeson.encode reqBody
rHeaders = [(hContentType,pack "application/json")]
in req {method = fromString "POST"
, requestBody = RequestBodyLBS reqBodyBS
,requestHeaders=rHeaders
}
readObject :: ResourceT IO Value
readObject = do -- I took a bunch out because I thought simplifiying would help me
-- solve this
return $ Data.Aeson.toJSON $ JobID BNAP 306
The Handler
postReadR :: Handler RepJson
postReadR = do
conf <- parseJsonBody_ :: Handler JobID
liftIO $ print conf
testJ <- jsonToRepJson $ toJSON $ JobID BNAP 305
jValue <- jsonToRepJson conf -- to be replaced with
-- Either ErrorReport Response
-- (or something like that)
return jValue
when I change the line to
conf <- parseJsonBody_ :: Handler Value
print conf yields
Array (fromList [String "BNAP",Number 306])
So it seems the problem lies with String "BNAP" but I don't know why. Any ideas on how I can suss this out? Is there an obvious answer I'm not seeing?
Update : I have a new error. I'm sure I borked the FromJSON instance.
test: ResponseTimeout
instance FromJSON Project where
parseJSON (String p) = parseJSON $ toJSON p
parseJSON _ = mzero
The challenge here was that Project is a unary type. None of the examples I studied seemed to address that. But I know p is a Text, and toJSON can make a Value out of that, and parseJSON makes a Parser out of the Value. So it's all good right? Well, I still get the above error which is not informative at all. Any ideas?
instance ToJSON Project where
toJSON = toJSON . show
instance FromJSON Project
The instances don't work together. The generic FromJSON instance expects the generic unit constructor U1, but the ToJSON instance produces a String "BNAP".
If you have a hand-written ToJSON instance, you also need a hand-written FromJSON instance.
instance FromJSON Project where
parseJSON (String p) = parseJSON $ toJSON p
parseJSON _ = mzero
p is a Text, and we have
instance ToJSON Text where
toJSON = String
so the above instance for Project loops, since it expands parseJSON (String p) = parseJSON (String p).
For the type as it stands now,
instance FromJSON Project where
parseJSON (String "BNAP") = return BNAP
parseJSON _ = mzero
should work if you have OverloadedStrings enabled, if not,
instance FromJSON Project where
parseJSON (String p)
| p == pack "BNAP" = return BNAP
parseJSON _ = mzero