AESON: Parse dynamic structure - json

I have a JSON structure like this
{
"tag1": 1,
"tag2": 7,
...
}
And I have a type like this
data TagResult { name :: String, numberOfDevicesTagged :: Int } deriving (Show, Eq)
newtype TagResultList = TagResultList { tags :: [TagResult] }
The tag names are of course fully dynamic and I don't know them at compile time.
I'd like to create an instance FromJSON to parse the JSON data but I just cannot make it compile.
How can I define parseJSON to make this happen?

You can use the fact that Object is an HasMap and extract the key at runtime. You can then write the FromJSON instance as follows -
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Data.Aeson
import qualified Data.Text as T
import qualified Data.HashMap.Lazy as HashMap
data TagResult = TagResult { name :: String
, numberOfDevicesTagged :: Int
} deriving (Show, Eq)
newtype TagResultList = TagResultList { tags :: [TagResult] } deriving Show
instance ToJSON TagResult where
toJSON (TagResult tag ntag) =
object [ T.pack tag .= ntag ]
instance ToJSON TagResultList where
toJSON (TagResultList tags) =
object [ "tagresults" .= toJSON tags ]
instance FromJSON TagResult where
parseJSON (Object v) =
let (k, _) = head (HashMap.toList v)
in TagResult (T.unpack k) <$> v .: k
parseJSON _ = fail "Invalid JSON type"
instance FromJSON TagResultList where
parseJSON (Object v) =
TagResultList <$> v .: "tagresults"
main :: IO ()
main = do
let tag1 = TagResult "tag1" 1
tag2 = TagResult "tag2" 7
taglist = TagResultList [tag1, tag2]
let encoded = encode taglist
decoded = decode encoded :: Maybe TagResultList
print decoded
The above program should print the tag result list.
Just (TagResultList {tags = [TagResult {name = "tag1", numberOfDevicesTagged = 1},TagResult {name = "tag2", numberOfDevicesTagged = 7}]})

Related

Escaping Haskell record's field

Question
Is there a way to create a record with a field called "data"?
data MyRecord =
MyRecord {
otherField :: String,
data :: String -- compilation error
}
Why do I need it?
I've been writing a wrapper around a JSON API using Aeson and the remote service decided to call one of the fields data.
{
pagination: {
..
},
data: [
{ .. },
{ .. },
]
}
Yes, you can name the field something else than data, like:
data MyRecord =
MyRecord {
otherField :: String,
recordData :: String
}
And then derive a ToJSON with a key modifier:
labelMapping :: String -> String
labelMapping "recordData" = "data"
labelMapping x = x
instance ToJSON MyRecord where
toJSON = genericToJSON defaultOptions {
fieldLabelModifier = labelMapping
}
instance FromJSON Coord where
parseJSON = genericParseJSON defaultOptions {
fieldLabelModifier = labelMapping
}

isNothing throws an exception when parsing JSON file

I've got the following functions to decode JSON files using the Data.Aeson library:
data SearchResult = SearchResult {
items :: [Item]
} deriving (Show)
instance FromJSON SearchResult where
parseJSON :: Value -> Parser SearchResult
parseJSON (Object v) = SearchResult <$>
parseJSON (fromJust $ HM.lookup "items" v)
parseJSON _ = mzero
data Item = Item {
volumeInfo :: VolumeInfo
} deriving (Show)
instance FromJSON Item where
parseJSON :: Value -> Parser Item
parseJSON (Object v) = Item <$>
parseJSON (fromJust $ HM.lookup "volumeInfo" v)
parseJSON _ = mzero
data VolumeInfo = VolumeInfo {
title :: String,
authors :: [String],
publisher :: String,
publishedDate :: String,
industryIdentifiers :: [IndustryIdentifier],
pageCount :: Int,
categories :: [String]
} deriving (Show)
instance FromJSON VolumeInfo where
parseJSON :: Value -> Parser VolumeInfo
parseJSON (Object v) = VolumeInfo <$>
v .: "title" <*>
v .: "authors" <*>
v .: "publisher" <*>
v .: "publishedDate" <*>
parseJSON (fromJust $ HM.lookup "industryIdentifiers" v) <*>
v .: "pageCount" <*>
v .: "categories"
parseJSON _ = mzero
data IndustryIdentifier = IndustryIdentifier {
identifierType :: String,
identifier :: String
} deriving (Show)
instance FromJSON IndustryIdentifier where
parseJSON :: Value -> Parser IndustryIdentifier
parseJSON (Object v) = IndustryIdentifier <$>
v .: "type" <*>
v .: "identifier"
parseJSON _ = mzero
And this function:
getBook content = do
putStrLn (Data.ByteString.Lazy.Char8.unpack content)
let searchResult = decode content :: Maybe SearchResult
print (isNothing searchResult)
print searchResult
The function getBook works with many JSON files. Here's an example:
False
Just (SearchResult {items = [Item {volumeInfo = VolumeInfo {title = "A Memoir of Jane Austen", authors = ["James Edward Austen-Leigh","Jane Austen, James Austen-Leigh"], publisher = "Wordsworth Editions", publishedDate = "2007", industryIdentifiers = [IndustryIdentifier {identifierType = "ISBN_10", identifier = "1840225602"},IndustryIdentifier {identifierType = "ISBN_13", identifier = "9781840225600"}], pageCount = 256, categories = ["Novelists, English"]}}]})
The JSON content was successfully decoded, and therefore isNothing returns False in the first line, followed by the decoded content. If I run the function again with this JSON file as content, I get the following output:
True
Nothing
The file couldn't be decoded (as there is no field categories on the JSON file), and so isNothing returns True, and Nothing is printed on screen. Now the problem is when I run it with this JSON file as content. I get this:
*** Exception: Maybe.fromJust: Nothing
An exception is thrown when print (isNothing searchResult) is executed, and I don't understand why True isn't returned like in the previous example (because in this case there is no field industryIdentifiers, for example). What am I missing or doing wrong?
EDIT:
I found out the problem happens every time the JSON file doesn't include the field industryIdentifiers. It fails in this line:
parseJSON (fromJust $ HM.lookup "industryIdentifiers" v) <*>
The github package defines a convenience operator to define array fields:
-- | A slightly more generic version of Aeson's #(.:?)#, using `mzero' instead
-- of `Nothing'.
(.:<) :: (FromJSON a) => Object -> T.Text -> Parser [a]
obj .:< key = case Map.lookup key obj of
Nothing -> pure mzero
Just v -> parseJSON v
(here Map is an alias for Data.HashMap.Lazy)
Then the FromJSON instance of VolumeInfo would be defined like this:
instance FromJSON VolumeInfo
v .: "title" <*>
...
v .:< "industryIdentifiers" <*>
...

Parsing Nested JSON in Haskell with Aeson

I'm trying to parse JSON from a RESTful API. The returned JSON is highly nested and may/may not include certain fields. Here is an example of some returned data:
{
resultSet : {
location : [{
desc : "Tuality Hospital/SE 8th Ave MAX Station",
locid : 9843,
dir : "Eastbound",
lng : -122.978016886765,
lat : 45.5212880911494
}
],
arrival : [{
detour : false,
status : "estimated",
locid : 9843,
block : 9024,
scheduled : "2014-03-02T16:48:15.000-0800",
shortSign : "Blue to Gresham",
dir : 0,
estimated : "2014-03-02T16:48:15.000-0800",
route : 100,
departed : false,
blockPosition : {
at : "2014-03-02T16:16:43.579-0800",
feet : 3821,
lng : -122.9909514,
trip : [{
progress : 171494,
desc : "Hatfield Government Center",
pattern : 140,
dir : 1,
route : 100,
tripNum : "4365647",
destDist : 171739
}, {
progress : 0,
desc : "Cleveland Ave",
pattern : 10,
dir : 0,
route : 100,
tripNum : "4365248",
destDist : 3577
}
],
lat : 45.5215368,
heading : 328
},
fullSign : "MAX Blue Line to Gresham",
piece : "1"
}, {
detour : false,
status : "estimated",
locid : 9843,
block : 9003,
scheduled : "2014-03-02T17:05:45.000-0800",
shortSign : "Blue to Gresham",
dir : 0,
estimated : "2014-03-02T17:05:45.000-0800",
route : 100,
departed : false,
blockPosition : {
at : "2014-03-02T16:34:33.787-0800",
feet : 3794,
lng : -122.9909918,
trip : [{
progress : 171521,
desc : "Hatfield Government Center",
pattern : 140,
dir : 1,
route : 100,
tripNum : "4365648",
destDist : 171739
}, {
progress : 0,
desc : "Cleveland Ave",
pattern : 10,
dir : 0,
route : 100,
tripNum : "4365250",
destDist : 3577
}
],
lat : 45.5216054,
heading : 345
},
fullSign : "MAX Blue Line to Gresham",
piece : "1"
}
],
queryTime : "2014-03-02T16:35:21.039-0800"
}
}
As you can see, the JSON schema starts with a resultSet which contains a location, arrival, and queryTime. The location in turn, contains a list of locations, arrival contains a list of arrivals, and queryTime is just a UTC time. Then, an arrival can contain a blockPosition, which can contain a trip, etc. Lots of nesting. Lots of optional fields.
To hold all this, I've created a set of new data types. The data types are nested similarly. For each data type, I have an instance of FromJSON (from the Aeson library).
-- Data Type Definitions and FromJSON Instance Definitions ---------------------
data ResultSet
= ResultSet { locations :: LocationList
,arrivals :: ArrivalList
,queryTime :: String
} deriving Show
instance FromJSON ResultSet where
parseJSON (Object o) =
ResultSet <$> ((o .: "resultSet") >>= (.: "location"))
<*> ((o .: "resultSet") >>= (.: "arrival"))
<*> ((o .: "resultSet") >>= (.: "queryTime"))
parseJSON _ = mzero
data TripList = TripList {triplist :: [Trip]} deriving Show
instance FromJSON TripList where
parseJSON (Object o) =
TripList <$> (o .: "trip")
parseJSON _ = mzero
data LocationList = LocationList {locationList :: [Location]} deriving Show
instance FromJSON LocationList where
parseJSON (Object o) =
LocationList <$> (o .: "location")
parseJSON _ = mzero
data Location
= Location { loc_desc :: String
,loc_locid :: Int
,loc_dir :: String
,loc_lng :: Double
,loc_lat :: Double
} deriving Show
instance FromJSON Location where
parseJSON (Object o) =
Location <$> (o .: "desc")
<*> (o .: "locid")
<*> (o .: "dir")
<*> (o .: "lng")
<*> (o .: "lat")
parseJSON _ = mzero
data ArrivalList = ArrivalList {arrivalList :: [Arrival]} deriving Show
instance FromJSON ArrivalList where
parseJSON (Object o) =
ArrivalList <$> (o .: "arrival")
parseJSON _ = mzero
data Arrival
= Arrival { arr_detour :: Bool
,arr_status :: String
,arr_locid :: Int
,arr_block :: Int
,arr_scheduled :: String
,arr_shortSign :: String
,arr_dir :: Int
,estimated :: Maybe String
,route :: Int
,departed :: Bool
,blockPosition :: Maybe BlockPosition
,fullSign :: String
,piece :: String
} deriving Show
instance FromJSON Arrival where
parseJSON (Object o) =
Arrival <$> (o .: "detour")
<*> (o .: "status")
<*> (o .: "locid")
<*> (o .: "block")
<*> (o .: "scheduled")
<*> (o .: "shortSign")
<*> (o .: "dir")
<*> (o .:? "estimated")
<*> (o .: "route")
<*> (o .: "departed")
<*> (o .:? "blockPosition")
<*> (o .: "fullSign")
<*> (o .: "piece")
parseJSON _ = mzero
data BlockPosition
= BlockPosition { bp_at :: String
,bp_feet :: Int
,bp_lng :: Double
,bp_trip :: Trip
,bp_lat :: Double
,bp_heading :: Int
} deriving Show
instance FromJSON BlockPosition where
parseJSON (Object o) =
BlockPosition <$> (o .: "at")
<*> (o .: "feet")
<*> (o .: "lng")
<*> (o .: "trip")
<*> (o .: "lat")
<*> (o .: "heading")
parseJSON _ = mzero
data Trip
= Trip { trip_progress :: Int
,trip_desc :: String
,trip_pattern :: Int
,trip_dir :: Int
,trip_route :: Int
,trip_tripNum :: Int
,trip_destDist :: Int
} deriving Show
instance FromJSON Trip where
parseJSON (Object o) =
Trip <$> (o .: "progress")
<*> (o .: "desc")
<*> (o .: "pattern")
<*> (o .: "dir")
<*> (o .: "route")
<*> (o .: "tripNum")
<*> (o .: "destDist")
parseJSON _ = mzero
Now, the problem: Retrieving the data is easy. I can show the raw JSON by
json <- getJSON stopID
putStrLn (show (decode json :: (Maybe Value)))
But when I try to get the ResultSet data, it fails with Nothing.
putStrLn (show (decode json :: Maybe ResultSet))
However, if I remove the nested data and simply try to get the queryString field (by removing the fields from the data type and instance of FromJSON, it succeeds and returns the queryString field.
data ResultSet
= ResultSet {
queryTime :: String
} deriving Show
instance FromJSON ResultSet where
parseJSON (Object o)
= ResultSet <$> ((o .: "resultSet") >>= (.: "queryTime"))
parseJSON _ = mzero
What am I doing wrong? Is this the easiest method of parsing JSON in Haskell? I'm a total noob at this (a student), so please be gentle.
I solved my problem. I was trying to create data types for my lists of JSON objects returned. For example, for location data, which is returned as a list of locations:
resultSet : {
location : [{
desc : "Tuality Hospital/SE 8th Ave MAX Station",
locid : 9843,
dir : "Eastbound",
lng : -122.978016886765,
lat : 45.5212880911494
}
],
I was setting up an Arrivals data type containing a list of [Arrival]:
data ArrivalList = ArrivalList {arrivalList :: [Arrival]} deriving Show
Then, when I tried to parse the JSON, I was trying to stuff an ArrivalList into my ResultSet, which is later used to parse the JSON data inside it. But since ArrivalList is not a JSON object, it was failing.
The fix is to NOT use custom data types for the lists. Instead, assign a list to a JSON !Array object, which can later be parsed into its own objects and sub-objects.
data ResultSet
= ResultSet {
locations :: !Array
,arrivals :: !Array
,queryTime :: String
} deriving Show
Putting it all together:
data ResultSet
= ResultSet {
locations :: !Array
,arrivals :: !Array
,queryTime :: String
} deriving Show
instance FromJSON ResultSet where
parseJSON (Object o) = ResultSet <$>
((o .: "resultSet") >>= (.: "location"))
<*> ((o .: "resultSet") >>= (.: "arrival"))
<*> ((o .: "resultSet") >>= (.: "queryTime"))
parseJSON _ = mzero
data Location
= Location { loc_desc :: String
,loc_locid :: Int
,loc_dir :: String
,loc_lng :: Double
,loc_lat :: Double
} deriving Show
instance FromJSON Location where
parseJSON (Object o) =
Location <$> (o .: "desc")
<*> (o .: "locid")
<*> (o .: "dir")
<*> (o .: "lng")
<*> (o .: "lat")
parseJSON _ = mzero
I tried !Array. It won't parse further.
I found what this blog says.
"Aeson can even parse a nested JSON if state correctly."
We just need to use [] in field definition like photo :: [Photo], making the parser keep parsing an array.

Arbitrary JSON keys with Aeson - Haskell

I have a bunch of nested JSON objects with arbitrary keys.
{
"A": {
"B": {
"C": "hello"
}
}
}
Where A, B, C are unknown ahead of time. Each of those three could also
have siblings.
I'm wondering if there is a way to parse this into a custom type with Aeson in
some elegant way. What I have been doing is loading it into an Aeson Object.
How would you go about implementing the FromJSON for this kind of JSON
object?
Thanks!
Edit:
{
"USA": {
"California": {
"San Francisco": "Some text"
}
},
"Canada": {
...
}
}
This should compile to CountryDatabase where...
type City = Map String String
type Country = Map String City
type CountryDatabase = Map String Country
You can reuse FromJSON instance of Map String v. Something like the next:
{-# LANGUAGE OverloadedStrings #-}
import Data.Functor
import Data.Monoid
import Data.Aeson
import Data.Map (Map)
import qualified Data.ByteString.Lazy as LBS
import System.Environment
newtype City = City (Map String String)
deriving Show
instance FromJSON City where
parseJSON val = City <$> parseJSON val
newtype Country = Country (Map String City)
deriving Show
instance FromJSON Country where
parseJSON val = Country <$> parseJSON val
newtype DB = DB (Map String Country)
deriving Show
instance FromJSON DB where
parseJSON val = DB <$> parseJSON val
main :: IO ()
main = do
file <- head <$> getArgs
str <- LBS.readFile file
print (decode str :: Maybe DB)
The output:
shum#shum-lt:/tmp/shum$ cat in.js
{
"A": {
"A1": {
"A11": "1111",
"A22": "2222"
}
},
"B": {
}
}
shum#shum-lt:/tmp/shum$ runhaskell test.hs in.js
Just (DB (fromList [("A",Country (fromList [("A1",City (fromList [("A11","1111"),("A22","2222")]))])),("B",Country (fromList []))]))
shum#shum-lt:/tmp/shum$
PS: You can do it without newtypes, I used them only for clarity.

FlagName to JSON

I need to convert values of type Distribution.PackageDescription.FlagName to a JSON object with the Text.JSON of the json package.
I ended up with the following approach:
instance JSON FlagName where
showJSON (FlagName n) = makeObj [ ("FlagName", showJSON n) ]
readJSON object = do
obj <- readJSON object
flag <- valFromObj "FlagName" obj
return flag
When I try to encode a value and decode it again the following happens:
> showJSON (FlagName "foo")
JSObject (JSONObject {fromJSObject = [("FlagName",JSString (JSONString {fromJSString = "foo"}))]})
> readJSON (showJSON (FlagName "foo")) :: Result FlagName
Error "Unable to read JSObject"
I guess the error is in that line: obj <- readJSON object
How do I force Haskell to use the readJSON function from the JSON String instance ?
Update: I found now a rather hackish solution:
instance JSON FlagName where
showJSON (FlagName n) = makeObj [ ("FlagName", showJSON n) ]
readJSON object = do
obj <- readJSON (showJSON (FlagName "foo")) :: Result (JSObject JSValue)
let maybeFlagName = lookup "FlagName" $ fromJSObject obj
maybe (fail "Not a FlagName object") (\jsn -> liftM FlagName $ (readJSON jsn :: Result String)) maybeFlagName
I'd appreciate it if someone comes up with a more elegant solution ...
Ok, I found the answer myself:
valFromObj returns the name of the FlagName (i.e. a String) rather than the FlagName itself.
instance JSON FlagName where
showJSON (FlagName n) = makeObj [ ("FlagName", showJSON n) ]
readJSON object = do
obj <- readJSON object
n <- valFromObj "FlagName" obj
return $ FlagName n
Better to pattern match so you can handle the case when object is no of JSObject. Instead of fail you can do some other thing. You will need to import Control.Applicative to use <$>. I just like the applicative syntax better for such things.
instance JSON FlagName where
showJSON (FlagName n) = makeObj [ ("FlagName", showJSON n) ]
readJSON (JSObject obj) = FlagName <$> valFromObj "FlagName" obj
readJSON _ = fail "unknown object"