Haskell JSON parsing with Aeson - json

Question

I have a JSON data source which looks like this:
{ "fields": [
{ "type": "datetime",
"name": "Observation Valid",
"description": "Observation Valid Time"},
{ "type": "datetime",
"name": "Observation Valid UTC",
"description": "Observation Valid Time UTC"},
{ "type": "number",
"name": "Air Temperature[F]",
"description": "Air Temperature at 2m AGL"},
{ "type": "number",
"name": "Wind Speed[kt]",
"description": "Wind Speed"},
{ "type": "number",
"name": "Wind Gust[kt]",
"description": "Wind Gust"},
{ "type": "number", "name":
"Wind Direction[deg]",
"description": "Wind Direction"}
],
"rows": [
["2018-04-22T00:10:00", "2018-04-22T05:10:00Z", 50.0, 9.0, null, 50.0],
["2018-04-22T00:15:00", "2018-04-22T05:15:00Z", 50.0, 9.0, null, 60.0],
["2018-04-22T00:20:00", "2018-04-22T05:20:00Z", 50.0, 8.0, null, 60.0],
["2018-04-22T00:30:00", "2018-04-22T05:30:00Z", 50.0, 9.0, null, 60.0]
]
}
( https://mesonet.agron.iastate.edu/json/obhistory.py?station=TVK&network=AWOS&date=2018-04-22 )
And tried several data descriptions, lastly this:
data Entry = -- Data entries
Entry { time :: Text -- Observation Valid Time
, timeUTC :: Text -- Observation Valid Time UTC
, airTemp :: Float -- Air Temperature[F] at 2m AGL
, wind :: Float -- Wind Speed [kt]
, gust :: Float -- Wind Gust [kt]
, direction :: Int -- Wind Direction[deg]
} deriving (Show,Generic)
data Field = -- Schema Definition
Field { ftype :: String --
, name :: String --
, description :: String --
} deriving (Show,Generic)
data Record =
Record { fields :: [Field] --
, rows :: [Entry] -- data
} deriving (Show,Generic)
-- Instances to convert our type to/from JSON.
instance FromJSON Entry
instance FromJSON Field
instance FromJSON Record
-- Get JSON data and decode it
dat <- (eitherDecode <$> getJSON) :: IO (Either String Record)
which gives this error:
Error in $.fields[0]: key "ftype" not present
The (first) error comes from the field definitions (which I don’t use). In the JSON the Entry’s are arrays of mixed types, but in the Haskell it is just a data structure, not an array – not sure how to reconcile these.
No doubt a beginner error – but I haven’t found any examples which seem to have this structure. Do I need to write a custom parser for this?

Answer 1

Three things prevent this from working as intended:
The JSON data contains a field named "type" . A custom FromJson instance for the Field record type can handle this.
The data in the Entry type is unnamed so it is better represented as either a data record without field names or a tuple.
The Float representing wind gust is sometimes null so it should be a Maybe Float
The code below contains all of these modifications and parses your example JSON data :
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
import Data.ByteString.Lazy as BSL
import Data.Text (Text)
import Data.Aeson
import GHC.Generics
-- Either this tuple definition of Entry or the data definition without
-- names (commented out) will work.
type Entry = -- Data entries
( Text -- Observation Valid Time
, Text -- Observation Valid Time UTC
, Float -- Air Temperature[F] at 2m AGL
, Float -- Wind Speed [kt]
, Maybe Float -- Wind Gust [kt]
, Int -- Wind Direction[deg]
)
-- data Entry = -- Data entries
-- Entry Text -- Observation Valid Time
-- Text -- Observation Valid Time UTC
-- Float -- Air Temperature[F] at 2m AGL
-- Float -- Wind Speed [kt]
-- (Maybe Float) -- Wind Gust [kt]
-- Int -- Wind Direction[deg]
-- deriving (Show,Generic)
-- instance FromJSON Entry
data Field = -- Schema Definition
Field { ftype :: String --
, name :: String --
, description :: String --
} deriving (Show,Generic)
instance FromJSON Field where
parseJSON = withObject "Field" $ \v -> Field
<$> v .: "type"
<*> v .: "name"
<*> v .: "description"
data Record =
Record { fields :: [Field] --
, rows :: [Entry] -- data
} deriving (Show,Generic)
instance FromJSON Record
getJSON :: IO ByteString
getJSON = BSL.readFile "json.txt"
main :: IO()
main = do
-- Get JSON data and decode it
dat <- (eitherDecode <$> getJSON) :: IO (Either String Record)
case dat of
Right parsed -> print parsed
Left err -> print err

Answer 2

Field has an ftype field, so AESON is trying to find ftype in the JSON but can't (as it contains ftype). I understand you can't name a field type in Haskell so you need to find a way make AESON use a different name. You need to use template Haskell and set fieldLabelModifier accordingly. Alternatively, writing the insistence manually might be simpler.