Fault tolerant JSON parsing - json

I'm using Data.Aeson to parse some JSON into a Record type. From time to time data is added to the JSON and this breaks my code as Aeson complains something to the effect of:
expected Object with 21
name/value pairs but got 23 name/value
I'd really prefer to parse the JSON in a fault tolerant way -- I don't care if more fields are added to the JSON at a later date, just parse whatever you can! Is there a way to achieve this fault tolerance? Here's my code:
myRecordFromJSONString :: BS.ByteString -> Maybe MyRecord
myRecordFromJSONString s = case Data.Attoparsec.parse json s of
Done _rest res -> Data.Aeson.Types.parseMaybe parseJSON res
_ -> Nothing
I should add that I'm using deriveJSON from Data.Aeson.TH to generate the parsing code. If I write the FromJSON code manually it's fault tolerant but I'd like to not have to do that...

If you are using GHC 7.2 or 7.4, the new generics support in aeson doesn't check for extra fields. I'm not sure if this is by design or not but we use it for the same reason.
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import qualified Data.Aeson.Types
import Data.Attoparsec
import qualified Data.ByteString as BS
import Data.ByteString.Char8 ()
import GHC.Generics
data MyRecord = MyRecord
{ field1 :: Int
} deriving (Generic, Show)
instance FromJSON MyRecord
myRecordFromJSONString :: BS.ByteString -> Maybe MyRecord
myRecordFromJSONString s = case Data.Attoparsec.parse json s of
Done _rest res -> Data.Aeson.Types.parseMaybe parseJSON res
_ -> Nothing
main :: IO ()
main = do
let parsed = myRecordFromJSONString "{ \"field1\": 1, \"field2\": 2 }"
print parsed
Running this would fail with the TH derived instance due to 'field2' not existing in the record. The Generic instance returns the desired result:
Just (MyRecord {field1 = 1})

Related

Haskell - how do I convert piped-in JSON-based to a data record?

I'm building a reinforcement learning library where I'd like to pass certain instance information into the executables via a piped JSON.
Using aeson's Simplest.hs, I'm able to get the following basic example working as intended. Note that the parameters are sitting in Main.hs as a String params as a placeholder.
I tried to modify Main.hs so I would pipe the Nim game parameters in from a JSON file via getContents, but am running into the expected [Char] vs. IO String issue. I've tried to read up as much as possible about IO, but can't figure out how to lift my JSON parsing method to deal with IO.
How would I modify the below so that I can work with piped-in JSON?
Main.hs
module Main where
import qualified System.Random as Random
import qualified Data.ByteString.Lazy.Char8 as BL
import qualified Games.Engine as Engine
import qualified Games.IO.Nim as NimIO
import qualified Games.Rules.Nim as Nim
import qualified Games.Learn.ValueIteration as VI
main :: IO ()
main = do
let params = "{\"players\":[\"Bob\", \"Alice\", \"Charlie\"], \"initialPiles\": [3, 4, 5], \"isMisere\": false}"
let result = NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
case result of
Nothing -> putStrLn "Parameter errors."
Just game -> do
putStrLn "Let's play some Nim! Remainder of code omitted"
Games.IO.Nim.hs
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-}
module Games.IO.Nim
( decode
, NimGame
, players
, initialPiles
, isMisere
) where
import Control.Applicative (empty)
import qualified Data.ByteString.Lazy.Char8 as BL
import Data.Aeson
( pairs,
(.:),
object,
FromJSON(parseJSON),
Value(Object),
KeyValue((.=)),
ToJSON(toJSON, toEncoding),
decode)
data NimGame = NimGame
{ players :: [String]
, initialPiles :: [Int]
, isMisere :: Bool
} deriving (Show)
instance ToJSON NimGame where
toJSON (NimGame playersV initialPilesV isMisereV) = object [ "players" .= playersV,
"initialPiles" .= initialPilesV,
"isMisere" .= isMisereV]
toEncoding NimGame{..} = pairs $
"players" .= players <>
"initialPiles" .= initialPiles <>
"isMisere" .= isMisere
instance FromJSON NimGame where
parseJSON (Object v) = NimGame <$>
v .: "players" <*>
v .: "initialPiles" <*>
v .: "isMisere"
parseJSON _ = empty
Alternative Main.hs that generates compile error
module Main where
import qualified System.Random as Random
import qualified Data.ByteString.Lazy.Char8 as BL
import qualified Games.Engine as Engine
import qualified Games.IO.Nim as NimIO
import qualified Games.Rules.Nim as Nim
import qualified Games.Learn.ValueIteration as VI
main :: IO ()
main = do
--let params = "{\"players\":[\"Bob\", \"Alice\", \"Charlie\"], \"initialPiles\": [3, 4, 5], \"isMisere\": false}"
let params = getContents
let result = NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
case result of
Nothing -> putStrLn "Parameter errors."
Just game -> do
putStrLn "Let's play some Nim!"
Compile Error
(base) randm#pearljam ~/Projects/gameshs $ stack build
gameshs-0.1.0.0: unregistering (local file changes: app/Nim.hs)
gameshs> configure (lib + exe)
Configuring gameshs-0.1.0.0...
gameshs> build (lib + exe)
Preprocessing library for gameshs-0.1.0.0..
Building library for gameshs-0.1.0.0..
Preprocessing executable 'nim-exe' for gameshs-0.1.0.0..
Building executable 'nim-exe' for gameshs-0.1.0.0..
[2 of 2] Compiling Main
/home/randm/Projects/gameshs/app/Nim.hs:17:41: error:
• Couldn't match expected type ‘[Char]’
with actual type ‘IO String’
• In the first argument of ‘BL.pack’, namely ‘params’
In the second argument of ‘($)’, namely ‘BL.pack params’
In the expression:
NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
|
17 | let result = NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
| ^^^^^^
-- While building package gameshs-0.1.0.0 (scroll up to its section to see the error) using:
/home/randm/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_3.2.1.0_ghc-8.10.4 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0 build lib:gameshs exe:nim-exe --ghc-options " -fdiagnostics-color=always"
Process exited with code: ExitFailure 1
getContents returns not a String as you apparently expect, but IO String, which is a "program", which, when executed, will produce a String. So when you're trying to parse this program with decode, of course that doesn't work: decode parses a String, it cannot parse a program.
So how do you execute this program to obtain the String? There are two ways: either you make it part of another program or you call it main and it becomes your entry point.
In your case, the sensible thing to do would be to make getContent part of your main program. To do that, use the left arrow <-, like this:
main = do
params <- getContents
let result = NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
...

Mustache not rendering JSON value as JSON encoded string

The full example is here:
{-# LANGUAGE OverloadedStrings #-}
module Test2 where
import Data.Aeson
import Text.Mustache
main :: IO ()
main = do
let example = Data.Aeson.object [ "key" .= (5 :: Integer), "somethingElse" .= (2 :: Integer) ]
print . encode $ example
print ("Start" :: String)
case compileTemplate "" "{{{jsonData}}}" of
Right x -> do
print $ substituteValue x (Text.Mustache.object ["jsonData" ~= example])
Left e -> error . show $ e
The above produces the following output:
"{\"somethingElse\":2,\"key\":5}"
"Start"
"fromList [(\"somethingElse\",2.0),(\"key\",5.0)]"
My expectation is it would produce:
"{\"somethingElse\":2,\"key\":5}"
"Start"
"{\"somethingElse\":2,\"key\":5}"
Mustache doesn't seem to support straight substitution of JSON objects. Setting up a similar example here, I receive
[object Object]
as output. Not identical to yours, but it indicates that the issue is not necessarily with the Haskell implementation.
In other words, I believe the issue is with your template {{{jsonData}}}.
If you change it to {{{jsonData.somethingElse}}}, it works fine (I know this is not what you want).
Alternatively, encode the JSON data as text prior to passing it to the substitution function as suggested here. Basically something like this:
substituteValue x (Text.Mustache.object ["jsonData" ~= (encodeToLazyText jsonData)])
This results in your desired output. encodeToLazyText is found in Data.Aeson.Text.
Working code:
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Data.Aeson ((.=))
import qualified Data.Aeson as A
import Data.Aeson.Text (encodeToLazyText)
import Text.Mustache ((~=))
import qualified Text.Mustache as M
import qualified Text.Mustache.Types as M
main :: IO ()
main = do
print . A.encode $ jsonData
putStrLn "Start"
case M.compileTemplate "" "in mustache: {{{jsonData}}}" of
Right template ->
print (M.substituteValue template mustacheVals)
Left e ->
error . show $ e
jsonData :: A.Value
jsonData =
A.object
[ "key" .= (5 :: Integer)
, "somethingElse" .= (2 :: Integer)
]
mustacheVals :: M.Value
mustacheVals =
M.object
[ "jsonData" ~= encodeToLazyText jsonData
]

Write json to a file in Haskell (with Text rather than [Char])

I'm trying to serialize an object into a JSON string and write it to a file.
In python, I'd do something like:
>>> meowmers = {"name" : "meowmers", "age" : 1}
>>> import json
>>> with open("myfile.json","wb") as f
json.dump(meowmers, f)
$ cat myfile.json
{"age": 1, "name": "meowmers"}
I'm looking at this in Haskell
$ stack ghci
{-# LANGUAGE OverloadedStrings #-}
:set -XOverloadedStrings
import GHC.Generics
import Data.Aeson as A
import Data.Text.Lazy as T
import Data.Text.Lazy.IO as I
:{
data Cat = Cat {
name :: Text
, age :: Int
} deriving Show
:}
let meowmers = Cat {name = "meowmers", age = 1}
writeFile "myfile.json" (encode meowmers)
Oh no!
*A T I GHC.Generics> I.writeFile "myfile2.json" (encode meowmers)
<interactive>:34:29:
Couldn't match expected type ‘Text’
with actual type ‘bytestring-0.10.6.0:Data.ByteString.Lazy.Internal.ByteString’
In the second argument of ‘I.writeFile’, namely ‘(encode meowmers)’
In the expression: I.writeFile "myfile2.json" (encode meowmers)
Two questions:
This appears to be a bytestring. How can I work with that?
If that's not what I want to do, is there a Haskell json serialization solution using Text rather than String that is yet rather simple?
So, to iron everything out (since most of the work has already been done). You actually have two problems:
You are mixing string types
You don't have an instance of ToJSON declared for Cat
Here is a working example that relies on recent versions of aeson and text (for me that is aeson-1.0.0.0 and text-1.2.2.1.
{-# LANGUAGE OverloadedStrings, DeriveGeneric, DeriveAnyClass #-}
import GHC.Generics
import Data.Text.Lazy (Text)
import Data.Text.Lazy.IO as I
import Data.Aeson.Text (encodeToLazyText)
import Data.Aeson (ToJSON)
data Cat = Cat { name :: Text, age :: Int } deriving (Show, Generic, ToJSON)
meowmers = Cat { name = "meowmers", age = 1 }
main = I.writeFile "myfile.json" (encodeToLazyText meowmers)
As you can probably tell from the imports, I rely on aeson to convert between string types through encodeToLazyText. That deals with problem number 1.
Then, I use the language extension DeriveGeneric to get a Generic instance for Cat, and use that in conjunction with the extension DeriveAnyClass to get an instance of ToJSON for Cat. The magic of that instance is again part of aeson.
Running this, I get a new file myfile.json that contains {"age":1,"name":"meowmers"} in it.
You can encode JSON to a lazy Text value directly using Data.Aeson.Text.encodeToLazyText.
{-# LANGUAGE DeriveGeneric #-}
import Data.Aeson.Text (encodeToLazyText)
...
I.writeFile "myfile.json" (encodeToLazyText meowmers)
A bytestring is a type for binary data—not necessarily text. To represent textual data in a bytestring, you need to encode it with some encoding like UTF-8. Once you have a bytestring (encoded with UTF-8 or whatever format makes sense), you can write it to a file using Data.ByteString functions:
import qualified Data.ByteString.Lazy as BS
BS.writeFile "myfile.json" (encode meowmers)
To make this work you need to give your Cat type a ToJSON instance that specifies how to encode it in JSON. You can do this automatically with the DeriveGeneric extension:
data Cat = Cat { ... } deriving (Show, Generic)
instance ToJSON Cat
You can also do this manually if you need finer control over what the resulting JSON looks like.
Rolling all the comments and answers into one (credit to the other folks here, please accept one of their answers).
Derive Generic and ToJSON (or manually write a ToJSON instance if you'd like). This required a few more LANGUAGE pragmas.
Use either a newer version of text and encodeToLazyText OR use Data.ByteString.Lazy.writeFile to write out the bytestring.
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE DeriveAnyClass #-}
import GHC.Generics
import Data.Aeson (encode,ToJSON(..))
import Data.Text.Lazy (Text)
import qualified Data.ByteString.Lazy as BS
data Cat = Cat { name :: Text
, age :: Int
} deriving (Show,Generic,ToJSON)
main =
do let meowmers = Cat {name = "meowmers", age = 1}
BS.writeFile "myfile.json" (encode meowmers)
Resulting in:
tommd#HalfAndHalf /tmp% runhaskell so.hs
tommd#HalfAndHalf /tmp% cat myfile.json
{"age":1,"name":"meowmers"}

How to validate JSON with schema in Haskell?

I want to validate JSON with a schema. hjsonschema seemed like a good choice as it is fairly new and supports the latest draft.
But the plotly json schema always gives me valid responses.
I may be misunderstanding something here but this should not be valid JSON
bad.json
{
"fjsdklj" : 5
}
even though it is considered valid by the following code
module Main where
import Control.Applicative
import Data.Aeson
import Data.HashMap.Strict (HashMap)
import qualified Data.HashMap.Strict as H
import Data.Monoid
import qualified Data.ByteString.Lazy as B
import qualified Data.JsonSchema as JS
import Data.Maybe
main :: IO ()
main = do
schemaJSON <- (fromJust . decode) <$> B.readFile "simple-schema.json"
bad <- (fromJust . decode) <$> B.readFile "bad.json"
let schemaData = JS.RawSchema {
JS._rsURI = Nothing,
JS._rsData = schemaJSON
}
schema <- compileSchema (JS.SchemaGraph schemaData H.empty) schemaData
checkResults (JS.validate schema bad)
compileSchema :: JS.SchemaGraph -> JS.RawSchema -> IO (JS.Schema JS.Draft4Failure)
compileSchema graph rs =
case JS.compileDraft4 graph rs of
Left failure -> error $ "Not a valid schema: " <> show failure
Right schema -> return schema
checkResults :: [JS.ValidationFailure JS.Draft4Failure] -> IO ()
checkResults [] = putStrLn "Just fine"
checkResults x = putStrLn $ "ERROR: " ++ show x
simple-schema.json is the plotly schema and bad.json the snippet I posted above.
It's nothing about Haskell.
Your schema doesn't have a required property at top level so an empty json object is acceptable.
Also it contains no "additionalProperties": false property so anything that does not fit into one of the defined patterns are ignored.
BTW I doubt whether it's a valid Draft4 json schema. It passed the validation of the Draft4 meta-schema but the syntax is a little bit different, maybe something Python-specific. You'd better run the test suites which came along with the hjsonscheme package to see if everything works fine.

Creating a csv file in haskell

I suspect this a bit of a noobie question...
So I have a list myList that I would like to write to a csv file, using the cassava library:
import qualified Data.ByteString.Lazy as BL
import qualified Data.Vector as V
import Data.Csv
main = BL.writeFile "myFileLocation" (encode $ V.fromList myList)
as far as I can establish, encode has type ToRecord a => Vector a -> ByteString yet I'm getting the following error:
Couldn't match expected type `[a0]'
with actual type `V.Vector (a,b,c,d)'
In the return type of a call of `V.fromList'
In the second argument of `($)', namely `V.fromList myList'
In the second argument of `BL.writeFile', namely
`(encode $ V.fromList myList)'
I'm confused!