Yesod: how to send Redis results as JSON - json

Apologies in advance for my Haskell inexperience. I am writing a little wrapper for a Redis instance for a learning project. So far Yesod has been an absolute wonder. With very little Haskell experience I got browserId Auth working, and I'm inserting records into Redis successfully and quickly.
I'm stuck figuring out how to get the Redis responses converted into JSON and sent back. Here is a working, un-scaffold, app that shows getting a static RepJson or a RepPlain with Redis info (App is called LRedis):
{-# LANGUAGE OverloadedStrings, TemplateHaskell, TypeFamilies,
MultiParamTypeClasses, QuasiQuotes #-}
import Yesod
import Data.Text
import Data.Text.Encoding
import Data.ByteString.UTF8
import Database.Redis
import qualified Data.ByteString.Lazy as L
data LRedis = LRedis
instance Yesod LRedis where
mkYesod "LRedis" [parseRoutes|
/ HomeR GET
/learnJson LearnJsonR GET
/redisWorks RedisWorksR GET
|]
getHomeR :: Handler RepHtml
getHomeR = do
defaultLayout[whamlet|
<p>Hi this is a headless API thing.
|]
getLearnJsonR :: Handler RepJson
getLearnJsonR = do
jsonToRepJson $ object [("json", ("ftw"::Text))]
getRedisWorksR :: Handler RepPlain
getRedisWorksR = do
conn <- liftIO $ connect defaultConnectInfo
liftIO $ runRedis conn $ do
result <- Database.Redis.get (fromString "hello")
case result of
Left e -> return $ RepPlain "Error"
Right mAnswer -> do
case mAnswer of
Nothing -> return $ RepPlain "Not found."
Just x -> return $ RepPlain (toContent x)
main :: IO()
main = do
warpDebug 3000 $ LRedis
Again, that is all working. It will return the string stored in "hello" in redis if you curl /redisWorks, or it will return JSON if you curl /learnJson, but I want to give the redis answer as JSON, not a plain string. I thought I could just naively combine the two, like:
getRedisJsonR :: Handler RepJson
getRedisJsonR = do
conn <- liftIO $ connect defaultConnectInfo
liftIO $ runRedis conn $ do
result <- Database.Redis.get (fromString "hello")
case result of
Left e -> jsonToRepJson $ object [("response", ("error"::Text))]
Right mAnswer -> do
case mAnswer of
Nothing -> jsonToRepJson $ object [("response", ("Nothing"::Text))]
Just x -> jsonToRepJson $ object [("response", ((decodeUtf8 x)::Text))]
But after adding the route /redisJson RedisJsonR GET it fails with this compilation error:
Couldn't match expected type `Redis a0'
with actual type `GHandler sub0 master0 RepJson'
In the expression:
jsonToRepJson $ object [("response", ("error" :: Text))]
In a case alternative:
Left e -> jsonToRepJson $ object [("response", ("error" :: Text))]
In a stmt of a 'do' block:
case result of {
Left e -> jsonToRepJson $ object [("response", ("error" :: Text))]
Right mAnswer
-> do { case mAnswer of {
Nothing -> ...
Just x -> ... } } }
It seems like its telling me I need to do something different with the result in case of an error, but I don't know what that would be, or why it's necessary given the RepPlain version is working.
Is there an example of getting the results from Redis into JSON within Yesod?
Is there just something simple I'm doing wrong with IO or something?
Convenient link to Hedis docs: http://hackage.haskell.org/package/hedis Thank you for helping me with this. Sorry again if it turns out to be super simple.

getRedisJsonR :: Handler RepJson
getRedisJsonR = do
conn <- liftIO $ connect defaultConnectInfo
res <- liftIO $ runRedis conn $ do
result <- Database.Redis.get (fromString "hello")
case result of
Left e -> return $ jsonToRepJson $ object [("response", ("error"::Text))]
Right mAnswer -> do
case mAnswer of
Nothing -> return $ jsonToRepJson $ object [("response", ("Nothing"::Text))]
Just x -> return $ jsonToRepJson $ object [("response", ((decodeUtf8 x)::Text))]
res
I actually don't trust myself to explain why this works and your original code doesn't -- I've been programming in Haskell on a daily basis for just under three months, so I have a developing gut sense for what will work but I'm really not there yet on the theory side, especially when it comes to stacked monads, which I think is what we're dealing with here (either Redis is on top of Handler or vice versa, and liftIO is facilitating the stacking).
Hopefully someone else can weigh in -- seems like a great concrete example to illustrate some monad concepts.

Related

Haskell Errors Binding a CSV File to a Handle

So I was writing a small toybox of utility functions for CSV files and throughout testing the functions I'd been binding the file by hand with
table' <- parseCSVFromFile filepath
but (from Text.CSV)
parseCSVFromFile :: FilePath -> IO (Either parsec-3.1.9:Text.Parsec.Error.ParseError CSV)
and so I had to write a quick line to strip off this either error csv crap
stripBare csv = head $ rights $ flip (:) [] csv
and redefine it as table = stripBare table'. After this, list functions work on the csv files contents and life goes on
(Digression: Surprisingly there isn't a direct Either a b -> b function in Data.Either. So I used Data.Either.rights :: [Either a b] -> [b])
I wanted to do the work of undressing the csv type and binding it to a handle in one shot. Something like
table = stripBare $ table' <- parseCSVFromFile filepath
but this gives a parse error on (<-) saying I may be missing a do... then
table = stripBare $ do table' <- parseCSVFromFile filepath
yells at me saying that the last statement in a do block must be an expression.
What am I doing wrong?
As a separate curiosity, I saw here that
do notation in Haskell desugars in a pretty simple way.
do
x <- foo
e1
e2
...
turns into
foo >>= \x ->
do
e1
e2
I find this attractive and tried the following line which gave me a type error
*Toy> :type (parseCSVFromFile "~/.csv") >>= \x -> x
<interactive>:1:52: error:
* Couldn't match type `Either
parsec-3.1.9:Text.Parsec.Error.ParseError'
with `IO'
Expected type: IO CSV
Actual type: Either parsec-3.1.9:Text.Parsec.Error.ParseError CSV
* In the expression: x
In the second argument of `(>>=)', namely `\ x -> x'
In the expression:
(parseCSVFromFile "~/.csv") >>= \ x -> x
Code such as
head $ rights $ flip (:) [] csv
is dangerous. You are exploiting the partiality of head to hide the fact that csv might be a Left something. We can rewrite it as
= head $ rights $ (:) csv []
= head $ rights [csv]
= case csv of
Left _ -> error "Left found!"
Right x -> x
Usually it's better to handle the Left case directly in the do IO block. Something like: (pseudo code follows)
foo :: String -> IO ()
foo filepath = do
table' <- parseCSVFromFile filepath
case table' of
Left err -> do
putStrLn "Error in parsing CSV"
print err
moreErrorHandlingHere
Right table -> do
putStrLn "CSV loaded!"
use table

How to use Data.Text.Lazy.IO to parse JSON files with Aeson

I want to parse all json files in a given directory into a data type Result.
So i have a decode function
decodeResult :: Data.ByteString.Lazy.ByteString -> Maybe Result
I began with Data.Text.Lazy.IO to load file into Lazy ByteString,
import qualified Data.Text.Lazy.IO as T
import qualified Data.Text.Lazy.Encoding as T
getFileContent :: FilePath -> IO B.ByteString
getFileContent path = T.encodeUtf8 `fmap` T.readFile path
It compiled, but I ran into Too many files opened problem, so I thought maybe I should use withFile.
import System.IO
import qualified Data.ByteString.Lazy as B
import qualified Data.Text.Lazy.IO as T
import qualified Data.Text.Lazy.Encoding as T
getFileContent :: FilePath -> IO (Maybe Result)
getFileContent path = withFile path ReadMode $ \hnd -> do
content <- T.hGetContents hnd
return $ (decodeAnalytic . T.encodeUtf8) content
loadAllResults :: FilePath -> IO [Result]
loadAllResults path = do
paths <- listDirectory path
results <- sequence $ fmap getFileContent (fmap (path ++ ) $ filter (endswith ".json") paths)
return $ catMaybes results
In this version, the lazy io seems never got evaluated, it always return empty list. But If i print content inside getFileContent function, then everything seems work correctly.
getFileContent :: FilePath -> IO (Maybe Result)
getFileContent path = withFile path ReadMode $ \hnd -> do
content <- T.hGetContents hnd
print content
return $ (decodeAnalytic . T.encodeUtf8) content
So I am not sure what am I missing, should I use conduit for this type of things?
Generally speaking I would recommend using a streaming library for parsing arbitrarily sized data like a JSON file. However, in the specific case of parsing JSON with aeson, the concerns of overrunning memory are not as significant IMO, since the aeson library itself will ultimately represent the entire file in memory as a Value type. So given that, you may choose to simply use strict bytestring I/O. I've given an example of using both conduit and strict I/O for parsing a JSON value. (I think the conduit version exists in some libraries already, I'm not sure.)
#!/usr/bin/env stack
{- stack --resolver lts-7.14 --install-ghc runghc
--package aeson --package conduit-extra
-}
import Control.Monad.Catch (MonadThrow, throwM)
import Control.Monad.IO.Class (MonadIO, liftIO)
import Data.Aeson (FromJSON, Result (..), eitherDecodeStrict',
fromJSON, json, Value)
import Data.ByteString (ByteString)
import qualified Data.ByteString as B
import Data.Conduit (ConduitM, runConduitRes, (.|))
import Data.Conduit.Attoparsec (sinkParser)
import Data.Conduit.Binary (sourceFile)
sinkFromJSON :: (MonadThrow m, FromJSON a) => ConduitM ByteString o m a
sinkFromJSON = do
value <- sinkParser json
case fromJSON value of
Error e -> throwM $ userError e
Success x -> return x
readJSONFile :: (MonadIO m, FromJSON a) => FilePath -> m a
readJSONFile fp = liftIO $ runConduitRes $ sourceFile fp .| sinkFromJSON
-- Or using strict I/O
readJSONFileStrict :: (MonadIO m, FromJSON a) => FilePath -> m a
readJSONFileStrict fp = liftIO $ do
bs <- B.readFile fp
case eitherDecodeStrict' bs of
Left e -> throwM $ userError e
Right x -> return x
main :: IO ()
main = do
x <- readJSONFile "test.json"
y <- readJSONFileStrict "test.json"
print (x :: Value)
print (y :: Value)
EDIT Forgot to mention: I'd strongly recommend against using textual I/O for reading your JSON files. JSON files should be encoded with UTF-8, while the textual I/O functions will use whatever your system settings specify for character encoding. Relying on Data.ByteString.readFile and similar is more reliable. I went into more detail in a recent blog post.

Catching exceptions in a pipe without terminating it

This feels like kind of a long shot, but I wrote a pipe to connect to a database, get a list of databases on the server, connect to each one, then perform a query on each one (a user count), then print those counts. Unfortunately this is about as much as I can simplify it from my real example. I'm using pipes-4.1.0, pipes-safe-2.0.2, and mysql-simple-0.2.2.4. Here is the code:
{-# LANGUAGE RankNTypes, OverloadedStrings #-}
import Pipes
import qualified Pipes.Safe as PS
import qualified Pipes.Prelude as P
import Database.MySQL.Simple
import qualified Data.Text as T
import Control.Monad.Catch as MC
import Control.Monad (forever)
import Database.MySQL.Simple.QueryParams
import Database.MySQL.Simple.QueryResults
data DBName = DBName T.Text deriving Show
-- connect to a database and use a table.
mydb :: T.Text -> ConnectInfo
mydb = undefined
-- Quirk of (mysql|postgresql)-simple libraries
unOnly (Only a) = a
queryProducer :: (MonadIO m, QueryParams params, QueryResults r) => Connection -> Query -> params -> Pipes.Producer' r m ()
queryProducer = undefined
myDBNames :: (PS.MonadSafe m, MonadIO m) => Producer DBName m ()
myDBNames = PS.bracket (liftIO $ connect $ mydb "sometable") (liftIO . close) $ \db ->
queryProducer db "show databases" () >-> P.map (DBName . unOnly)
-- I realize this is inefficient, one step at a time.
connectToDB :: (PS.MonadSafe m, MonadIO m) => Pipe DBName Connection m ()
connectToDB = forever $ do
(DBName dbname) <- await
PS.bracket
(liftIO . connect . mydb $ dbname)
(liftIO . close)
yield
userCount :: (PS.MonadCatch m, MonadIO m) => Pipe Connection Int m ()
userCount = forever $ do
db <- await
queryProducer db "select count(*) from user" () >-> P.map unOnly
main :: IO ()
main = PS.runSafeT $ runEffect $ myDBNames >-> P.tee P.print >-> connectToDB >-> userCount >-> P.print
This works fine. However, let's say in one of those databases, the user table is named users instead of user, therefore mysql-simple will throw an exception when that query is run. I want to catch that error inline, and just return 0 users for those queries, but keep going. Things I've tried:
(queryProducer db "select count(*) from user" () `PS.catchAll` (\e -> (liftIO $ putStrLn "failure") >> yield (Only 0))) >-> P.map unOnly
This doesn't work. Sometimes it will print failure and yield a 0, only to immediately terminate on exception. I thought maybe it is because I broke out of queryProducer with the exception, and I should call it again so I tried this recursive version:
thequery db >-> P.map unOnly
where
thequery db = queryProducer db "select count(*) from user" () `PS.catchAll` (\e -> (liftIO $ putStrLn "failure") >> yield (Only 0) >> thequery db)
But this also fails. However sometimes it will actually perform a several queries, printing out failure a few times and yielding a few 0's before terminating with an exception again. I'm really confused about why this is happening.
According to the async library, exceptions should be send up into the thread that the pipe is running in, so it doesn't seem like it could be a threading problem.
In case the implementation of my queryProducer matters, it is modeled after the pipes-postgresql query function, generalized to Producer' so I can embed it in other combinators. Below mysql-simple, in the mysql library there is a throw which throws a ConnectionError if your sql doesn't make sense, that percolates all the way up through this function.
{-# LANGUAGE RankNTypes #-}
import qualified Control.Concurrent.Async as Async
import qualified Control.Concurrent.STM as STM
import qualified Database.MySQL.Simple as My
import Database.MySQL.Simple.QueryParams
import Database.MySQL.Simple.QueryResults
import qualified Pipes
import qualified Pipes.Concurrent as Pipes
--------------------------------------------------------------------------------
-- | Convert a query to a 'Producer' of rows.
--
-- For example,
--
-- > pg <- connectToMysql
-- > query pg "SELECT * FROM widgets WHERE ID = ?" (Only widgetId) >-> print
--
-- Will select all widgets for a given #widgetId#, and then print each row to
-- standard output.
queryProducer
:: (MonadIO m, QueryResults r, QueryParams params)
=> My.Connection -> My.Query -> params -> Pipes.Producer' r m ()
queryProducer c q p = do
(o, i, seal) <- liftIO (Pipes.spawn' Pipes.Single)
worker <- liftIO $ Async.async $ do
My.fold c q p () (const $ void . STM.atomically . Pipes.send o)
STM.atomically seal
liftIO $ Async.link worker
Pipes.fromInput i
I also attempted to use EitherT to try and catch exceptions since that seems to be the way it was done in the past in pipes. But the documentation for that in pipes' tutorial disappeared between 3 and 4 making me wonder if that technique is still recommended or not. Unfortunately I could not get it to work because the way I am using queryProducer instead of singular await/yields, I'm not sure how to structure it.
Based on Gabe's comment, I fixed my queryProducer function by making sure the query cannot happen until the link function has fired.
query :: (MonadIO m, QueryResults r, QueryParams params) => My.Connection -> My.Query -> params -> Pipes.Producer' r m ()
query c q p = do
(o, i, seal) <- liftIO (Pipes.spawn' Pipes.Single)
mvar <- liftIO $ newEmptyMVar
worker <- liftIO $ Async.async $ do
takeMVar mvar
My.fold c q p () (const $ void . STM.atomically . Pipes.send o)
STM.atomically seal
liftIO $ Async.link worker
liftIO $ putMVar mvar ()
Pipes.fromInput i
I've tested this and it seems to work.

Exceptions not being caught by 'try'

I'm using hedis and trying to handle the case of the server going dead. According to the documentation:
Connection to the server lost:
In case of a lost connection, command functions throw a ConnectionLostException. It can only be caught outside of runRedis.
So I would assume I want to catch the ConnectionLostException. However, whilst I can seem to catch it correctly, it seems to also bubble up to the top, and I'm not sure why. Here's some code (just running in GHCI):
:set -XOverloadedStrings
import Database.Redis
import Control.Exception
conn <- connect defaultConnectInfo
runRedis conn $ ping
Now, if I kill the redis server between making the connection and running the command, I get the result I expect:
&langle;interactive&rangle;: ConnectionLost
*** Exception: ConnectionLost
So instead I try to do the following (I added >>= evaluate in order to attempt to force evaluation of the error, but it made no difference):
let tryR = try :: IO a -> IO (Either ConnectionLostException a)
tryR . (>>= evaluate) . runRedis conn $ ping
This gives me:
Left Con: ConnectionLost
nectionLost
So I'm getting the Left result as expected, but halfway through the exception is also presumably being caught and displayed by GHCI. Is this a problem with things not being evaluated?
Like John hinted, there seems to be somthing that prints this message to stderr.
Consider this example:
{-# LANGUAGE OverloadedStrings #-}
import Control.Concurrent (threadDelay)
import Control.Exception
import Control.Monad
import Database.Redis
import System.Mem
tryR :: IO a -> IO (Either ConnectionLostException a)
tryR = try
main :: IO ()
main = do
conn <- connect defaultConnectInfo
loop conn
putStrLn $ "exiting gracefully after counting up some numbers"
performGC
forM_ [1..10] $ \i -> do
print i
threadDelay 10000 -- 0.05 seconds
where
loop conn = do
e <- tryR . (>>= evaluate) . runRedis conn $ ping
case e of
Right x -> do print x
threadDelay 1000000
loop conn
Left err -> do putStrLn $ "tryR caught exception: " ++ show err
It prints:
Right Pong
Right Pong <-------------- after this I Ctrl-C the redis server
tryR caught exception: ConnectionLost
exiting gracefully after counting up some numbers
1
test: ConnectionLost
2
3
4
5
6
7
8
9
10
This looks like something in the stack is printing this test: ConnectionLost (or test.hs: ConnectionLost if you use GHCI/runghc) asynchronously.
If that's GHC, that is probably a bug, but chances are high it is done by hedis or one of its dependencies (I haven't found it in hedis itself yet).

haskell word searching program development

hello I am making some word searching program
for example
when "text.txt" file contains "foo foos foor fo.. foo fool"
and search "foo"
then only number 2 printed
and search again and again
but I am haskell beginner
my code is here
:module +Text.Regex.Posix
putStrLn "type text file"
filepath <- getLine
data <- readFile filepath
--1. this makes <interactive>:1:1: parse error on input `data' how to fix it?
parsedData =~ "[^- \".,\n]+" :: [[String]]
--2. I want to make function and call it again and again
searchingFunc = do putStrLn "search for ..."
search <- getLine
result <- map (\each -> if each == search then count = count + 1) data
putStrLn result
searchingFunc
}
sorry for very very poor code
my development environment is Windows XP SP3 WinGhci 1.0.2
I started the haskell several hours ago sorry
thank you very much for reading!
edit: here's original scheme code
thanks!
#lang scheme/gui
(define count 0)
(define (search str)
(set! count 0)
(map (λ (each) (when (equal? str each) (set! count (+ count 1)))) data)
(send msg set-label (format "~a Found" count)))
(define path (get-file))
(define port (open-input-file path))
(define data '())
(define (loop [line (read-line port)])
(when (not (eof-object? line))
(set! data (append data
(regexp-match* #rx"[^- \".,\n]+" line)))
(loop)))
(loop)
(define (cb-txt t e) (search (send t get-value)))
(define f (new frame% (label "text search") (min-width 300)))
(define txt (new text-field% (label "type here to search") (parent f) (callback (λ (t e) (cb-txt t e)))))
(define msg (new message% (label "0Found ") (parent f)))
(send f show #t)
I should start by iterating what everyone would (and should) say: Start with a book like Real World Haskell! That said, I'll post a quick walkthrough of code that compiles, and hopefully does something close to what you originally intended. Comments are inline, and hopefully should illustrate some of the shortcomings of your approach.
import Text.Regex.Posix
-- Let's start by wrapping your first attempt into a 'Monadic Action'
-- IO is a monad, and hence we can sequence 'actions' (read as: functions)
-- together using do-notation.
attemptOne :: IO [[String]]
-- ^ type declaration of the function 'attemptOne'
-- read as: function returning value having type 'IO [[String]]'
attemptOne = do
putStrLn "type text file"
filePath <- getLine
fileData <- readFile filePath
putStrLn fileData
let parsed = fileData =~ "[^- \".,\n]+" :: [[String]]
-- ^ this form of let syntax allows us to declare that
-- 'wherever there is a use of the left-hand-side, we can
-- substitute it for the right-hand-side and get equivalent
-- results.
putStrLn ("The data after running the regex: " ++ concatMap concat parsed)
return parsed
-- ^ return is a monadic action that 'lifts' a value
-- into the encapsulating monad (in this case, the 'IO' Monad).
-- Here we show that given a search term (a String), and a body of text to
-- search in, we can return the frequency of occurrence of the term within the
-- text.
searchingFunc :: String -> [String] -> Int
searchingFunc term
= length . filter predicate
where
predicate = (==)term
-- ^ we use function composition (.) to create a new function from two
-- existing ones:
-- filter (drop any elements of a list that don't satisfy
-- our predicate)
-- length: return the size of the list
-- Here we build a wrapper-function that allows us to run our 'pure'
-- searchingFunc on an input of the form returned by 'attemptOne'.
runSearchingFunc :: String -> [[String]] -> [Int]
runSearchingFunc term parsedData
= map (searchingFunc term) parsedData
-- Here's an example of piecing everything together with IO actions
main :: IO ()
main = do
results <- attemptOne
-- ^ run our attemptOne function (representing IO actions)
-- and save the result
let searchResults = runSearchingFunc "foo" results
-- ^ us a 'let' binding to state that searchResults is
-- equivalent to running 'runSearchingFunc'
print searchResults
-- ^ run the IO action that prints searchResults
print (runSearchingFunc "foo" results)
-- ^ run the IO action that prints the 'definition'
-- of 'searchResults'; i.e. the above two IO actions
-- are equivalent.
return ()
-- as before, lift a value into the encapsulating Monad;
-- this time, we're lifting a value corresponding to 'null/void'.
To load this code, save it into a .hs file (I saved it into 'temp.hs'), and run the following from ghci. Note: the file 'f' contains a few input words:
*Main Text.Regex.Posix> :l temp.hs
[1 of 1] Compiling Main ( temp.hs, interpreted )
Ok, modules loaded: Main.
*Main Text.Regex.Posix> main
type text file
f
foo foos foor fo foo foo
The data after running the regex: foofoosfoorfofoofoo
[1,0,0,0,1,1]
[1,0,0,0,1,1]
There is a lot going on here, from do notation to Monadic actions, 'let' bindings to the distinction between pure and impure functions/values. I can't stress the value of learning the fundamentals from a good book!
Here is what I made of it. It doesn't does any error checking and is as basic as possible.
import Text.Regex.Posix ((=~))
import Control.Monad (when)
import Text.Printf (printf)
-- Calculates the number of matching words
matchWord :: String -> String -> Int
matchWord file word = length . filter (== word) . concat $ file =~ "[^- \".,\n]+"
getInputFile :: IO String
getInputFile = do putStrLn "Enter the file to search through:"
path <- getLine
readFile path -- Attention! No error checking here
repl :: String -> IO ()
repl file = do putStrLn "Enter word to search for (empty for exit):"
word <- getLine
when (word /= "") $
do print $ matchWord file word
repl file
main :: IO ()
main = do file <- getInputFile
repl file
Please start step by step. IO in Haskell is hard, so you shouldn't start with file manipulation. I would suggest to write a function that works properly on a given String. That way you can learn about syntax, pattern matching, list manipulation (maps, folds) and recursion without beeing distracted by the do notation (which kinda looks imperative, but isn't, and really needs a deeper understanding).
You should check out Learn you a Haskell or Real World Haskell to get a sound foundation. What you do now is just stumbling in the dark - which may work if you learn languages that are similar to the ones you know, but definitely not for Haskell.