How to extract data using Text.JSON? - json

I'm trying to write a wrapper around ffprobe that extracts value in JSON of the format {"format": {"format_name": value}}. The JSON is output by a created process. Here's what I've gotten to.
import System.Process
import System.Environment
import System.IO
import Text.JSON
main = do
args <- getArgs
(_, Just out, _, p) <- createProcess
(proc "ffprobe" [args!!0, "-of", "json", "-show_format"])
{ std_out = CreatePipe }
s <- hGetContents out
--putStrLn $ show (decode s :: Result JSValue)
--waitForProcess p
--putStrLn $ valFromObj "format_name" format
-- where format = valFromObj "format" rootObj
-- (Ok rootObj) = decode s :: Result (JSObject (JSValue))
let (Ok rootObj) = decode s :: Result (JSObject (JSValue))
let (Ok format) = valFromObj "format" rootObj :: Result (JSObject (JSValue))
putStrLn format_name
where (Ok format_name) = valFromObj "format_name" format
It fails to compile with:
[1 of 1] Compiling Main ( ffprobe.hs, ffprobe.o )
ffprobe.hs:20:59: error:
Variable not in scope: format :: JSObject JSValue
I'm confused about several things, including why I can't get the last line to compile:
Why can't I assert for Ok in the Result after the ::. Like :: Result Ok JSObject JSValue?
Why can't I extract the values in a where clause?
Why is it Result (JSObject (JSValue)) and not Result JSObject JSValue?
Why is format out of scope?
I have a feeling I'm mixing the IO and Result monads together in the same do block or something. Is Result even a monad? Can I extract the value I want in a separate do without crapping all over the IO do?

I think your compile error is because of the position of the where. Try
main = do
...
let (Ok format) = valFromObj "format" rootObj :: Result (JSObject (JSValue))
let (Ok format_name) = valFromObj "format_name" format
putStrLn format_name
The scope of the where is outside the do so it isn't aware of format.

You cannot do this:
main = do
let bar = "only visible inside main? "
return baz
where
baz = bar ++ " yes, this will break!"
This gives:
test.hs:7:11:
Not in scope: ‘bar’
Perhaps you meant ‘baz’ (line 7)
Let bindings unlike function arguments are not available in where bindings. Above bar is not in scope for baz to use it. Compare to your code.

Related

Haskell - how do I convert piped-in JSON-based to a data record?

I'm building a reinforcement learning library where I'd like to pass certain instance information into the executables via a piped JSON.
Using aeson's Simplest.hs, I'm able to get the following basic example working as intended. Note that the parameters are sitting in Main.hs as a String params as a placeholder.
I tried to modify Main.hs so I would pipe the Nim game parameters in from a JSON file via getContents, but am running into the expected [Char] vs. IO String issue. I've tried to read up as much as possible about IO, but can't figure out how to lift my JSON parsing method to deal with IO.
How would I modify the below so that I can work with piped-in JSON?
Main.hs
module Main where
import qualified System.Random as Random
import qualified Data.ByteString.Lazy.Char8 as BL
import qualified Games.Engine as Engine
import qualified Games.IO.Nim as NimIO
import qualified Games.Rules.Nim as Nim
import qualified Games.Learn.ValueIteration as VI
main :: IO ()
main = do
let params = "{\"players\":[\"Bob\", \"Alice\", \"Charlie\"], \"initialPiles\": [3, 4, 5], \"isMisere\": false}"
let result = NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
case result of
Nothing -> putStrLn "Parameter errors."
Just game -> do
putStrLn "Let's play some Nim! Remainder of code omitted"
Games.IO.Nim.hs
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-}
module Games.IO.Nim
( decode
, NimGame
, players
, initialPiles
, isMisere
) where
import Control.Applicative (empty)
import qualified Data.ByteString.Lazy.Char8 as BL
import Data.Aeson
( pairs,
(.:),
object,
FromJSON(parseJSON),
Value(Object),
KeyValue((.=)),
ToJSON(toJSON, toEncoding),
decode)
data NimGame = NimGame
{ players :: [String]
, initialPiles :: [Int]
, isMisere :: Bool
} deriving (Show)
instance ToJSON NimGame where
toJSON (NimGame playersV initialPilesV isMisereV) = object [ "players" .= playersV,
"initialPiles" .= initialPilesV,
"isMisere" .= isMisereV]
toEncoding NimGame{..} = pairs $
"players" .= players <>
"initialPiles" .= initialPiles <>
"isMisere" .= isMisere
instance FromJSON NimGame where
parseJSON (Object v) = NimGame <$>
v .: "players" <*>
v .: "initialPiles" <*>
v .: "isMisere"
parseJSON _ = empty
Alternative Main.hs that generates compile error
module Main where
import qualified System.Random as Random
import qualified Data.ByteString.Lazy.Char8 as BL
import qualified Games.Engine as Engine
import qualified Games.IO.Nim as NimIO
import qualified Games.Rules.Nim as Nim
import qualified Games.Learn.ValueIteration as VI
main :: IO ()
main = do
--let params = "{\"players\":[\"Bob\", \"Alice\", \"Charlie\"], \"initialPiles\": [3, 4, 5], \"isMisere\": false}"
let params = getContents
let result = NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
case result of
Nothing -> putStrLn "Parameter errors."
Just game -> do
putStrLn "Let's play some Nim!"
Compile Error
(base) randm#pearljam ~/Projects/gameshs $ stack build
gameshs-0.1.0.0: unregistering (local file changes: app/Nim.hs)
gameshs> configure (lib + exe)
Configuring gameshs-0.1.0.0...
gameshs> build (lib + exe)
Preprocessing library for gameshs-0.1.0.0..
Building library for gameshs-0.1.0.0..
Preprocessing executable 'nim-exe' for gameshs-0.1.0.0..
Building executable 'nim-exe' for gameshs-0.1.0.0..
[2 of 2] Compiling Main
/home/randm/Projects/gameshs/app/Nim.hs:17:41: error:
• Couldn't match expected type ‘[Char]’
with actual type ‘IO String’
• In the first argument of ‘BL.pack’, namely ‘params’
In the second argument of ‘($)’, namely ‘BL.pack params’
In the expression:
NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
|
17 | let result = NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
| ^^^^^^
-- While building package gameshs-0.1.0.0 (scroll up to its section to see the error) using:
/home/randm/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_3.2.1.0_ghc-8.10.4 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.2.1.0 build lib:gameshs exe:nim-exe --ghc-options " -fdiagnostics-color=always"
Process exited with code: ExitFailure 1
getContents returns not a String as you apparently expect, but IO String, which is a "program", which, when executed, will produce a String. So when you're trying to parse this program with decode, of course that doesn't work: decode parses a String, it cannot parse a program.
So how do you execute this program to obtain the String? There are two ways: either you make it part of another program or you call it main and it becomes your entry point.
In your case, the sensible thing to do would be to make getContent part of your main program. To do that, use the left arrow <-, like this:
main = do
params <- getContents
let result = NimIO.decode $ BL.pack params :: Maybe NimIO.NimGame
...

Optimize lens based JSON handling

In my current "learning haskell" project I try to fetch weather data from a third party api. I want to extract the name and main.temp value from the following response body:
{
...
"main": {
"temp": 280.32,
...
},
...
"name": "London",
...
}
I wrote a getWeather service to perform IO and transform the response to construct GetCityWeather data:
....
data WeatherService = GetCityWeather String Double
deriving (Show)
....
getWeather :: IO (ServiceResult WeatherService)
getWeather = do
...
response <- httpLbs request manager
...
-- work thru the response
return $ case ((maybeCityName response, maybeTemp response)) of
(Just name, Just temp) -> success name temp
bork -> err ("borked data >:( " ++ show bork))
where
showStatus r = show $ statusCode $ responseStatus r
maybeCityName r = (responseBody r)^?key "name"._String
maybeTemp r = (responseBody r)^?key "main".key "temp"._Double
success n t = Right (GetCityWeather (T.unpack n) t)
err e = Left (SimpleServiceError e)
I stuck optimizing the JSON parsing part in maybeCityName, and maybeTemp, my thoughts are:
Currently the JSON is parsed twice (I apply ^? two times on the raw response responseBody r).
I would like to get the data in "one shot". ?.. is able to get a list of values. But I extract different types (String, Double) so the ?.. does not fit here.
I'm looking for more elegant / more natural ways to safely parse JSON, read desired the values and apply them to the data constructor GetCityWeather. Thanks in advance for any help and feedback.
Update: using Folds I am able to solve the problem with two case matches
getWeather :: IO (ServiceResult WeatherService)
getWeather = do
...
let value = decode $ responseBody response
return $ case value of
Just v -> case (v ^? weatherService) of
Just wr -> Right wr
Nothing -> err "incompatible data"
Nothing -> err "bad json"
where
err t = Left (SimpleServiceError t)
weatherService :: Fold Value WeatherService
weatherService = runFold $ GetCityWeather
<$> Fold (key "name" . _String . unpacked)
<*> Fold (key "main" . key "temp" . _Double)
As #jpath point out, the real problem you have here is one about lens and JSON handling. The crux of the issue seems to be that you want to do the lens operation all at once. For that, check out the handy ReifiedFold: the "parallel" functionality you want is packed into the Applicative instance.
import Control.Lens
import Data.Aeson
import Data.Aeson.Lens
import Data.Text.Lens ( unpacked )
-- | Extract a `WeatherService` from a `Value` if possible
weatherService :: Fold Value WeatherService
weatherService = runFold $ GetCityWeather
<$> Fold (key "name" . _String . unpacked)
<*> Fold (key "main" . key "temp" . _Double))
Then, you can try to get your WeatherService all at once:
...
-- work thru the response
let body = responseBody r
return $ case body ^? weatherService of
Just wr -> Right wr
Nothing -> Left (SimpleServiceError ("borked data >:( " ++ show body))
However, for the sake of error messages, it might be a better idea to take advantage of aeson's ToJSON/FromJSON if you plan on scaling this more.

Reading first row from a csv file with pipes-csv

I am reading a csv file with pipes-csv library. I want to read first line and read the rest later. Unfortunately after Pipes.Prelude.head function returns. pipe is being closed somehow. Is there a way to read head of the csv first and read the rest later.
import qualified Data.Vector as V
import Pipes
import qualified Pipes.Prelude as P
import qualified System.IO as IO
import qualified Pipes.ByteString as PB
import qualified Data.Text as Text
import qualified Pipes.Csv as PCsv
import Control.Monad (forever)
showPipe :: Proxy () (Either String (V.Vector Text.Text)) () String IO b
showPipe = forever $ do
x::(Either String (V.Vector Text.Text)) <- await
yield $ show x
main :: IO ()
main = do
IO.withFile "./test.csv"
IO.ReadMode
(\handle -> do
let producer = (PCsv.decode PCsv.NoHeader (PB.fromHandle handle))
headers <- P.head producer
putStrLn "Header"
putStrLn $ show headers
putStrLn $ "Rows"
runEffect ( producer>->
(showPipe) >->
P.stdoutLn)
)
If we do not read the header first, we can read whole csv without any problem:
main :: IO ()
main = do
IO.withFile "./test.csv"
IO.ReadMode
(\handle -> do
let producer = (PCsv.decode PCsv.NoHeader (PB.fromHandle handle))
putStrLn $ "Rows"
runEffect ( producer>->
(showPipe) >->
P.stdoutLn)
)
Pipes.Csv has material for handling headers, but I think that this question is really looking for a more sophisticated use of Pipes.await or else Pipes.next. First next:
>>> :t Pipes.next
Pipes.next :: Monad m => Producer a m r -> m (Either r (a, Producer a m r))
next is the basic way of inspecting a producer. It is sort of like pattern matching on a list. With a list the two possibilities are [] and x:xs - here they are Left () and Right (headers, rows). The latter pair is what you are looking for. Of course an action (here in IO) is needed to get one's hands on it:
main :: IO ()
main = do
handle <- IO.openFile "./test.csv" IO.ReadMode
let producer :: Producer (V.Vector Text.Text) IO ()
producer = PCsv.decode PCsv.NoHeader (PB.fromHandle handle) >-> P.concat
e <- next producer
case e of
Left () -> putStrLn "No lines!"
Right (headers, rows) -> do
putStrLn "Header"
print headers
putStrLn $ "Rows"
runEffect ( rows >-> P.print)
IO.hClose handle
Since the Either values are distraction here, I eliminate Left values - the lines that don't parse - with P.concat
next does not act inside a pipeline, but directly on the Producer, which it treats as a sort of "effectful list" with a final return value at the end. The particular effect we got above can of course be achieved with await, which acts inside a pipeline. I can use it to intercept the first item that comes along in a pipeline, do some IO based on it, and then forward the remaining elements:
main :: IO ()
main = do
handle <- IO.openFile "./grades.csv" IO.ReadMode
let producer :: Producer (V.Vector Text.Text) IO ()
producer = PCsv.decode PCsv.NoHeader (PB.fromHandle handle) >-> P.concat
handleHeader :: Pipe (V.Vector Text.Text) (V.Vector Text.Text) IO ()
handleHeader = do
headers <- await -- intercept first value
liftIO $ do -- use it for IO
putStrLn "Header"
print headers
putStrLn $ "Rows"
cat -- pass along all later values
runEffect (producer >-> handleHeader >-> P.print)
IO.hClose handle
The difference is just that if producer is empty, I won't be able to declare this, as I do with No lines! in the previous program.
Note by the way that showPipe can be defined as P.map show, or simply as P.show (but with the specialized type you add.)

Getting a collection of values from a JSON ByteString using lens-aeson

Say I have a JSON ByteString that looks something like
{
messages: [
{...},
{...}
]
}
I'd like to use lens to get a list/vector of messages out of it. I have a function toMessage that can turn a Value into a Maybe Message.
I've tried this composition key "messages" . values . to toMessage (to is from Control.Lens.Getter but the result is Maybe Message and it simply becomes Nothing.
Currently I'm doing this
msgsJson <- c ^? key "messages"
let msgs = toList $ mapMaybe message $ msgsJson ^.. values
(mapMaybe is from witherable, toList is to convert the Vector into a list)
but I'd like to know if there's a way to compose various lenses to get a single lens that does this.
Hmm, this works for me:
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Data.ByteString (ByteString)
import Control.Lens
import Data.Aeson
import Data.Aeson.Lens
newtype Message =
Message Integer
deriving (Show)
toMessage :: Value -> Maybe Message
toMessage json_ = do
i <- json_ ^? key "a" . _Integer
return (Message i)
input :: ByteString
input = "{\"messages\":[{\"a\":1},{\"a\":2},{\"a\":3}]}"
main :: IO ()
main =
print (input ^.. (key "messages" . values . to toMessage))
λ> main
[Just (Message 1),Just (Message 2),Just (Message 3)]
If you're getting Nothing, it could mean that your JSON is invalid (you can test it with decode json :: Maybe Value). Or that any other optic in the composition is failing. With long dotted optics it's sometimes hard to tell exactly which one is failing except by lopping parts off the end and retrying. But your composed optic key "messages" . values . to toMessage should Just Work™.
By the way:
msgJson <- c ^? key "messages"
let msgs = toList $ mapMaybe message $ msgsJson ^.. values
should be the same as
msgJson <- c ^? key "messages"
let msgs = toList $ msgsJson ^.. (values . to message . _Just)
and
msgJson <- c ^? key "messages"
let msgs = msgsJson ^.. (values . to message . _Just)
and
let msgs = c ^.. (key "messages" . values . to message . _Just)

Example of how to parse exiftool JSON output in Haskell

I can't make sense of any of the documentation. Can someone please provide an example of how I can parse the following shortened exiftool output using the Haskell module Text.JSON? The data is generating using the command exiftool -G -j <files.jpg>.
[{
"SourceFile": "DSC00690.JPG",
"ExifTool:ExifToolVersion": 7.82,
"File:FileName": "DSC00690.JPG",
"Composite:LightValue": 11.6
},
{
"SourceFile": "DSC00693.JPG",
"ExifTool:ExifToolVersion": 7.82,
"File:FileName": "DSC00693.JPG",
"EXIF:Compression": "JPEG (old-style)",
"EXIF:ThumbnailLength": 4817,
"Composite:LightValue": 13.0
},
{
"SourceFile": "DSC00694.JPG",
"ExifTool:ExifToolVersion": 7.82,
"File:FileName": "DSC00694.JPG",
"Composite:LightValue": 3.7
}]
Well, the easiest way is to get back a JSValue from the json package, like so (assuming your data is in text.json):
Prelude Text.JSON> s <- readFile "test.json"
Prelude Text.JSON> decode s :: Result JSValue
Ok (JSArray [JSObject (JSONObject {fromJSObject = [("SourceFile",JSString (JSONString {fromJSString = "DSC00690.JPG"})),("ExifTool:ExifToolVersion",JSRational False (391 % 50)),("File:FileName",JSString (JSONString {fromJSString = "DSC00690.JPG"})),("Composite:LightValue",JSRational False (58 % 5))]}),JSObject (JSONObject {fromJSObject = [("SourceFile",JSString (JSONString {fromJSString = "DSC00693.JPG"})),("ExifTool:ExifToolVersion",JSRational False (391 % 50)),("File:FileName",JSString (JSONString {fromJSString = "DSC00693.JPG"})),("EXIF:Compression",JSString (JSONString {fromJSString = "JPEG (old-style)"})),("EXIF:ThumbnailLength",JSRational False (4817 % 1)),("Composite:LightValue",JSRational False (13 % 1))]}),JSObject (JSONObject {fromJSObject = [("SourceFile",JSString (JSONString {fromJSString = "DSC00694.JPG"})),("ExifTool:ExifToolVersion",JSRational False (391 % 50)),("File:FileName",JSString (JSONString {fromJSString = "DSC00694.JPG"})),("Composite:LightValue",JSRational False (37 % 10))]})])
this just gives you a generic json Haskell data type.
The next step will be to define a custom Haskell data type for your data, and write an instance of JSON for that, that converts between JSValue's as above, and your type.
Thanks to all. From your suggestions I was able to put together the following which translates the JSON back into name-value pairs.
data Exif =
Exif [(String, String)]
deriving (Eq, Ord, Show)
instance JSON Exif where
showJSON (Exif xs) = showJSONs xs
readJSON (JSObject obj) = Ok $ Exif [(n, s v) | (n, JSString v) <- o]
where
o = fromJSObject obj
s = fromJSString
Unfortunately, it seems the library is unable to translate the JSON straight back into a simple Haskell data structure. In Python, it is a one-liner: json.loads(s).