Working my way through Haskell and I'm trying to learn how to serialized to/from JSON.
I'm using aeson-0.8.0.2 & I'm stuck at basic decoding. Here's what I have:
file playground/aeson.hs:
{-# LANGUAGE OverloadedStrings #-}
import Data.Text
import Data.Aeson
data Person = Person
{ name :: Text
, age :: Int
} deriving Show
instance FromJSON Person where
parseJSON (Object v) = Person <$>
v .: "name" <*>
v .: "age"
parseJSON _ = mzero
main = do
let a = decode "{\"name\":\"Joe\",\"age\":12}" :: Maybe Person
print "aa"
ghc --make playground/aeson.hs yields:
[1 of 1] Compiling Main ( playground/aeson.hs,
playground/aeson.o )
playground/aeson.hs:13:35: Not in scope: `'
playground/aeson.hs:14:40: Not in scope: `<*>'
playground/aeson.hs:17:28: Not in scope: `mzero'
Any idea what I'm doing wrong?
Why is OverloadedString needed here?
Also, I have no idea what <$>, <*>, or mzero are supposed to mean; I'd appreciate tips on where I can read about any of these.
You need to import Control.Applicative and Control.Monad to get <$>, <*> and mzero. <$> just an infix operator for fmap, and <*> is the Applicative operator, you can think of it as a more generalized form of fmap for now. mzero is defined for the MonadPlus class, which is a class representing that a Monad has the operation
mplus :: m a -> m a -> m a
And a "monadic zero" element called mzero. The simplest example is for lists:
> mzero :: [Int]
[]
> [1, 2, 3] `mplus` [4, 5, 6]
[1, 2, 3, 4, 5, 6]
Here mzero is being used to represent a failure to parse. For looking up symbols in the future, I recommend using hoogle or FP Complete's version. Once you find the symbol, read the documentation, the source, and look around for examples of its use on the internet. You'll learn a lot by looking for it yourself, although it'll take you a little while to get used to this kind of research.
The OverloadedStrings extension is needed here because the Aeson library works with the Text type from Data.Text instead of the built-in String type. This extension lets you use string literals as Text instead of String, just as the numeric literal 0 can be an Int, Integer, Float, Double, Complex and other types. OverloadedStrings makes string literals have type Text.String.IsString s => s instead of just String, so it makes it easy to use alternate string-y types.
For <$> you need to import Control.Applicative and for mzero you need to import Control.Monad.
You can determine this by using the web-based version of hoogle (http://www.haskell.org/hoogle) or the command line version:
$ hoogle '<$>'
$ hoogle mzero
Related
I have this weird JSON to parse containing nested JSON ... a string. So instead of
{\"title\": \"Lord of the rings\", \"author\": {\"666\": \"Tolkien\"}\"}"
I have
{\"title\": \"Lord of the rings\", \"author\": \"{\\\"666\\\": \\\"Tolkien\\\"}\"}"
Here's my (failed) attempt to parse the nested string using decode, inside an instance of FromJSON :
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Data.Maybe
import GHC.Generics
import Data.Aeson
import qualified Data.Map as M
type Authors = M.Map Int String
data Book = Book
{
title :: String,
author :: Authors
}
deriving (Show, Generic)
decodeAuthors x = fromJust (decode x :: Maybe Authors)
instance FromJSON Book where
parseJSON = withObject "Book" $ \v -> do
t <- v .: "title"
a <- decodeAuthors <?> v .: "author"
return $ Book t a
jsonTest = "{\"title\": \"Lord of the rings\", \"author\": \"{\\\"666\\\": \\\"Tolkien\\\"}\"}"
test = decode jsonTest :: Maybe Book
Is there a way to decode the whole JSON in a single pass ? Thanks !
A couple problems here.
First, your use of <?> is nonsensical. I'm going to assume it's a typo, and what you actually meant was <$>.
Second, the type of decodeAuthors is ByteString -> Authors, which means its parameter is of type ByteString, which means that the expression v .: "author" must be of type Parser ByteString, which means that there must be an instance FromJSON ByteString, but such instance doesn't exists (for reasons that escape me at the moment).
What you actually want is for v .: "author" to return a Parser String (or perhaps Parser Text), and then have decodeAuthors accept a String and convert it to ByteString (using pack) before passing to decode:
import Data.ByteString.Lazy.Char8 (pack)
decodeAuthors :: String -> Authors
decodeAuthors x = fromJust (decode (pack x) :: Maybe Authors)
(also note: it's a good idea to give you declarations type signatures that you think they should have. This lets the compiler point out errors earlier)
Edit:
As #DanielWagner correctly points out, pack may garble Unicode text. If you want to handle it correctly, use Data.ByteString.Lazy.UTF8.fromString from utf8-string to do the conversion:
import Data.ByteString.Lazy.UTF8 (fromString)
decodeAuthors :: String -> Authors
decodeAuthors x = fromJust (decode (fromString x) :: Maybe Authors)
But in that case you should also be careful about the type of jsonTest: the way your code is written, its type would be ByteString, but any non-ASCII characters that may be inside would be cut off because of the way IsString works. To preserve them, you need to use the same fromString on it:
jsonTest = fromString "{\"title\": \"Lord of the rings\", \"author\": \"{\\\"666\\\": \\\"Tolkien\\\"}\"}"
How can I set up cassava to ignore missing columns/fields and fill the respective data type with a default value? Consider this example:
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
import Data.ByteString.Lazy.Char8
import Data.Csv
import Data.Vector
import GHC.Generics
data Foo = Foo {
a :: String
, b :: Int
} deriving (Eq, Show, Generic)
instance FromNamedRecord Foo
decodeAndPrint :: ByteString -> IO ()
decodeAndPrint csv = do
print $ (decodeByName csv :: Either String (Header, Vector Foo))
main :: IO ()
main = do
decodeAndPrint "a,b,ignore\nhu,1,pu" -- [1]
decodeAndPrint "ignore,b,a\npu,1,hu" -- [2]
decodeAndPrint "ignore,b\npu,1" -- [3]
[1] and [2] work perfectly fine, but [3] fails with
Left "parse error (Failed reading: conversion error: no field named \"a\") at \"\""
How could I make decodeAndPrint capable of handling this incomplete input?
I could of course manipulate the input bytestring, but maybe there is a more elegant solution.
A Solution thanks to the input of Daniel Wagner below:
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
import Control.Applicative
import Data.ByteString.Lazy.Char8
import Data.Csv
import Data.Vector
import GHC.Generics
data Foo = Foo {
a :: Maybe String
, b :: Maybe Int
} deriving (Eq, Show, Generic)
instance FromNamedRecord Foo where
parseNamedRecord rec = pure Foo
<*> ((Just <$> Data.Csv.lookup rec "a") <|> pure Nothing)
<*> ((Just <$> Data.Csv.lookup rec "b") <|> pure Nothing)
decodeAndPrint :: ByteString -> IO ()
decodeAndPrint csv = do
print $ (decodeByName csv :: Either String (Header, Vector Foo))
main :: IO ()
main = do
decodeAndPrint "a,b,ignore\nhu,1,pu" -- [1]
decodeAndPrint "ignore,b,a\npu,1,hu" -- [2]
decodeAndPrint "ignore,b\npu,1" -- [3]
(Warning: completely untested! Code is for idea transmission only, not suitable for any use, etc. etc.)
The Parser type demanded by FromNamedRecord is an Alternative, so just toss a default on with (<|>).
instance FromNamedRecord Foo where
parseNamedRecord rec = pure Foo
<*> (lookup rec "a" <|> pure "missing")
<*> (lookup rec "b" <|> pure 0)
If you want to know later whether the field was there or not, make your fields rich enough to record that:
data RichFoo = RichFoo
{ a :: Maybe String
, b :: Maybe Int
}
instance FromNamedRecord Foo where
parseNamedRecord rec = pure RichFoo
<*> ((Just <$> lookup rec "a") <|> pure Nothing)
<*> ((Just <$> lookup rec "b") <|> pure Nothing)
I'm relatively new to haskell and right now I'm trying to get a deeper understanding and trying to get used to different popular libraries.
Right now I'm trying "aeson".
What I want to do is parse MSFT quote request from https://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=MSFT&apikey=demo
This is what it looks like
{
"Global Quote": {
"01. symbol": "MSFT",
"02. open": "105.3500",
"03. high": "108.2400",
"04. low": "105.2700",
"05. price": "107.6000",
"06. volume": "23308066",
"07. latest trading day": "2018-10-11",
"08. previous close": "106.1600",
"09. change": "1.4400",
"10. change percent": "1.3564%"
}
}
This is what I've got so far
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import qualified Data.ByteString.Lazy as B
import GHC.Exts
import GHC.Generics
import Network.HTTP
import Network.URI
jsonURL :: String
jsonURL = "http://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=MSFT&apikey=demo"
getRequest_ :: HStream ty => String -> Request ty
getRequest_ s = let Just u = parseURI s in defaultGETRequest_ u
jsonReq = getRequest_ jsonURL
data Quote = Quote {quote :: String,
symbol :: String,
open :: Float,
high :: Float,
low :: Float,
price :: Float,
volume :: Float,
ltd :: String,
previousClose :: Float,
change :: Float,
changePerct :: Float
} deriving (Show, Generic)
instance FromJSON Quote
instance ToJSON Quote
main :: IO ()
main = do
d <- simpleHTTP jsonReq
body <- getResponseBody d
print (decode body :: Maybe Quote)
What am I doing wrong?
Edit: Fixed version in the answers.
First off: Aeson is not the easiest library for a beginner. There are more difficult ones, sure, but it supposes you already a fair number of things about the language. You didn't pick the "simplest task" to begin with. I know this can be surprising, and you might think that parsing JSON should be simple, but parsing JSON with strong type guarantees is actually not that simple.
But here's what I can tell you to help you a bit:
First, use eitherDecode rather than decode: you will get an error message rather than simply Nothing, which will help you a bit.
Deriving through Generic is neat and very often, a time saver, but it's not magic either. The name of the object key and the name of your datatype fields have to match exactly. Sadly, this is not the case here and due to haskell syntax, you couldn't name your fields like the keys of the object. Your best solution is to implement FromJSON manually (see the recommended link below). A good way to see "what is expect" by the generic FromJSON is to also derive ToJSON, create a dummy Quote and see the result of encode.
Your first field (quote) is not a key of the object itself, but rather the name of this object. So you have dynamic keys ("Global Quote" being one here). Once again, this typically a case where you want to write the FromJSON instance manually.
I recommend you read this famous tutorial written by Artyom Kazak on Aeson. This will help you tremendously and is probably the best advice I can give.
For your manual instance, supposing it was exactly the document you want to parse and you had only the "Global Quote" to deal with, it would look more or less like this:
instance ToJSON Quote where
parseJSON = withObject "Document" $
\d -> do
glob <- d .: "Global Quote"
withObject "Quote" v (\gq ->
Quote <$> gq .: "01. symbol"
<*> pure "Global Quote"
<*> gq .: "02. open"
<*> gq .: "03. high"
-- ... and so on
) v
(It's not the most pretty way, nor the best way, to write it, but it should be one possible way).
Also note that, as an astute commenter wrote, the types of your fields are not always aligned with the type of your example JSON document. "volume" is an Int (byte-limited int), potentially an Integer ("mathematical" integer, no bound), but not a Float. Your "ltd" can be parsed a string - but it probably should be a date (Day from Data.Time would be the first choice - it already has a FromJSON instance so chances are it should be parseable as is). Change percent is most likely not parseable as Float like that, you'll need to write a dedicated parser for this type (and decide how you want to store it - Ratio is a potential solution).
#Raveline with their answer above pointed me to the right direction. I was able to solve all those issues, here's the final product!
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-}
module Test where
import Data.Aeson
import qualified Data.ByteString.Lazy as B
import GHC.Exts
import GHC.Generics
import Network.HTTP.Conduit (simpleHttp)
jsonURL :: String
jsonURL = "https://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=MSFT&apikey=demo"
getJSON :: IO B.ByteString
getJSON = simpleHttp jsonURL
data Quote = Quote {
symbol :: String,
open :: String,
high :: String,
low :: String,
price :: String,
volume :: String,
ltd :: String,
previousClose :: String,
change :: String,
changePercent :: String
} deriving (Show, Generic)
instance FromJSON Quote where
parseJSON = withObject "Global Quote" $
\o -> do
globalQuote <- o .: "Global Quote"
symbol <- globalQuote .: "01. symbol"
open <- globalQuote .: "02. open"
high <- globalQuote .: "03. high"
low <- globalQuote .: "04. low"
price <- globalQuote .: "05. price"
volume <- globalQuote .: "06. volume"
ltd <- globalQuote .: "07. latest trading day"
previousClose <- globalQuote .: "08. previous close"
change <- globalQuote .: "09. change"
changePercent <- globalQuote .: "10. change percent"
return Quote {..}
main :: IO ()
main = do
d <- (eitherDecode <$> getJSON) :: IO (Either String Quote)
case d of
Left e -> print e
Right qt -> print (read (price qt) :: Float)
I'm new to Haskell and in order to learn the language I am working on a project that involves dealing with JSON. I am currently getting the feeling Haskell is the wrong language for the job, but that isn't the point here.
I've been struggling to understand how this works for a few days. I have searched and everything I have found does not seem to work. Here's the issue:
I have some JSON in the following format:
>>>less "path/to/json"
{
"stringA1_stringA2": {"stringA1":floatA1,
"stringA2":foatA2},
"stringB1_stringB2": {"stringB1":floatB1,
"stringB2":floatB2}
...
}
Here floatX1 and floatX2 are actually strings of the form "0.535613567", "1.221362183" etc. What I want to do is parse this into the following data
data Mydat = Mydat { name :: String, num :: Float} deriving (Show)
where name would correspond to "stringX1_stringX2" and num to floatX1 for X = A,B,...
So far I have reached a 'solution' which feels fairly hackish and convoluted and doesn't work properly.
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}
import Data.Functor
import Data.Monoid
import Data.Aeson
import Data.List
import Data.Text
import Data.Map (Map)
import qualified Data.HashMap.Strict as DHM
--import qualified Data.HashMap as DHM
import qualified Data.ByteString.Lazy as LBS
import System.Environment
import GHC.Generics
import Text.Read
data Mydat = Mydat {name :: String, num :: Float} deriving (Show)
test s = do
d <- LBS.readFile s
let v = decode d :: Maybe (DHM.HashMap String Object)
case v of
-- Just v -> print v
Just v -> return $ Prelude.map dataFromList $ DHM.toList $ DHM.map (DHM.lookup "StringA1") v
good = ['1','2','3','4','5','6','7','8','9','0','.']
f x = elem x good
dataFromList :: (String, Maybe Value) -> Mydat
dataFromList (a,b) = Mydat a (read (Prelude.filter f (show b)) :: Float)
Now I can compile this and run
test "path/to/json"
in ghci and it prints a list of Mydat's in the case where "stringX1"="stringA1" for all X. In reality there are two values for "stringX1" so aside from the hackyness this is not satisfactory. There must be a better way to do this. I get that I need to write my own parser probably but I am confused about how this works so any suggestions would be great. Thanks in advance.
The structure of your JSON is pretty nasty, but here's a basic working solution:
#!/usr/bin/env stack
-- stack --resolver lts-11.5 script --package containers --package aeson
{-# LANGUAGE OverloadedStrings #-}
import qualified Data.Map as Map
import qualified Data.Aeson as Aeson
data Mydat = Mydat { name :: String
, num :: Float
} deriving (Show)
instance Eq Mydat where
(Mydat _ x1) == (Mydat _ x2) = x1 == x2
instance Ord Mydat where
(Mydat _ x1) `compare` (Mydat _ x2) = x1 `compare` x2
type MydatRaw = Map.Map String (Map.Map String String)
processRaw :: MydatRaw -> [Mydat]
processRaw = Map.foldrWithKey go []
where go key value accum =
accum ++ (Mydat key . read <$> Map.elems value)
main :: IO ()
main =
do let json = "{\"stringA1_stringA2\":{\"stringA1\":\"0.1\",\"stringA2\":\"0.2\"}}"
print $ fmap processRaw (Aeson.eitherDecode json)
Note that read is partial and generally not a good idea. But I'll leave it to you to flesh out a safer version :)
As I commented, the best thing would probably be to make your JSON file well-formed in the sense that the float fields should really be floats, not strings.
If that's not an option, I would recommend you phrase out the type that the JSON file seems to represent as simple as possible (but without dynamic Objects), and then convert that to the type you actually want.
import Data.Map (Map)
import qualified Data.Map as Map
type GarbledJSON = Map String (Map String String)
-- ^ you could also stick with hash maps for everything, but
-- usually `Map` is actually more sensible in Haskell.
data MyDat = MyDat {name :: String, num :: Float} deriving (Show)
test :: FilePath -> IO [MyDat]
test s = do
d <- LBS.readFile s
case decode d :: Maybe GarbledJSON of
Just v -> return [ MyDat iName ( read . filter (`elem`good)
$ iVals Map.! valKey )
| (iName, iVals) <- Map.toList v
, let valKey = takeWhile (/='_') iName ]
Note that this will crash completely if any of the items don't contain the first part of the name as a string of float format, and likely give bogus items when you filter out characters that aren't good. If you just want to ignore any malformed items (which is also not a very clean approach...), you can do it this way:
test :: FilePath -> IO [MyDat]
test s = do
d <- LBS.readFile s
return $ case decode d :: Maybe GarbledJSON of
Just v -> [ MyDat iName iVal
| (iName, iVals) <- Map.toList v
, let valKey = takeWhile (/='_') iName
, Just iValStr <- [iVals Map.!? valKey]
, [(iVal,"")] <- [reads iValStr] ]
Nothing -> []
I've created my own parseJSON function for my Valuation type. Unfortunately, I get compile errors about there not being a "Generic" version of Valuation and I'm not sure what to make of it. I've read and re-read as many aeson tutorials as I can and none of them seem to mention this. I've made my Valuation type an instance of the FromJSON class and provided my own implemention of the parseJSON function, but for some reason, the compiler seems to think that it's an implementation of the other parseJSON function, and I don't know why or how to fix it. Any help would be very much appreciated.
C:\Users\John\GitHub\haskell_projects\learning\src\Lib.hs:17:10:
error:
* No instance for (GHC.Generics.Generic Valuation)
arising from a use of aeson-0.11.2.1:Data.Aeson.Types.Class.$dmparseJSON'
* In the expression:
aeson-0.11.2.1:Data.Aeson.Types.Class.$dmparseJSON
In an equation forparseJSON':
parseJSON = aeson-0.11.2.1:Data.Aeson.Types.Class.$dmparseJSON
In the instance declaration for `FromJSON Valuation'
-- While building package learning-0.1.0.0 using:
C:\Users\John\AppData\Roaming\stack\setup-exe-cache\x86_64-windows\setup-Simple-Cabal-1.24.0.0-ghc-8.0.1.exe
--builddir=.stack-work\dist\b7fec021 build lib:learning exe:learning-exe --ghc-options " -ddump-hi -ddump-to-file"
Process exited with code: ExitFailure 1
Here's the code:
{-# LANGUAGE DuplicateRecordFields #-}
{-# LANGUAGE OverloadedStrings #-}
{- The DuplicateRecordFields language extension allows records to use the same name for field labels. Without it, all the records in this module would need to have unique names for all their fields.
-}
module Lib
( someFunc,
) where
import Data.Time.Calendar
import Control.Lens
import Data.Aeson.Lens (_String, _Object, key)
import Network.Wreq
import Data.Aeson (Value(..), FromJSON, (.:), (.=), withObject)
data Valuation = Valuation {valued_on :: Day, price :: Double}
instance FromJSON Valuation where
parseJSON = withObject "valuation" $ \o -> do
query <- o .: "query"
results <- query .: "results"
quote <- results .: "quote"
price <- quote .: "Open"
return Valuation{valued_on=today, price=price}