How can I access JSON values from GHCI? - json

I'm trying to navigate JSON values using Haskell, in GHCI. I can get a JSON payload from an API, with something like this:
import Network.HTTP.Simple
baseURL <- parseRequest "https://www.googleapis.com/books/v1/volumes"
let queryString = B8.pack $ unpack q
let request = setRequestQueryString [("q", Just queryString)] $ baseURL
resp <- httpJSON request
let body = getResponseBody resp :: Object
And that gives me an Object. That object (a HashMap) contains the key "items" whose value is an Array of Objects. I want to get the first object from taht array, then gets its volumeInfo, then its industryIdentifiers, then its isbn.
In Python I would do:
identifiers = body['items'][0]['industryIdentifiers']
isbn = [id['itentifier'] for id in identifiers if id['type'] == 'ISBN_10'][0]
Or in other words, just chain accessors. How can I do this in Haskell? I've tried something like ((body ! "items") !! 0) ! "volumeInfo") but I keep getting errors like Couldn't match expected type ‘[a]’ with actual type ‘Value’.
All the tutorials I can find just say to model the data by creating a complete picture of the data as a Haskell data structure, then writing a decoder to turn that JSON data into a Haskell data object. That seems like massive overkill in this case, when the data structure I'm getting from the API is way bigger than the bit that I need, which is just the ISBN.
How does one normally drill down through a big data structure in Haskell?

The preferred way is to not manually deal with JSON values, but instead parse them into a suitable Haskell type and then index into that, which is much safer: if the input doesn't conform to the expected format, you get a clear parsing error show up at which location in the data structure something is missing, instead of an obscure key-missing error somewhere deep in your code.
{-# LANGUAGE DeriveGeneric, DeriveAnyClass #-}
data GoogleBooksVolumes = GoogleBooksVolumes
{ items :: Array GoogleBooksVolume
, ...
} deriving (Generic, FromJSON, ToJSON)
data GoogleBooksVolume = GoogleBooksVolume
{ ...
, industryIdentifiers :: Array IndustryIdentifier
, ...
} deriving (Generic, FromJSON, ToJSON)
...
If you're going to ad-hoc index into the JSON object, your best bet is the aeson-lens package. That allows you to do something very similar – and similarly unsafe – as in Python.
{-# LANGUAGE OverloadedStrings #-}
Just identifiers = Just body ^. key "items" . nth 0 . key "industryIdentifiers"
Just isbn = head [ Just idf ^. key "identifier"
| idf <- identifiers
, Just idf ^. key "type" == Just (String "ISBN_10") ]
TBH this is even worse than in Python, because if a key fails to match you just get a Nothing result without any information at all what went wrong.
A safer option is to manually pattern-match at every decision where something could go wrong, but that is a lot of boilerplate.

Related

Haskell building simple JSON parser

Getting my feet wet with building stuff, and not being able to get Aeson to work properly I decided my new project is building a JSON parser. Very abstract since it is one way or another, so it wouldn't make sense to put all the code here.
The ByteString library lets me do what I need. Remove characters, replace stuff, but: I have a very hard time reconstructing it the exact way I took it apart. Data.Text however seems more appropriate for the job but when generated a lot of noise with /"/, \n etc.
What would be the best and fastest way to clear a file from all rubbish and restore the remaining parts to useful text? Very small part below. Remarks on the code are welcome. Learning here.
import Network.HTTP.Simple
import GHC.Generics
import qualified Data.ByteString as B
import qualified Data.ByteString.Char8 as C
import Data.Text as T
import Data.Char
import Data.Text.Encoding as DTE
word8QuoteMark = fromIntegral (ord '"')
word8Newline = fromIntegral (ord '\n')
word8Backslash = fromIntegral (ord ':')
filterJson jsonData = B.filter (/= word8Backslash)
(B.filter (/= word8Newline)
(B.filter (/= word8QuoteMark) jsonData))
importJson :: IO ()
importJson = do
jsonData <- B.readFile "local.json"
output <- return (filterJson jsonData)
print $ (output)
Now the downside is, that if someone is called eg. François, it is now returned as Fran\195\167ois. I think I would need a lot more steps to do this in Data.Text, but correct me if I am wrong...
Note: i saw in a post that Daniel Wagner strongly advises against ByteString for text, but just for the sake of argument.
JSON is, by definition, a Unicode string that represents a data structure. What you get from B.readFile, though, is a raw byte string that you must first decode to get a Unicode string. To do that, you need to know what encoding was used to create the file. Assuming the file uses UTF-8 encoding, you can do something like
import Data.Text
importJson :: String -> IO Text
importJson name = do
jsonData <- B.readFile name
return (Data.Text.Encoding.decodeUtf8 jsonData)
Once you have a Text value, you can parse that into some data structure according to the JSON grammar.

Parsing problematic JSON with Aeson

I am trying to parse JSON objects, which are generally of the form
{
"objects": [a bunch of records that can assume a few different forms],
"parameters": [same deal],
"values": {
"k1": "v1",
"k2": "v2",
...
}
}
using Haskell's Aeson library. Part of this task is simple in the sense that the parameters and values fields need no custom parsing whatsoever (and so seem to need only a generically derived instance of FromJSON), and most of the records contained within the array associated to objects also need no special parsing. However, there are some parts of parsing the records within the array of objects that, when considered separately, have documented solutions, but together present problems that I haven't figured out how to address.
Now, the possible variants of record inside the objects and parameters arrays are finite in number and often contain the same keys; for example, all of them have a "name" key or an "id" key, or such. But also many of them have a "type" key, which is a reserved keyword, and so cannot be parsed generically. This is the first problem.
The second problem is that one of the possible variants of record inside objects can have a key -- "depends" let's say -- whose value may assume different types. It can either be a single record
{
"objects": [
{
"depends": {
"reference": "r1"
},
...
],
...
}
or a list of records
{
"objects": [
"depends": [
{"reference": "r1"},
{"reference": "r2"},
etc.
],
],
...
}
and it happens that this is the one field that I would like to manipulate in a custom fashion after converting to a Haskell object (eventually I want to represent the collection of such "depends" references as a Data.Graph graph).
My initial attempt was to create one huge record type that subsumes all of the possible keys in the elements of the objects and parameters arrays. Something like this:
{-# LANGUAGE DeriveAnyClass #-}
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RecordWildCards #-}
import Data.Aeson
import GHC.Generics
data Ref = Ref
{ ref :: String
} deriving (Show, Generic, FromJSON, ToJSON)
data Reference
= Reference Ref
| References [Ref]
deriving (Show, Generic, FromJSON, ToJSON)
type MString = Maybe String -- I'm writing this a lot using this approach
data PObject = PObject
-- Each of the object/parameter records have these keys
{ _name :: String
, _id :: String
-- Other keys that might appear in a given object/parameter record
, _type :: MString
, _role :: MString
, _depends :: Maybe Reference
-- A bunch more
} deriving Show
instance FromJSON PObject where
parseJSON = withObject "PObject" $ \o -> do
_name <- o .: "name"
_id <- o .: "id"
_type <- o .:? "type"
_role <- o .:? "role"
_depends <- o .:? "depends"
-- etc.
return PObject{..}
And then finally, the whole JSON object would be represented like
data MyJSONObject = MyJSONObject
{ objects :: Maybe [PObject]
, parameters :: Maybe [PObject]
, values :: Maybe Object
} deriving (Show, Generic, FromJSON)
This works until it tries to parse a "depends" field, reporting that
"Error in $.objects[2].depends: key \"tag\" not present"
There are no "tag" keys, so I'm not sure what this means. I suspect it has to do with the generic instances of FromJSON for Ref and Reference.
My questions:
What does this error indicate? So far in my learning of Haskell, the errors have always been very helpful. This one is not. Do I need to do something special for the "depends" key in my parseJSON function?
All of this boilerplate is really because of two keys -- "type" and "depends". Is there a more elegant way to deal with these keys?
Relatedly, this is part of my first real Haskell project, so I have a more general design question. Experienced Haskellers and Aeson users, how would you lay out your types and instances for this type of JSON? I tried listing out each possible variant of objects/parameters record as its own separate type, and only writing custom FromJSON instances for those that have a "depends" or "type" key, but this produced a lot more boilerplate code and in any case doesn't solve any of the other issues I have. General pointers on "best practices", idiomatic usage, etc. would be extremely useful and appreciated.
There are no "tag" keys, so I'm not sure what this means. I suspect it has to do with the generic instances of FromJSON for Ref and Reference.
That's spot on. By default, aeson will use the defaultTaggedObject to encode sum types. References is a sum type. Therefore, aeson introduces a tag to distinguish the constructors. You can try that with a short example:
ghci> data Example = A () | B deriving (Generic,ToJSON)
ghci> encode B
"{\"tag\":\"B\",\"contents\":[]}"
When you use _depends <- o .:? "depends", the Reference parser does not find its tag. You have to write some parsing code there yourself.
All of this boilerplate is really because of two keys -- "type" and
"depends". Is there a more elegant way to deal with these keys?
You could keep the underscores in the field names and use fieldLabelModifier in the Options data type to strip them for parsing purposes.

Parsing the JSON representation of database rows in Scala.js

I am trying out scala.js, and would like to use simple extracted row data in json format from my Postgres database.
Here is an contrived example of the type of json I would like to parse into some strongly typed scala collection, features to note are that there are n rows, various column types including an array just to cover likely real life scenarios, don't worry about the SQL which creates an inline table to extract the JSON from, I've included it for completeness, its the parsing of the JSON in scala.js that is causing me problems
with inline_t as (
select * from (values('One',1,True,ARRAY[1],1.0),
('Six',6,False,ARRAY[1,2,3,6],2.4494),
('Eight',8,False,ARRAY[1,2,4,8],2.8284)) as v (as_str,as_int,is_odd,factors,sroot))
select json_agg(t) from inline_t t;
[{"as_str":"One","as_int":1,"is_odd":true,"factors":[1],"sroot":1.0},
{"as_str":"Six","as_int":6,"is_odd":false,"factors":[1,2,3,6],"sroot":2.4494},
{"as_str":"Eight","as_int":8,"is_odd":false,"factors":[1,2,4,8],"sroot":2.8284}]
I think this should be fairly easy using something like upickle or prickle as hinted at here: How to parse a json string to a case class in scaja.js and vice versa but I haven't been able to find a code example, and I'm not up to speed enough with Scala or Scala.js to work it out myself. I'd be very grateful if someone could post some working code to show how to achieve the above
EDIT
This is the sort of thing I've tried, but I'm not getting very far
val jsparsed = scala.scalajs.js.JSON.parse(jsonStr3)
val jsp1 = jsparsed.selectDynamic("1")
val items = jsp1.map{ (item: js.Dynamic) =>
(item.as_str, item.as_int, item.is_odd, item.factors, item.sroot)
.asInstanceOf[js.Array[(String, Int, Boolean, Array[Int], Double)]].toSeq
}
println(items._1)
So you are in a situation where you actually want to manipulate JSON values. Since you're not serializing/deserializing Scala values from end-to-end, serialization libraries like uPickle or Prickle will not be very helpful to you.
You could have a look at a cross-platform JSON manipulation library, such as circe. That would give you the advantage that you wouldn't have to "deal with JavaScript data structures" at all. Instead, the library would parse your JSON and expose it as a Scala data structure. This is probably the best option if you want your code to also cross-compile.
If you're only writing Scala.js code, and you want a more lightweight version (no dependency), I recommend declaring types for your JSON "schema", then use those types to perform the conversion in a safer way:
import scala.scalajs.js
import scala.scalajs.js.annotation._
// type of {"as_str":"Six","as_int":6,"is_odd":false,"factors":[1,2,3,6],"sroot":2.4494}
#ScalaJSDefined
trait Row extends js.Object {
val as_str: String
val as_int: Int
val is_odd: Boolean
val factors: js.Array[Int]
val sroot: Double
}
type Rows = js.Array[Row]
val rows = js.JSON.parse(jsonStr3).asInstanceOf[Rows]
val items = (for (row <- rows) yield {
import row._
(as_str, as_int, is_odd, factors.toArray, sroot)
}).toSeq

Finding a value in ByteString (which is actually JSON)

A web service returns a response as ByteString
req <- parseUrl "https://api.example.com"
res <- withManager $ httpLbs $ configReq req
case (HashMap.lookup "result" $ responseBody res) of .... -- error - responseBody returns ByteString
where
configReq r = --......
To be more specific, responseBody returns data in ByteString, although it's actually valid JSON. I need to find a value in it. Obviously, it would be easier to find it if it was JSON and not ByteString.
If that's the case, how do I convert it to JSON?
UPDATE:
decode $ responseBody resp :: IO (Either String Aeson.Value)
error:
Couldn't match expected type `IO (Either String Value)'
with actual type `Maybe a0'
You'll find several resources for converting bytestring to JSON. The simplest use cases are on the hackage page itself, and the rest you can infer using type signatures of the entities involved.
https://hackage.haskell.org/package/aeson-0.7.0.6/docs/Data-Aeson.html
But here's a super quick intro to JSON with Aeson:
In most languages, you have things like this:
someString = '{ "name" : ["value1", 2] }'
aDict = json.loads(someString)
This is obviously great, because JSON has a nearly one to one mapping with a fundamental data-structure of the language. Containers in most dynamic languages can contain values of any type, and so moving from JSON to data structure is a single step.
However, that is not the case with Haskell. You can't put things of arbitrary types into a container like type (A list, or a dictionary).
So Aeson does a neat thing. It defines an intermediate Haskell type for you, that maps directly to JSON.
A fundamental unit in Aeson is a Value. The Value can contain many things. Like an integer, string, an array, or an object.
https://hackage.haskell.org/package/aeson-0.7.0.6/docs/Data-Aeson.html#t:Value
An aeson array is a Vector (like a list but better) of Values and an aeson object is a HashMap of Text to Values
The next interesting step is that you can define functions that will convert an Aeson value to your Haskell type. This completes the loop. ByteString to Value to a custom type.
So all you do is implement parseJSON and toJSON functions that convert aeson Values to your type and vice-versa. The bit that converts a bytestring into a valid aeson value is implemented by aeson. So the heavy lifting is all done.
Just important to note, that Aeson bytestring is a lazy bytestring, so you might need some strict to lazy helpers.
stringToLazy :: String -> ByteString
stringToLazy x = Data.Bytestring.Lazy.fromChunks [(Data.ByteString.Char8.pack x)]
lazyToString :: ByteString -> String
lazyToString x = Data.ByteString.Char8.unpack $ Data.ByteString.Char8.concat $ Data.ByteString.Lazy.toChunks
That should be enough to get started with Aeson.
--
Common decoding functions with Aeson:
decode :: ByteString -> Maybe YourType
eitherDecode :: ByteString -> Either String YourType.
In your case, you're looking for eitherDecode.

FromJSON custom for custom type

The newest version of Data.Aeson changed the way that ToJSON and FromJSON work for simple types like:
data Permission = Read | Write
It used to be that the generic call:
instance ToJSON Permission where
...Would create JSON that looked like {"Read":[]} or {"Write":[]}.
But now it creates:
{tag:"Read",contents:"[]"}
Which makes sense but breaks code I have written. I wrote a toJSON part by hand to give the correct looking stuff but writing the fromJSON is confusing me.
Any ideas?
Thanks
You could control how datatype with all nullary constructors is encoded using allNullaryToStringTag field on Data.Aeson.Options. Set it to True and it will be encoded simply as string.
import Data.Aeson.Types (Options (..), defaultOptions)
data Permission = Read | Write
$(deriveToJSON (defaultOptions {allNullaryToStringTag = True}) ''Permission)
Take a look at Options definition, it contains other handy fields.
Since the value contained in the Object constructor for Data.Aeson.Value is just a strict HashMap, we can extract the keys from it and make a decision based on that. I tried this and it worked pretty well.
{-# LANGUAGE OverloadedStrings #-}
module StackOverflow where
import Data.Aeson
import Control.Monad
import Data.HashMap.Strict (keys)
data Permission = Read | Write
instance FromJSON Permission where
parseJSON (Object v) =
let ks = keys v
in case ks of
["Read"] -> return Read
["Write"] -> return Write
_ -> mzero
parseJSON _ = mzero
You can test it with decode "{\"Read\": []}" :: Maybe Permission. The mzero in parseJSON ensures that if something else is passed in, it'll just return Nothing. Since you seem to want to only check if there is a single key matching one of your two permissions, this is pretty straightforward and will properly return Nothing on all other inputs.