I have another question regarding decoding JSON in Common Lisp. I settled on ST-JSON as my tool. I am able to get a JSO object containing the JSON data, and access all fields with st-json:getjso. I wanted to write a macro similar in principle to destructuring-bind that would provide local bindings to variables named after JSON fields (since then I started doubting if this is a good idea, but that's a different question). I came up with the following:
(defmacro destructure-jso (jso &body body)
(let (keys values)
(st-json:mapjso #'(lambda (key value)
(push key keys)
(push value values))
jso)
`(destructuring-bind ,keys ,values
,#body)))
But when I try to use it on a JSO object, I get the error The value PARAMS is not of the expected type STRUCTURE. where PARAMS is the object. Can someone explain this to me?
Thanks.
Apparently, you are using destructure-jso like this:
(let ((params (st-json:read-json-from-string "{\"foo\":42,\"bar\":\"baz\"}")))
(destructure-jso params
(list foo bar)))
However, destructure-jso, being a macro, gets handled at macro expansion time, much before the JSON even gets parsed. params is passed to your macro as a symbol, without being evaluated; and even if its evaluation was attempted, it would be unbound.
So, if you want to write a destructure-jso, you will need the list of keys at macro expansion time. You could pass the list in a normal way:
> (defmacro destructure-jso-2 (vars json &body body)
`(let ,(mapcar #'(lambda (var)
(list var `(getjso ,(string-downcase (symbol-name var)) ,json)))
vars)
,#body))
DESTRUCTURE-JSO-2
> (let ((params (st-json:read-json-from-string "{\"foo\":42,\"bar\":\"baz\"}")))
(destructure-jso-2 (foo bar)
params
(list foo bar)))
(42 "baz")
Or, if you like, use a "template" JSON for creating the mappings:
> (defmacro destructure-jso-3 (template json &body body)
(let (bindings)
(st-json:mapjso #'(lambda (key val)
(declare (ignore val))
(push (list (intern (string-upcase key)) `(getjso ,key ,json))
bindings))
(st-json:read-json-from-string template))
`(let ,bindings
,#body)))
DESTRUCTURE-JSO-3
> (let ((params (st-json:read-json-from-string "{\"foo\":42,\"bar\":\"baz\"}")))
(destructure-jso-3 "{\"foo\":null,\"bar\":null}"
params
(list foo bar)))
(42 "baz")
Here, the variable bindings come from the first (template) JSON, values from the second one. The template JSON is parsed at macroexpansion time, the params JSON every time your code is executed.
Whether either of these is a useful approach for you or not, I do not know.
Related
I'm trying to navigate JSON values using Haskell, in GHCI. I can get a JSON payload from an API, with something like this:
import Network.HTTP.Simple
baseURL <- parseRequest "https://www.googleapis.com/books/v1/volumes"
let queryString = B8.pack $ unpack q
let request = setRequestQueryString [("q", Just queryString)] $ baseURL
resp <- httpJSON request
let body = getResponseBody resp :: Object
And that gives me an Object. That object (a HashMap) contains the key "items" whose value is an Array of Objects. I want to get the first object from taht array, then gets its volumeInfo, then its industryIdentifiers, then its isbn.
In Python I would do:
identifiers = body['items'][0]['industryIdentifiers']
isbn = [id['itentifier'] for id in identifiers if id['type'] == 'ISBN_10'][0]
Or in other words, just chain accessors. How can I do this in Haskell? I've tried something like ((body ! "items") !! 0) ! "volumeInfo") but I keep getting errors like Couldn't match expected type ‘[a]’ with actual type ‘Value’.
All the tutorials I can find just say to model the data by creating a complete picture of the data as a Haskell data structure, then writing a decoder to turn that JSON data into a Haskell data object. That seems like massive overkill in this case, when the data structure I'm getting from the API is way bigger than the bit that I need, which is just the ISBN.
How does one normally drill down through a big data structure in Haskell?
The preferred way is to not manually deal with JSON values, but instead parse them into a suitable Haskell type and then index into that, which is much safer: if the input doesn't conform to the expected format, you get a clear parsing error show up at which location in the data structure something is missing, instead of an obscure key-missing error somewhere deep in your code.
{-# LANGUAGE DeriveGeneric, DeriveAnyClass #-}
data GoogleBooksVolumes = GoogleBooksVolumes
{ items :: Array GoogleBooksVolume
, ...
} deriving (Generic, FromJSON, ToJSON)
data GoogleBooksVolume = GoogleBooksVolume
{ ...
, industryIdentifiers :: Array IndustryIdentifier
, ...
} deriving (Generic, FromJSON, ToJSON)
...
If you're going to ad-hoc index into the JSON object, your best bet is the aeson-lens package. That allows you to do something very similar – and similarly unsafe – as in Python.
{-# LANGUAGE OverloadedStrings #-}
Just identifiers = Just body ^. key "items" . nth 0 . key "industryIdentifiers"
Just isbn = head [ Just idf ^. key "identifier"
| idf <- identifiers
, Just idf ^. key "type" == Just (String "ISBN_10") ]
TBH this is even worse than in Python, because if a key fails to match you just get a Nothing result without any information at all what went wrong.
A safer option is to manually pattern-match at every decision where something could go wrong, but that is a lot of boilerplate.
Using F# I am trying to scan through a JSON file and compare its arrays against a single array of (randomly generated) numbers. The formatting for the json is:
{"1":[#,#,#,#,#],"2":[#,#,#,#,#],...}
etc for 121 entries. I'm currently trying Json.NET. My problems are:
How can I import a local file with Json.NET?
How would I set about making a simple call of the json key that'd return it's array value that's fit to run it through a for loop?
Here is my code of how far I've gotten:
open System
open System.IO
open Newtonsoft.Json
(*open FSharp.Data
type Provider = JsonProvider<"powernum.json">
let numbers = Provider.Load("powernum.json")
//numbers.``1`` gets me the array but can't scan through all the IDs with an iterating for loop
(*
if numbers.``3`` = [|29;30;41;48;64|] then
printfn "True"
else printfn "False"
*)
(*numbers.JsonValue.Item "1"
let test (a: int) = string a
let testPile =
for i = 1 to 10 do
let goNum = numbers.JsonValue.Item (test i)
Console.Write goNum
Console.WriteLine ""
testPile // This works but is not usable for data comparison with an F# Array
*)
*)
let r = new StreamReader("\PATH\powernum.json")
let (json: string) = r.ReadToEnd();
let conv = JsonConvert.DeserializeObject<> (json)
Console.WriteLine("{0}",conv)//where I'm stuck with Json.NET
[<EntryPoint>]
let main argv =
let rnd = Random()
let numberGen = Set.empty.Add(rnd.Next(1,69)).Add(rnd.Next(1,69)).Add(rnd.Next(1,69)).Add(rnd.Next(1,69)).Add(rnd.Next(1,69)) |>Set.toArray |>Array.sort
Console.ReadKey() |> ignore
0// return an integer exit code
Jsontocsharp.com renders invalid.
I've tried using F# Data but from what I've found it's impossible to make an iterating loop because I have to pull up the "key" with accents encapsulating the number to read it as an int like this numbers.``1``.It doesn't take variables. Tried another method while still using F# Data but it only works as a string that errors when I try to convert it.
For comparison this is the version I prototyped in python:
import random
import json
with open('/PATH/powernum.json') as pnumbers:
data = json.load(pnumbers)
#makes an array with the range of numbers
Valid_numbers =[]
for x in range(69):
Valid_numbers.append(x+1)
generated = []
def pulledNumber():
generated[:]=[]
#adds numbers to the array from 0-4
while len(generated) !=5:
#takes a random number from the range of numbers
generate_number = random.choice(Valid_numbers)
#check if the two arrays have the same values
if generate_number not in generated:
#add to the array if values don't match
generated.append(generate_number)
generated.sort()
return generated
pulledNumber()
for x, y in data.items():
if generated not in y:
print("Id: %s passed" % x)
else:
print("Id: %s failed" % x)
pulledNumber()
break
print (pulledNumber())
f# is a statically typed language - we simply often don't notice because of its excellent type inferencing. But when deserializing from a JSON file, before writing any code, it is useful to determine whether the JSON has a fixed schema, and if so, create or choose an appropriate data model to which the JSON can be mapped automatically.
In your case, your JSON looks like:
{
"1": [29,30,41,48,64],
"2": [29,320,441,548,11]
// Additional items omitted
}
When you have here is a root object with variable property names whose values are always an array of integers. As explained in the Newtonsoft documentation Deserialize a Dictionary, such a JSON object can be deserialized to some IDictionary<string, T> for appropriate value type T. And as explained in Deserialize a Collection a JSON array can be deserialized to a collection of an appropriate item type.
In f# we use Map<'Key,'Value> to represent a dictionary, and lists to represent a materialized list of statically typed values. Thus your JSON corresponds to a
Map<string, int list>
Having determined an appropriate data model, introduce the following function to deserialize JSON from a file:
//https://www.newtonsoft.com/json/help/html/DeserializeWithJsonSerializerFromFile.htm
let jsonFromFile<'T>(fileName : string, settings : JsonSerializerSettings) =
use streamReader = new StreamReader(fileName)
use jsonReader = new JsonTextReader(streamReader)
let serializer = JsonSerializer.CreateDefault(settings)
serializer.Deserialize<'T>(jsonReader)
Now you can deserialize your JSON and filter the values for lists matching some
let fileName = "\PATH\powernum.json"
let requiredValues = [29;30;41;48;64] // Or whatever list of values you are looking for
let map = jsonFromFile<Map<string, int list>>(fileName, null)
let filteredMap =
map |> Map.toSeq
|> Seq.filter (fun (key, value) -> requiredValues = value)
|> Map.ofSeq
// Print results
filteredMap |> Map.iter (fun key value ->
printf "Key = %A has matching list of values = %A\n" key value)
Which prints out
Key = "1" has matching list of values = [29; 30; 41; 48; 64]
Notes:
Always be sure to dispose of disposable resources such as StreamReader after you are done with them. The use keyword ensures this happens automatically when the resource goes out of scope.
If you would prefer to search for an unordered set of values, you can use set instead of list:
let requiredValues = set [64;29;30;41;48] // Or whatever set of values you are looking for
let map = jsonFromFile<Map<string, Set<int>>>(fileName, null)
let filteredMap =
map |> Map.toSeq
|> Seq.filter (fun (key, value) -> requiredValues = value)
|> Map.ofSeq
As explained in Equality and Comparison Constraints in F# by Don Syme, both list and set support structural equality comparisons, which is why requiredValues = value checks that the collections have identical contents.
Demo fiddle #1 here for list and #2 here for set.
I'm learning Elm and one thing that has puzzled me is 'Json.Decode.succeed'. According to the docs
succeed : a -> Decoder a
Ignore the JSON and produce a certain Elm value.
decodeString (succeed 42) "true" == Ok 42
decodeString (succeed 42) "[1,2,3]" == Ok 42
decodeString (succeed 42) "hello" == Err ...
I understand that (although, as a beginner, I don't yet see its use). But this method is also used in a Decode pipeline, thus:
somethingDecoder : Maybe Wookie -> Decoder Something
somethingDecoder maybeWookie =
Json.Decode.succeed Something
|> required "caterpillar" Caterpillar.decoder
|> required "author" (Author.decoder maybeWookie)
What is going on here? That is, if 'succeed' ignores the JSON that's passed to it, how is it used to read JSON and turn it into Elm values? Any clues appreciated!
Just to start, the intuition for a decoder pipeline is that it acts like a curried function where piping with required and optional applies arguments one-by-one. Expect that everything, both the function, its arguments and the return value are all wrapped in Decoders.
So as an example:
succeed Something
|> required (succeed 42)
|> required (succeed "foo")
is equivalent to
succeed (Something 42 "foo")
and
decodeString (succeed (Something 42 "foo")) whatever
will return Ok (Something 42 "foo") as long as whatever is valid JSON.
When everything succeeds it's just a really convoluted function call. The more interesting aspect of decoders, and the reason we use them in the first place, is in the error path. But since 'succeed' is what's of interest here, we'll ignore that and save a lot of time, text and brain cells. Just know that without considering the error path this will all seem very contrived.
Anyway, let's try to recreate this to see how it works.
Decode.map2
The key to the pipelines, apart form the pipe operator, is the Decode.map2 function. You've probably already used it, or its siblings, if you've tried writing JSON decoders without using pipelines. We can implement our example above using map2 like this:
map2 Something
(succeed 42)
(succeed "foo")
This will work exactly like the example above. But the problem with this, from a user POV, is that if we need to add another argument we also have to change map2 to map3. And also Something isn't wrapped in a decoder, which is boring.
Calling functions wrapped in Decoders
The reason this is useful anyway is because it gives us access to several values at the same time, and the ability to combine them in whatever way we want. We can use this to call a function inside a Decoder with an argument inside a Decoder and have the result also wrapped in a Decoder:
map2 (\f x -> f x)
(succeed String.fromInt)
(succeed 42)
Currying and partial application
Unfortunately this still has the problem of needing to change the map function if we need more arguments. If only there was a way to apply arguments to a function one at a time... like if we had currying and partial application. Since we have a way to call functions wrapped in decoders now, what if we return a partially applied function instead and apply the remaining arguments later?
map2 (\f x -> f x)
(succeed Something)
(succeed 42)
will return a Decoder (string -> Something), so now we just have to rinse and repeat with this and the last argument:
map2 (\f x -> f x)
(map2 (\f x -> f x)
(succeed Something)
(succeed 42))
(succeed "")
Et voila, we have now recreated JSON decode pipelines! Although it might not look like it on the surface.
Ceci n'est pas une pipe
The final trick is to use map2 with the pipe operator. The pipe is essentially defined as \x f -> f x. See how similar this looks to the function we've been using? The only difference is that the arguments are swapped around, so we need to swap the order we pass arguments as well:
map2 (|>)
(succeed "")
(map2 (|>)
(succeed 42)
(succeed Something))
and then we can use the pipe operator again to reach the final form
succeed Something
|> map2 (|>)
(succeed 42)
|> map2 (|>)
(succeed "")
It should now be apparent that required is just an alias for map2 (|>).
And that's all there is to it!
I am relying on rustc_serialize to parse JSON data into a struct, Orders, which represents a Vec of Order structs. The JSON data may have an array or a null value; my intent is to either parse the array of orders normally, if any, or parse the null value as an Orders with an empty Vec. If neither of these is the case, then an error is to be relayed. This is my attempt:
impl Decodable for Orders {
fn decode<D: Decoder>(d: &mut D) -> Result<Self, D::Error> {
let try = d.read_seq(|d, l| {
let mut orders = Vec::new();
for _ in 0..l {
let order = try!(Decodable::decode(d));
orders.push(order);
}
Ok(Orders(orders))
});
match try {
value # Ok(_) => value,
error # Err(e) => match e {
ExpectedError(_, x) if &x == "null" => Ok(Orders(Vec::new())),
_ => error,
},
}
}
}
My issue has to do with pattern matching on ExpectedError. The compiler gives me the following error:
expected `<D as rustc_serialize::serialize::Decoder>::Error`,
found `rustc_serialize::json::DecoderError`
(expected associated type,
found enum `rustc_serialize::json::DecoderError`) [E0308]
src/api/types/json.rs:125 ExpectedError(_, x) if &x == "null" => Ok(Orders(Vec::new())),
^~~~~~~~~~~~~~~~~~~
I am stumped on this one. How can I correct this?
How can I correct this?
In general, you would have to choose between being generic or specialized. You cannot pattern match on an associated type because this type can be anything and a generic method should work for any type which satisfies the constraints.
For example, in your case:
<D as rustc_serialize::serialize::Decoder>::Error can be anything
rustc_serialize::json::DecoderError is but one possibility
So you should normally choose between using some abstract D or specializing the decoding for json.
However, since you are implementing Decodable here, you cannot choose NOT to be generic as you do not get to pick the signature.
Furthermore, it appears that rustc_serialize does not support tentative parsing (or at least, its json implementation does not), so I am afraid that you cannot check for nullity (using d.read_nil()) beforehand.
I suppose those limitations can be seen as the reason that this library is being retired in favor of serde, and can only encourage you to try it out.
A web service returns a response as ByteString
req <- parseUrl "https://api.example.com"
res <- withManager $ httpLbs $ configReq req
case (HashMap.lookup "result" $ responseBody res) of .... -- error - responseBody returns ByteString
where
configReq r = --......
To be more specific, responseBody returns data in ByteString, although it's actually valid JSON. I need to find a value in it. Obviously, it would be easier to find it if it was JSON and not ByteString.
If that's the case, how do I convert it to JSON?
UPDATE:
decode $ responseBody resp :: IO (Either String Aeson.Value)
error:
Couldn't match expected type `IO (Either String Value)'
with actual type `Maybe a0'
You'll find several resources for converting bytestring to JSON. The simplest use cases are on the hackage page itself, and the rest you can infer using type signatures of the entities involved.
https://hackage.haskell.org/package/aeson-0.7.0.6/docs/Data-Aeson.html
But here's a super quick intro to JSON with Aeson:
In most languages, you have things like this:
someString = '{ "name" : ["value1", 2] }'
aDict = json.loads(someString)
This is obviously great, because JSON has a nearly one to one mapping with a fundamental data-structure of the language. Containers in most dynamic languages can contain values of any type, and so moving from JSON to data structure is a single step.
However, that is not the case with Haskell. You can't put things of arbitrary types into a container like type (A list, or a dictionary).
So Aeson does a neat thing. It defines an intermediate Haskell type for you, that maps directly to JSON.
A fundamental unit in Aeson is a Value. The Value can contain many things. Like an integer, string, an array, or an object.
https://hackage.haskell.org/package/aeson-0.7.0.6/docs/Data-Aeson.html#t:Value
An aeson array is a Vector (like a list but better) of Values and an aeson object is a HashMap of Text to Values
The next interesting step is that you can define functions that will convert an Aeson value to your Haskell type. This completes the loop. ByteString to Value to a custom type.
So all you do is implement parseJSON and toJSON functions that convert aeson Values to your type and vice-versa. The bit that converts a bytestring into a valid aeson value is implemented by aeson. So the heavy lifting is all done.
Just important to note, that Aeson bytestring is a lazy bytestring, so you might need some strict to lazy helpers.
stringToLazy :: String -> ByteString
stringToLazy x = Data.Bytestring.Lazy.fromChunks [(Data.ByteString.Char8.pack x)]
lazyToString :: ByteString -> String
lazyToString x = Data.ByteString.Char8.unpack $ Data.ByteString.Char8.concat $ Data.ByteString.Lazy.toChunks
That should be enough to get started with Aeson.
--
Common decoding functions with Aeson:
decode :: ByteString -> Maybe YourType
eitherDecode :: ByteString -> Either String YourType.
In your case, you're looking for eitherDecode.