Decode, in clojure, a JSON, clojure.data.json & cheshire.core, can't custom decode w/cheshire - json

My project parses JSONs, with a read/write library, called:
cheshire.core
I was having problems, trying to get the decode (func) to work, so I started messing around with:
data.json
My JSON contains data that consists of a field named "zone" this contains a vector with :keys inside, like so {:zone : [:hand :table]} that is stored into strings within the vector stored like so: {"zone" : ["hand" "table"]}
So I figured out how to convert the sample data using:
(mapv keyword {"zone" : ["hand"]})
which was great, I then needed to figure out how to implement a decoder for cheshire, I couldn't do this with my logic, I only spent like an hour working on this, but I had been using data.json, and the decoder function is relatively easy I think.
I got my project to work, here is some sample code:
(ns clojure-noob.core (:require
[cheshire.core :refer [decode]]
[clojure.data.json :as j-data]
) (:gen-class))
(defn -main
"I don't do a whole lot ... yet."
[& args]
)
this is using cheshire:
(let [init (decode "{\"zone\" : [\"hand\"]}" true
(fn [field-name]
(if (= field-name "zone")
(mapv keyword [])
[])))]
(println (str init)))
this is using data.json:
(defn my-value-reader [key value]
(if (= key :zone)
(mapv keyword value)
value))
(let [init (j-data/read-str
"{\"zone\" : [\"hand\"]}"
:value-fn my-value-reader
:key-fn keyword)]
(println (str init)))
I want the bottom result of these two from the console:
{:zone ["hand"]}
{:zone [:hand]}
The problem is I would like to do this using cheshire 😎
p.s. I am reading the factory section of cheshire? maybe this easier?

I would agree with #TaylorWood. Don't mess with the decoder, just do a bite in a time. First, parse json. Second, transform the result.
(def data "{\"zone\" : [\"hand\"]}")
(-> data
(cheshire.core/decode true)
(update-in ["zone"] (partial mapv keyword)))
#=> {:zone [:hand]}

I recommend you use a tool like schema.tools to coerce the input. You can add a second pass that attempts to coerce JSON strings into richer clojure types.
Here's some sample code!
;; require all the dependencies. See links below for libraries you need to add
(require '[cheshire.core :as json])
(require '[schema.core :as s])
(require '[schema.coerce :as sc])
(require '[schema-tools.core :as st])
;; your data (as before)
(def data "{\"zone\" : [\"hand\"]}")
;; a schema that wants an array of keywords
(s/defschema MyData {:zone [s/Keyword]})
;; use `select-schema` along with a JSON coercion matcher
(-> data
(json/decode true)
(st/select-schema MyData sc/json-coercion-matcher))
;; output: {:zone [:hand]}
Using defschema to define the shape of data you want gives you a general solution for serializing into JSON while getting the full benefit of Clojure's value types. Instead of explicitly "doing" the work of transforming, your schema describes the expected outcome, and hopefully coercions can do the right thing!
Links to libraries:
- https://github.com/plumatic/schema
- https://github.com/metosin/schema-tools#coercion
Note: you can do a similar thing with clojure.spec using metosin/spec-tools. Check out their readme for some help.

Related

Reading JSON from local file with Clojure?

Im pretty clear on how to parse JSON from http requests. But I have a JSON file locally I would like to use within my code.
I have tried to find a solution on google but I am struggling to figure out how to read a local JSON file from the file system
Thanks
Vinn
Use clojure/data.json library:
Add this dependency into project.clj:
[org.clojure/data.json "2.4.0"]
Add this requirement into namespace definition:
(:require [clojure.data.json :as json])
Then use read-str with slurp. I made example file filename.json with this content:
{"name":"John", "age":30, "car":null}
and read it like this:
(json/read-str (slurp "filename.json"))
=> {"name" "John", "age" 30, "car" nil}
Well, what's the difference between a json arriving from an http request and a json arriving from a local file? I suppose the real question then is "how to read from a local file", no?
Here is how to read a json from a string using clojure/data.json:
(def json-str (json/read-str "{\"a\":1,\"b\":{\"c\":\"d\"}}"))
Now, lets put the same string into a file
echo '{"a":1,"b":{"c":"d"}}' > /tmp/a.json
And lets read it from the file:
(def from-file (slurp "/tmp/a.json"))
(def json-file (json/read-str from-file))
Make sure they are the same:
(when (= json-str json-file)
(println "same" json-file))
Which would print "same" and the parsed json value.

ClojureScript - Ajax - Retrieving json

I have my dependencies
(ns test.core
(:require [reagent.core :as reagent :refer [atom]]
[ajax.core :refer [GET]]))
I then have my handler that handles the responses to my ajax call
(defn handle-response [response]
(println (type response))
(println film))
THe data from my call is JSON, in the browser it looks like this:
{"id":3,"name":"Sicario","star":"Emily Blunt","rating":5}
When I run the above Clojure, I see this:
cljs.core/PersistentArrayMap core.cljs:192:23
{id 3, name Sicario, star Emily Blunt, rating 5}
Is their a way for me to go from the PersistArrayMap, and destructure it into id, name, star and rating?
I tried
(let [film (js->clj (.parse js/JSON response) :keywordize-keys true)]
...)
Hoping I would get a map named film, but no avail!
I think maybe I could use (get-in), but can't get the syntax right for this either.
Thanks,
Got it,
I was missing :response-format :json and :keywords? true from my GET call
Now, it looks like this:
(GET (str "https://localhost:5001/api/films/" film-name)
{:response-format :json
:keywords? true
And it all works.

Clojure encode Joda DateTime with ring-json

With the following app:
; src/webapp/core.clj
(ns webapp.core
(:require [compojure.core :refer [defroutes GET]]
[ring.middleware.json :as mid-json]
[clj-time.jdbc]))
(defn foo [request]
{:body {:now (org.joda.time.DateTime/now)}})
(defroutes routes
(GET "/foo" [] foo))
(def app
(-> routes
(mid-json/wrap-json-response)))
Hitting the /foo endpoint gives me this error:
com.fasterxml.jackson.core.JsonGenerationException: Cannot JSON encode object of class: class org.joda.time.DateTime: 2017-10-21T03:38:16.207Z
Is there a simple way to get ring-json to encode the DateTime object? Do I have to write my own middleware to convert it to e.g. a string first? If so, how would I do that? (I've never written ring middleware before).
My project.clj has these dependencies FYI:
[[org.clojure/clojure "1.8.0"]
[org.clojure/java.jdbc "0.6.1"]
[ring/ring-jetty-adapter "1.4.0"]
[compojure "1.4.0"]
[ring/ring-json "0.4.0"]
[clj-time "0.14.0"]]
If you're using Cheshire to generate JSON, you can extend its protocol to handle serialization then it should "just work":
(extend-protocol cheshire.generate/JSONable
org.joda.time.DateTime
(to-json [dt gen]
(cheshire.generate/write-string gen (str dt))))
Additionally, the following modification made it work for projects that are using the time library tick.alpha.api. I was getting the error
#error {:cause Cannot JSON encode object of class:
class java.time.Instant: 2019-10-23T00:31:40.668Z
:via
[{:type com.fasterxml.jackson.core.JsonGenerationException
:message Cannot JSON encode object of class:
class java.time.Instant: 2019-10-23T00:31:40.668Z
:at [cheshire.generate$generate invokeStatic generate.clj 152]}]
Implementing the following in the file db/core.clj fixed the issue for me.
(extend-protocol cheshire.generate/JSONable
java.time.Instant
(to-json [dt gen]
(cheshire.generate/write-string gen (str dt))))

How to stream a large CSV response from a compojure API so that the whole response is not held in memory at once?

I'm new to using compojure, but have been enjoying using it so far. I'm
currently encountering a problem in one of my API endpoints that is generating
a large CSV file from the database and then passing this as the response body.
The problem I seem to be encountering is that the whole CSV file is being kept
in memory which is then causing an out of memory error in the API. What is the
best way to handle and generate this, ideally as a gzipped file? Is it possible
to stream the response so that a few thousand rows are returned at a time? When
I return a JSON response body for the same data, there is no problem returning
this.
Here is the current code I'm using to return this:
(defn complete
"Returns metrics for each completed benchmark instance"
[db-client response-format]
(let [benchmarks (completed-benchmark-metrics {} db-client)]
(case response-format
:json (json-grouped-output field-mappings benchmarks)
:csv (csv-output benchmarks))))
(defn csv-output [data-seq]
(let [header (map name (keys (first data-seq)))
out (java.io.StringWriter.)
write #(csv/write-csv out (list %))]
(write header)
(dorun (map (comp write vals) data-seq))
(.toString out)))
The data-seq is the results returned from the database, which I think is a
lazy sequence. I'm using yesql to perform the database call.
Here is my compojure resource for this API endpoint:
(defresource results-complete [db]
:available-media-types ["application/json" "text/csv"]
:allowed-methods [:get]
:handle-ok (fn [request]
(let [response-format (keyword (get-in request [:request :params :format] :json))
disposition (str "attachment; filename=\"nucleotides_benchmark_metrics." (name response-format) "\"")
response {:headers {"Content-Type" (content-types response-format)
"Content-Disposition" disposition}
:body (results/complete db response-format)}]
(ring-response response))))
Thanks to all the suggestion that were provided in this thread, I was able to create a solution using piped-input-stream:
(defn csv-output [data-seq]
(let [headers (map name (keys (first data-seq)))
rows (map vals data-seq)
stream-csv (fn [out] (csv/write-csv out (cons headers rows))
(.flush out))]
(piped-input-stream #(stream-csv (io/make-writer % {})))))
This differs from my solution because it does not realise the sequence using dorun and does not create a large String object either. This instead writes to a PipedInputStream connection asynchronously as described by the documentation:
Create an input stream from a function that takes an output stream as its
argument. The function will be executed in a separate thread. The stream
will be automatically closed after the function finishes.
Your csv-output function completely realises the dataset and turns it into a string. To lazily stream the data, you'll need to return something other than a concrete data type like a String. This suggests ring supports returning a stream, that can be lazily realised by Jetty. The answer to this question might prove useful.
I was also struggling with the streaming of large csv file. My solution was to use httpkit-channel to stream every single line of the data-seq to the client and then close the channel. My solution looks like that:
[org.httpkit.server :refer :all]
(fn handler [req]
(with-channel req channel (let [header "your$header"
data-seq ["your$seq-data"]]
(doseq [line (cons header data-seq)]
(send! channel
{:status 200
:headers {"Content-Type" "text/csv"}
:body (str line "\n")}
false))
(close channel))))

How to read json file into Clojure defrecord (to be searched later)

I have created a defrecord in a Clojure REPL:
user=> (defrecord Data [column1 column2 column3])
user.Data
How do I automate adding data to this record by reading in a .json file? Each of the columns in the defrecord corresponds exactly to a key in the json data. If the file contained a single record it would look similar to this:
[
{
"column1" : "value1"
"column2" : "value2"
"column3" : "value3"
}
]
But there are many thousands of such records in the file.
I can slurp the contents of the file like this:
(json/read-json (slurp "path/to/file.json")))
The dependencies for the read-json function are added to the project.clj file found in the directory where I am running lein repl from the command line: :dependencies [org.clojure/data.json "0.2.1"].
I would just like to be able to search the values of the records using a Clojure function, such that the value I am passing to the search function is between the values of a single record's column1 and column2 values (i.e., nth-record.column1.value <= query <= nth-record.column2.value). Once I've found a matching record, I want to return the value of another column in that same record (nth-record.column3.value). The values of columns 1 and 2 will be unique, representing a non-overlapping range of values. The value of column3 is not unique.
This seems like a fairly trivial task, but I can't figure out how to do it using the Clojure documentation or the examples I've found online. It doesn't matter to me how the records are stored internally in Clojure, as long as I can search them and return the value of a related field in the same record.
Using data.json package:
(require '[clojure.data.json :as json])
Read values into memory:
(def all-records (json/read-str (slurp "path/to/file.json")
:key-fn keyword))
;; ==> [ { :column1 "value1", :column2 "value2", :column3 "value3" }, ...]
Find matching records:
(def query "some-value")
(def matching (filter #(and (< (:column1 %) query) (< query (:column2 %))) all-records))
Get column3:
(map :column3 matching)
Collecting it all together (and making it more flexible):
(defn find-matching [select-fn result-fn records]
(map result-fn (filter select-fn records)))
(defn select-within [rec query]
(and (< (:column1 rec) query) (< query (:column2 rec))))
(find-matching #(select-within % "some-value") :column3 all-records)
Should probably use cheshire for speed.
If your queries get sufficiently complex, consider lucene, clojure has a nice wrapper.
I think you're thinking records are somehow more suitable for this than maps, as far as I can tell, you're not using any features that make records special like polymorphism. There might be a way to make cheshire spit out records, but I wouldn't bother.