Im pretty clear on how to parse JSON from http requests. But I have a JSON file locally I would like to use within my code.
I have tried to find a solution on google but I am struggling to figure out how to read a local JSON file from the file system
Thanks
Vinn
Use clojure/data.json library:
Add this dependency into project.clj:
[org.clojure/data.json "2.4.0"]
Add this requirement into namespace definition:
(:require [clojure.data.json :as json])
Then use read-str with slurp. I made example file filename.json with this content:
{"name":"John", "age":30, "car":null}
and read it like this:
(json/read-str (slurp "filename.json"))
=> {"name" "John", "age" 30, "car" nil}
Well, what's the difference between a json arriving from an http request and a json arriving from a local file? I suppose the real question then is "how to read from a local file", no?
Here is how to read a json from a string using clojure/data.json:
(def json-str (json/read-str "{\"a\":1,\"b\":{\"c\":\"d\"}}"))
Now, lets put the same string into a file
echo '{"a":1,"b":{"c":"d"}}' > /tmp/a.json
And lets read it from the file:
(def from-file (slurp "/tmp/a.json"))
(def json-file (json/read-str from-file))
Make sure they are the same:
(when (= json-str json-file)
(println "same" json-file))
Which would print "same" and the parsed json value.
Related
I have a configuration file with:
{path, "/mnt/test/"}.
{name, "Joe"}.
The path and the name could be changed by a user. As I know, there is a way to save those variables in a module by usage of file:consult/1 in
-define(VARIABLE, <parsing of the config file>).
Are there any better ways to read a config file when the module begins to work without making a parsing function in -define? (As I know, according to Erlang developers, it's not the best way to make a complicated functions in -define)
If you need to store config only when you start the application - you may use application config file which is defined in 'rebar.config'
{profiles, [
{local,
[{relx, [
{dev_mode, false},
{include_erts, true},
{include_src, false},
{vm_args, "config/local/vm.args"}]
{sys_config, "config/local/yourapplication.config"}]
}]
}
]}.
more info about this here: rebar3 configuration
next step to create yourapplication.config - store it in your application folder /app/config/local/yourapplication.config
this configuration should have structure like this example
[
{
yourapplicationname, [
{path, "/mnt/test/"},
{name, "Joe"}
]
}
].
so when your application is started
you can get the whole config data with
{ok, "/mnt/test/"} = application:get_env(yourapplicationname, path)
{ok, "Joe"} = application:get_env(yourapplicationname, name)
and now you may -define this variables like:
-define(VARIABLE,
case application:get_env(yourapplicationname, path) of
{ok, Data} -> Data
_ -> undefined
end
).
My project parses JSONs, with a read/write library, called:
cheshire.core
I was having problems, trying to get the decode (func) to work, so I started messing around with:
data.json
My JSON contains data that consists of a field named "zone" this contains a vector with :keys inside, like so {:zone : [:hand :table]} that is stored into strings within the vector stored like so: {"zone" : ["hand" "table"]}
So I figured out how to convert the sample data using:
(mapv keyword {"zone" : ["hand"]})
which was great, I then needed to figure out how to implement a decoder for cheshire, I couldn't do this with my logic, I only spent like an hour working on this, but I had been using data.json, and the decoder function is relatively easy I think.
I got my project to work, here is some sample code:
(ns clojure-noob.core (:require
[cheshire.core :refer [decode]]
[clojure.data.json :as j-data]
) (:gen-class))
(defn -main
"I don't do a whole lot ... yet."
[& args]
)
this is using cheshire:
(let [init (decode "{\"zone\" : [\"hand\"]}" true
(fn [field-name]
(if (= field-name "zone")
(mapv keyword [])
[])))]
(println (str init)))
this is using data.json:
(defn my-value-reader [key value]
(if (= key :zone)
(mapv keyword value)
value))
(let [init (j-data/read-str
"{\"zone\" : [\"hand\"]}"
:value-fn my-value-reader
:key-fn keyword)]
(println (str init)))
I want the bottom result of these two from the console:
{:zone ["hand"]}
{:zone [:hand]}
The problem is I would like to do this using cheshire 😎
p.s. I am reading the factory section of cheshire? maybe this easier?
I would agree with #TaylorWood. Don't mess with the decoder, just do a bite in a time. First, parse json. Second, transform the result.
(def data "{\"zone\" : [\"hand\"]}")
(-> data
(cheshire.core/decode true)
(update-in ["zone"] (partial mapv keyword)))
#=> {:zone [:hand]}
I recommend you use a tool like schema.tools to coerce the input. You can add a second pass that attempts to coerce JSON strings into richer clojure types.
Here's some sample code!
;; require all the dependencies. See links below for libraries you need to add
(require '[cheshire.core :as json])
(require '[schema.core :as s])
(require '[schema.coerce :as sc])
(require '[schema-tools.core :as st])
;; your data (as before)
(def data "{\"zone\" : [\"hand\"]}")
;; a schema that wants an array of keywords
(s/defschema MyData {:zone [s/Keyword]})
;; use `select-schema` along with a JSON coercion matcher
(-> data
(json/decode true)
(st/select-schema MyData sc/json-coercion-matcher))
;; output: {:zone [:hand]}
Using defschema to define the shape of data you want gives you a general solution for serializing into JSON while getting the full benefit of Clojure's value types. Instead of explicitly "doing" the work of transforming, your schema describes the expected outcome, and hopefully coercions can do the right thing!
Links to libraries:
- https://github.com/plumatic/schema
- https://github.com/metosin/schema-tools#coercion
Note: you can do a similar thing with clojure.spec using metosin/spec-tools. Check out their readme for some help.
With the following app:
; src/webapp/core.clj
(ns webapp.core
(:require [compojure.core :refer [defroutes GET]]
[ring.middleware.json :as mid-json]
[clj-time.jdbc]))
(defn foo [request]
{:body {:now (org.joda.time.DateTime/now)}})
(defroutes routes
(GET "/foo" [] foo))
(def app
(-> routes
(mid-json/wrap-json-response)))
Hitting the /foo endpoint gives me this error:
com.fasterxml.jackson.core.JsonGenerationException: Cannot JSON encode object of class: class org.joda.time.DateTime: 2017-10-21T03:38:16.207Z
Is there a simple way to get ring-json to encode the DateTime object? Do I have to write my own middleware to convert it to e.g. a string first? If so, how would I do that? (I've never written ring middleware before).
My project.clj has these dependencies FYI:
[[org.clojure/clojure "1.8.0"]
[org.clojure/java.jdbc "0.6.1"]
[ring/ring-jetty-adapter "1.4.0"]
[compojure "1.4.0"]
[ring/ring-json "0.4.0"]
[clj-time "0.14.0"]]
If you're using Cheshire to generate JSON, you can extend its protocol to handle serialization then it should "just work":
(extend-protocol cheshire.generate/JSONable
org.joda.time.DateTime
(to-json [dt gen]
(cheshire.generate/write-string gen (str dt))))
Additionally, the following modification made it work for projects that are using the time library tick.alpha.api. I was getting the error
#error {:cause Cannot JSON encode object of class:
class java.time.Instant: 2019-10-23T00:31:40.668Z
:via
[{:type com.fasterxml.jackson.core.JsonGenerationException
:message Cannot JSON encode object of class:
class java.time.Instant: 2019-10-23T00:31:40.668Z
:at [cheshire.generate$generate invokeStatic generate.clj 152]}]
Implementing the following in the file db/core.clj fixed the issue for me.
(extend-protocol cheshire.generate/JSONable
java.time.Instant
(to-json [dt gen]
(cheshire.generate/write-string gen (str dt))))
I'm new to using compojure, but have been enjoying using it so far. I'm
currently encountering a problem in one of my API endpoints that is generating
a large CSV file from the database and then passing this as the response body.
The problem I seem to be encountering is that the whole CSV file is being kept
in memory which is then causing an out of memory error in the API. What is the
best way to handle and generate this, ideally as a gzipped file? Is it possible
to stream the response so that a few thousand rows are returned at a time? When
I return a JSON response body for the same data, there is no problem returning
this.
Here is the current code I'm using to return this:
(defn complete
"Returns metrics for each completed benchmark instance"
[db-client response-format]
(let [benchmarks (completed-benchmark-metrics {} db-client)]
(case response-format
:json (json-grouped-output field-mappings benchmarks)
:csv (csv-output benchmarks))))
(defn csv-output [data-seq]
(let [header (map name (keys (first data-seq)))
out (java.io.StringWriter.)
write #(csv/write-csv out (list %))]
(write header)
(dorun (map (comp write vals) data-seq))
(.toString out)))
The data-seq is the results returned from the database, which I think is a
lazy sequence. I'm using yesql to perform the database call.
Here is my compojure resource for this API endpoint:
(defresource results-complete [db]
:available-media-types ["application/json" "text/csv"]
:allowed-methods [:get]
:handle-ok (fn [request]
(let [response-format (keyword (get-in request [:request :params :format] :json))
disposition (str "attachment; filename=\"nucleotides_benchmark_metrics." (name response-format) "\"")
response {:headers {"Content-Type" (content-types response-format)
"Content-Disposition" disposition}
:body (results/complete db response-format)}]
(ring-response response))))
Thanks to all the suggestion that were provided in this thread, I was able to create a solution using piped-input-stream:
(defn csv-output [data-seq]
(let [headers (map name (keys (first data-seq)))
rows (map vals data-seq)
stream-csv (fn [out] (csv/write-csv out (cons headers rows))
(.flush out))]
(piped-input-stream #(stream-csv (io/make-writer % {})))))
This differs from my solution because it does not realise the sequence using dorun and does not create a large String object either. This instead writes to a PipedInputStream connection asynchronously as described by the documentation:
Create an input stream from a function that takes an output stream as its
argument. The function will be executed in a separate thread. The stream
will be automatically closed after the function finishes.
Your csv-output function completely realises the dataset and turns it into a string. To lazily stream the data, you'll need to return something other than a concrete data type like a String. This suggests ring supports returning a stream, that can be lazily realised by Jetty. The answer to this question might prove useful.
I was also struggling with the streaming of large csv file. My solution was to use httpkit-channel to stream every single line of the data-seq to the client and then close the channel. My solution looks like that:
[org.httpkit.server :refer :all]
(fn handler [req]
(with-channel req channel (let [header "your$header"
data-seq ["your$seq-data"]]
(doseq [line (cons header data-seq)]
(send! channel
{:status 200
:headers {"Content-Type" "text/csv"}
:body (str line "\n")}
false))
(close channel))))
As per my understanding "ffprobe" will provide file related data in JSON format. So, I have installed the ffprobe in my Ubuntu machine but I don't know how to access the ffprobe JSON response using Java/Grails.
Expected response format:
{
"format": {
"filename": "/Users/karthick/Documents/videos/TestVideos/sample.ts",
"nb_streams": 2,
"nb_programs": 1,
"format_name": "mpegts",
"format_long_name": "MPEG-TS (MPEG-2 Transport Stream)",
"start_time": "1.430800",
"duration": "170.097489",
"size": "80425836",
"bit_rate": "3782576",
"probe_score": 100
}
}
This is my groovy code
def process = "ffprobe -v quiet -print_format json -show_format -show_streams HelloWorld.mpeg ".execute()
println "Found ${process.text}"
render process as JSON
I m able to get the process object and i m not able to get the json response
Should i want to convert the process object to json object?
OUTPUT:
Found java.lang.UNIXProcess#75566697
org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Error converting Bean with class java.lang.UNIXProcess
Grails has nothing to do with this. Groovy can execute arbitrary shell commands in a very simplistic way:
"mkdir foo".execute()
Or for more advanced features, you might look into using ProcessBuilder. At the end of the day, you need to execute ffprobe and then capture the output stream of JSON to use in your app.
Groovy provides a simple way to execute command line processes. Simply
write the command line as a string and call the execute() method.
The execute() method returns a java.lang.Process instance.
println "ffprobe <options>".execute().text
[Source]