I am trying to use clj-http/client to retrieve a web page. It looks like it is kind of working, but sometimes I receive this kind of Exception
Exception in thread "main" java.lang.ClassCastException: clojure.lang.Var$Unbound cannot be cast to clojure.lang.Named,compiling:(/tmp/form-init8570082100332402765.clj:1:72)
This is a simple function which retrieves an url from a database (from jdbc/query :row-fn), requests the contents of the url and writes it to the database. The proxy data is random and just for reference.
(defn get-source
"get content of an url"
[row]
(def my-proxy "72.159.148.20")
(def my-port 10000)
(def my-url (:url row))
(def h {"User-Agent" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/34.0"})
(try
(def my-body (:body (client/get my-url {:proxy-host my-proxy :proxy-port my-port :follow-redirects false :headers h :conn-timeout 100000})))
(catch clojure.lang.ExceptionInfo e
(prn "caught" e))
)
(write-data-to-db my-url my-body))
The value of row is coming from simple db query:
(defn -main
"I don't do a whole lot ... yet."
[& args]
(def db-spec
{
:subprotocol "mysql"
:subname "//localhost:3306/a"
:user "user"
:password "pass"})
((jdbc/query db-spec ["SELECT url FROM main where html is null and url is not null limit 2"] :row-fn get-source)))
Thanks for healthy criticism, I have fixed my example to be more correct.
(defn get-source
"get list of urls to grab in"
[row]
(let [proxy "107.161.31.220" port 8080 url (:url row)
h {"User-Agent" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/34.0"}]
(:status (client/get url {:proxy-host proxy :proxy-port port :follow-redirects false :headers h :conn-timeout 100000}))))
I hope this way it much better, and I can not get that Exception any more, so could be it was buried inside that mess I came with initially.
I think it's not idiomatic in Clojure (or it's just not declarative) to def something inside a function or "block". Def declares global variables.
From http://clojure.org/special_forms:
(def symbol init?)
Creates and interns or locates a global var with the name of symbol and a namespace of the value of the current namespace. <...> def yields the var itself (not its value).
I guess you have to put (write-data-to-db) inside try-block and use let instead of def. Right now you're trying to access value of something that might fail.
Related
I've got a website that I'm the process of developing, the backend route which isn't working is /login with the compojure route as:
(POST login "/" [] check-admin-before-logging)
This is what the check-admin-before-logging function does:
(defn redirect-admin-to-page
[]
(-> (response/redirect "/admin/home")
(assoc :body "logging you in...")
(assoc :cookies {:token (token/sign-token)})))
;; function that checks whether credentials are in fact valid.
;; check-admin is a db func which compares hashed pswd with input.
(defn check-admin-before-logging-in
[request]
(let [username (:username (:params request))
password (:password (:params request))]
(if (check-admin username password)
(redirect-admin-to-page username)
(str "username : " username " and password " password " is not valid.\n" request))))
For the moment it just returns a string, as for testing purposes.
and for the app I have wrapped it in the following middleware:
(def app
(-> all-routes
(wrap-defaults (assoc-in site-defaults [:security :anti-forgery] false))
(wrap-json-body)
;; (wrap-json-params)
(wrap-json-response)
))
Now when I make an AJAX request with the following on the frontend:
(defn submit
[]
(let [username *grabbed from input field...*
password *grabbed from input field...*]
(POST "/login" {:params {:username username :password password}})))
But the problem arises when I actually click the button I get this. The console shows the result of the post request itself, with :params being empty and some mangled JSON in the :body key. If I was to try and include
(wrap-json-params)
within app, I would get nil in the :body and :params key. Moreover it seems there is a problem with the output, as cheshire has problems with parsing it. If I leave the ring-json library to the side and just have wrap-defaults over app, I just get this for the body key
:body #object[org.eclipse.jetty.server.HttpInputOverHTTP 0x6ef9a9e1 HttpInputOverHTTP#6ef9a9e1]
My question is how do I change the json so that it is able to be understood by wrap-json-params or cheshire? Any advice is much appreciated.
I'm new to using compojure, but have been enjoying using it so far. I'm
currently encountering a problem in one of my API endpoints that is generating
a large CSV file from the database and then passing this as the response body.
The problem I seem to be encountering is that the whole CSV file is being kept
in memory which is then causing an out of memory error in the API. What is the
best way to handle and generate this, ideally as a gzipped file? Is it possible
to stream the response so that a few thousand rows are returned at a time? When
I return a JSON response body for the same data, there is no problem returning
this.
Here is the current code I'm using to return this:
(defn complete
"Returns metrics for each completed benchmark instance"
[db-client response-format]
(let [benchmarks (completed-benchmark-metrics {} db-client)]
(case response-format
:json (json-grouped-output field-mappings benchmarks)
:csv (csv-output benchmarks))))
(defn csv-output [data-seq]
(let [header (map name (keys (first data-seq)))
out (java.io.StringWriter.)
write #(csv/write-csv out (list %))]
(write header)
(dorun (map (comp write vals) data-seq))
(.toString out)))
The data-seq is the results returned from the database, which I think is a
lazy sequence. I'm using yesql to perform the database call.
Here is my compojure resource for this API endpoint:
(defresource results-complete [db]
:available-media-types ["application/json" "text/csv"]
:allowed-methods [:get]
:handle-ok (fn [request]
(let [response-format (keyword (get-in request [:request :params :format] :json))
disposition (str "attachment; filename=\"nucleotides_benchmark_metrics." (name response-format) "\"")
response {:headers {"Content-Type" (content-types response-format)
"Content-Disposition" disposition}
:body (results/complete db response-format)}]
(ring-response response))))
Thanks to all the suggestion that were provided in this thread, I was able to create a solution using piped-input-stream:
(defn csv-output [data-seq]
(let [headers (map name (keys (first data-seq)))
rows (map vals data-seq)
stream-csv (fn [out] (csv/write-csv out (cons headers rows))
(.flush out))]
(piped-input-stream #(stream-csv (io/make-writer % {})))))
This differs from my solution because it does not realise the sequence using dorun and does not create a large String object either. This instead writes to a PipedInputStream connection asynchronously as described by the documentation:
Create an input stream from a function that takes an output stream as its
argument. The function will be executed in a separate thread. The stream
will be automatically closed after the function finishes.
Your csv-output function completely realises the dataset and turns it into a string. To lazily stream the data, you'll need to return something other than a concrete data type like a String. This suggests ring supports returning a stream, that can be lazily realised by Jetty. The answer to this question might prove useful.
I was also struggling with the streaming of large csv file. My solution was to use httpkit-channel to stream every single line of the data-seq to the client and then close the channel. My solution looks like that:
[org.httpkit.server :refer :all]
(fn handler [req]
(with-channel req channel (let [header "your$header"
data-seq ["your$seq-data"]]
(doseq [line (cons header data-seq)]
(send! channel
{:status 200
:headers {"Content-Type" "text/csv"}
:body (str line "\n")}
false))
(close channel))))
When defining a compojure handler e.g. by using the defroutes macro, I can do something like this:
(defroutes home-routes
(GET "/myhome/:id" [ id ] (home-page)))
(defn home-page [ id ]
( ... do something ... ))
So I know how to pass a piece of the path parameter. But imagine I want to return a HAL+JSON object with a selflink. How would I get defroutes to pass the whole URI to the home-page function?
The Ring request map contains all the necessary information to construct a "selflink". Specifically, :scheme, :server-name, :server-port, and :uri values can be assembled into full request URL. When I faced this problem I created Ring middleware that adds the assembled request URL to the Ring request map. I could then use the request URL in my handlers as long as I pass the request map (or some subset of it) into the handler. The following snippet shows one way of implementing this:
(defroutes app-routes
(GET "/myhome/:id" [id :as {:keys [self-link]}] (home-page id self-link))
(route/resources "/")
(route/not-found "Not Found"))
(defn wrap-request-add-self-link [handler]
(fn add-self-link [{:keys [scheme server-name server-port uri] :as r}]
(let [link (str (name scheme) "://" server-name ":" server-port uri)]
(handler (assoc r :self-link link)))))
(def app
(-> app-routes
handler/site
wrap-request-add-self-link))
I have a rather simple problem I thought. I have a luminus web project with clojurescript and cljs-ajax. What I do is an ajax request to my server and in my error-handler I want to parse a json string to display a specific error message. I just cannot get it working.
This is my code:
Clojurescript:
(defn error-handler [response]
(js/alert (aget response "message")))
; (js/alert (:message response ))) neither works here
(defn test-connection []
(POST "/url"
{:params (get-form-data)
:response-format :json
:handler succ-handler
:error-handler error-handler}))
And my serverside code:
(defn connect-jira []
{:body {:message "No connection to JIRA. Please check your details."}}
)
(defroutes jira-routes
(POST "url" []
(connect-jira)))
What is the idiomatic way to return a custom error message and parse that with clojurescript?
Update
I tracked my problem down some more. There is a difference between the success and the error handler of the POST request. Lets say my server side looks like this:
(defn connect-jira []
{:status 200 :body {:message "No connection to JIRA. Please check your details."}})
And my success handler like this:
(defn succ-handler [{:keys [message]}]
(.log js/console message))
Then its all fine, the correct messages is logged to the console.
But if my server side looks like this (different statu):
(defn connect-jira []
{:status 500 :body {:message "No connection to JIRA. Please check your details."}})
I call the error handler on cljs side
(defn err-handler [{:keys [message]}]
(.log js/console message))
But the message is "null"
I also tried to overwrite the status-text on server side, but did not succeed. So, how do I pass a message to the error handler from server side?
Are you using this library for your AJAX calls?
https://github.com/yogthos/cljs-ajax
If so, your error handler would need to look something more like this:
(defn err-handler [{:keys [response]}]
(.log js/console (:message response)))
I am trying to POST some JSON data to a web service using drakma.
(ql:quickload :st-json)
(ql:quickload :cl-json)
(ql:quickload :drakma)
(defvar *rc* (merge-pathnames (user-homedir-pathname) ".apirc"))
(defvar *user*
(with-open-file (s *rc*)
(st-json:read-json s)))
(defvar api-url (st-json:getjso "url" *user*))
(defvar api-key (st-json:getjso "key" *user*))
(defvar api-email (st-json:getjso "email" *user*))
(setf drakma:*header-stream* *standard-output*)
(defvar *req* '(("date" . "20071001") ("time" . "00") ("origin" . "all")))
(format t "json:~S~%" (json:encode-json-to-string *req*))
(defun retrieve (api request)
(let* ((cookie-jar (make-instance 'drakma:cookie-jar))
(extra-headers (list (cons "From" api-email)
(cons "X-API-KEY" api-key)))
(url (concatenate 'string api-url api "/requests"))
(stream (drakma:http-request url
:additional-headers extra-headers
:accept "application/json"
:method :post
:content-type "application/json"
:external-format-out :utf-8
:external-format-in :utf-8
:redirect 100
:cookie-jar cookie-jar
:content (json:encode-json-to-string request)
:want-stream t)))
(st-json:read-json stream)))
(retrieve "/datasets/tigge" *req*)
Unfortunately, I get an error, although the data seems to be encoded OK to JSON and the headers generated by drakma too, I think. Apparently something is wrong with the :content (the list of integers in the errors message is just the list of ASCII codes of the JSON encoded data).
json:"{\"date\":\"20071001\",\"time\":\"00\",\"origin\":\"all\",\"type\":\"pf\",\"param\":\"tp\",\"area\":\"70\\/-130\\/30\\/-60\",\"grid\":\"2\\/2\",\"target\":\"data.grib\"}"
POST /v1/datasets/tigge/requests HTTP/1.1
Host: api.service.int
User-Agent: Drakma/1.3.0 (SBCL 1.1.5; Darwin; 12.2.0; http://weitz.de/drakma/)
Accept: application/json
Connection: close
From: me#gmail.com
X-API-KEY: 19a0edb6d8d8dda1e6a3b21223e4f86a
Content-Type: application/json
Content-Length: 193
debugger invoked on a SIMPLE-TYPE-ERROR:
The value of CL+SSL::THING is #(123 34 100 97 116 97 115 101 116 34 58 34
...), which is not of type (SIMPLE-ARRAY
(UNSIGNED-BYTE 8)
(*)).
Any idea what's wrong with this code? Many thanks in advance.
Thanks to Kevin and Hans from the General interest list for Drakma and Chunga drakma-devel for helping me out - it turned out that the problem was caused by a bug in a recent version of cl+ssl, already fixed in a development branch. I use quicklisp, and here is what Hans Hübner advised my to do to update my cl+ssl installation, which worked:
You can check out the latest cl+ssl - which contains a fix for the
problem:
cd ~/quicklisp/local-projects/
git clone git://gitorious.org/cl-plus-ssl/cl-plus-ssl.git
Quicklisp will automatically find cl+ssl from that location. Remember
to remove that checkout after you've upgraded to a newer quicklisp
release that has the fix in the future.