I have a result set in json with city names, and I'd like to get the lat-long for each. The following function works to an extent:
(:require [http.async.client :as client])
(defn get-geo-fact [row]
(let [
n (string/replace (:cityname row) " " "+")
url (str "http://maps.googleapis.com/maps/api/geocode/json?address="
n "&sensor=false")
resp (client/GET url) ]
(client/await resp)
(make-geo-fact row (json/read-json (client/string resp))) ))
That last call to make-geo-fact just returns an RDF rendering of the city coordinates. The problem I'm running into is that a run (of about 40 calls to this function) returns a few (3-5 lat-long pairs) null results for lat-longs. The cities that return null values differ from run to run - sometimes San Jose gets coordinates, and sometimes it doesn't.
I originally used slurp to grab the body of the url and got similarly-occasional nulls. I figured I wasn't waiting for the response properly, but switching to http.async.client doesn't seem to be doing the trick. Any ideas?
edit:
Here's the make-geo-fact function, which takes a "this team is located in this city" fact and a response from google maps, returning a vector of two triples that convey the latitude and longitude:
(defn make-geo-fact [row response]
(let [ g (:location (:geometry (first (:results response))))
lat (str "<" (:team row) ">"
" <http://www.nhl.com/latitude> \"" (:lat g)
"\"^^<http://www.w3.org/2001/XMLSchema#decimal> ." )
lon (str "<" (:team row) ">"
" <http://www.nhl.com/longitude> \"" (:lng g)
"\"^^<http://www.w3.org/2001/XMLSchema#decimal> ." ) ]
[lat lon] ))
And here's the function I call to kick the whole thing off:
(defn make-geo-facts []
(let [ a (bounce team-city (build "files/team-city.nt"))
f "files/geo-facts.nt" ]
(spit f (string/join "\n" (flatten (map get-geo-fact (:rows a)))))
f ))
Where the bounce function issues a SPARQL select query against an RDF model, which is instantiated with the build function.
edit 2
Here's a re-factor where make-geo-fact isn't needed:
(defn get-geo-fact [row]
(let [ n (string/replace (:cityname row) " " "+")
url (str "http://maps.googleapis.com/maps/api/geocode/json?address=" n "&sensor=false")
resp (client/GET url) ]
(-> (client/GET url)
client/await
client/string
json/read-json
:results
first
:geometry
:location )))
(defn make-geo-facts []
(let [ a (bounce tc-query (build "files/team-city.nt"))
f "files/geo-facts.nt"
*client* (client/create-client)]
(try (spit f (string/join "\n" (flatten (map get-geo-fact (:rows a))))))
(finally (client/close *client*)) ))
As you've said changing client implementations didn't make a difference.
I double checked and create a test for development version of http.async.client.
And always got responses with body.
Please provide make-geo-fact implementation.
Turns out my code needed a little sleep. I added (Thread/sleep 1000) to my core function and now I don't get null results:
(defn get-geo-fact [row]
(let [ n (string/replace (:cityname row) " " "+")
url (str "http://maps.googleapis.com/maps/api/geocode/json?address=" n "&sensor=false")
resp (client/GET url) ]
(Thread/sleep 1000)
(-> (client/GET url)
client/await
client/string
json/read-json
(make-geo-fact ,,, row ) )))
Related
I have a database that returns data as a tree like this:
'((7 "vince" "vince1#test.com" "space" "no value" 1)
(8 "vince" "vince2#test.com" "place" "no value" 1)
(9 "Smith" "Smith#gmail.com" "now" "no value" 1))
The second column is first name and the third column is email.
My goal is to return JSON key value pairs but im struggling
Here is what I have tried:
Function to get name and email from one list item
(defun get-name-&-emails-db1 (lst)
(if (null lst)
nil
(let* ((name (second lst))
(email (third lst)))
(cl-json:encode-json-to-string `((:name . ,name)(:email . ,email))))))
Map over data set
(mapcar #'get-name-&-emails-db1 (return-data-tree))
This returns a list of individual json blocks. But I want it to be ONE json block with all records.
What am I missing?
(ideally, I want to know how to do this without any additional libraries)
Thanks
I tried to encode a list of alists, and this is how it goes:
USER> (cl-json:encode-json
(list '(("a" . "0") ("b" . "1")) '(("a" . "2") ("b" . "3"))))
[{"a":"0","b":"1"},{"a":"2","b":"3"}]
If this is what you want to have, then you need to organize your data in Lisp first, then encode the whole list as JSON instead of formatting each entry individually.
Use mapcar, get the second and third element of each entry, and then call cl-json:encode-json-to-string on the result:
(let ((data '((7 "vince" "vince1#test.com" "space" "no value" 1)
(8 "vince" "vince2#test.com" "place" "no value" 1)
(9 "Smith" "Smith#gmail.com" "now" "no value" 1))))
(cl-json:encode-json-to-string
(mapcar (lambda (e) `((:name . ,(second e))(:email . ,(third e))))
data)))
Here I don't use comma, backquote, alists or plists, but simply: I create a list of hash-tables. I'm quite sure how a list and a hash table are rendered in JSON, so let's rework our data a bit to come back in known territories.
(loop for row in '((7 "vince" "vince1#test.com" "space" "no value" 1)
(8 "vince" "vince2#test.com" "place" "no value" 1)
(9 "Smith" "Smith#gmail.com" "now" "no value" 1))
with result = (list) ;; the list of hash-tables to encode.
for ht = (make-hash-table) ;; intermediary hash-table.
do (setf (gethash "name" ht)
(second row)
(gethash "email" ht)
(third row))
(push ht result)
finally (return (cl-json:encode-json-to-string result)))
;; =>
"[{\"name\":\"Smith\",\"email\":\"Smith#gmail.com\"},{\"name\":\"vince\",\"email\":\"vince2#test.com\"},{\"name\":\"vince\",\"email\":\"vince1#test.com\"}]"
I like Serapeum's dict:
;; replaces (for ht = (make-hash-table)) and the setf
for ht = (dict :name (second row)
:email (third row))
Answers were given. Just a general way to deal with alist and json:
(ql:quickload :yason)
(defparameter *data* '((7 "vince" "vince1#test.com" "space" "no value" 1)
(8 "vince" "vince2#test.com" "place" "no value" 1)
(9 "Smith" "Smith#gmail.com" "now" "no value" 1)))
(defparameter *scheme* '(:id :name :email :meta :value :count))
(defun pairing (keys values)
(loop for a in keys
for b in values
collect (cons (string-downcase (format nil "~A" a)) b)))
(defun alist-json (x &keys keys)
(with-output-to-string (*standard-output*)
(yason:encode-alist (pairing keys x))))
(defun list-json (l)
(format nil "[~{~A~^, ~}]" l))
(defun values-list-keys-json (values-list keys)
(list-json (mapcar (lambda (x) (alist-json x :keys keys)) values-list)))
(values-list-keys-json *data* *scheme*)
#|
=> "[{\"id\":7,\"name\":\"vince\",\"email\":\"vince1#test.com\",\"meta\":\"space\",\"value\":\"no value\",\"count\":1},
{\"id\":8,\"name\":\"vince\",\"email\":\"vince2#test.com\",\"meta\":\"place\",\"value\":\"no value\",\"count\":1},
{\"id\":9,\"name\":\"Smith\",\"email\":\"Smith#gmail.com\",\"meta\":\"now\",\"value\":\"no value\",\"count\":1}]"
|#
(defun second-third (l)
(subseq l 1 3))
(values-list-keys-json (mapcar #'second-third *data*) (second-third *scheme*))
#|
=> "[{\"name\":\"vince\",\"email\":\"vince1#test.com\"}, {\"name\":\"vince\",\"email\":\"vince2#test.com\"}, {\"name\":\"Smith\",\"email\":\"Smith#gmail.com\"}]"
|#
As a minimal example of what I want to do:
(defn mkfn [func]
(fn func [a] (print "I am a function")))
(mkfn 'x) ; => #function[user/mkfn/func--10871]
(type x)
(x)
The last two both result in:
Syntax error compiling at (conjure-log-12628.cljc:1:1).
Unable to resolve symbol: x in this context
I'm not sure why this doesn't work since fn takes symbols as input and 'x is a symbol. I'm also not sure how to accomplish this task.
For that matter:
user=> (def (eval 'y) 3)
Syntax error compiling def at (conjure-log-12628.cljc:1:1).
user=> (def 'y 3)
Syntax error compiling def at (conjure-log-12628.cljc:1:1).
First argument to def must be a Symbol
First argument to def must be a Symbol
user=> (type 'y)
clojure.lang.Symbol
Other things that don't work:
(defn mkfn [func]
(fn (sympol func) [a] (print "i am a function")))
(symbol "y") ; => y ; a symbol
(def (symbol "y") 3) ; => an err
You will probably need a macro. It seems that you want to call that function by the provided name, so you also have to replace fn with defn.
And you have to be careful about a number of arguments, because function x with argument vector [a] must be called with one argument, and not like (x).
(defmacro mkfn [func]
`(defn ~func [~'a]
(print "I am a function")))
(mkfn x)
=> #'user/x
(x 1)
I am a function=> nil
There is also other way, using intern, so you can completely avoid writing macros:
(intern *ns* 'x (fn [a] (print "I am a function")))
=> #object...
(x 1)
I am a function=> nil
Example with intern:
(defn mkfn [func]
(intern *ns* func (fn [a] (print "I am a function"))))
=> #'user/mkfn
(mkfn 'y)
=> #'user/y
(y 1)
I am a function=> nil
As for your errors, def is a special form, so it has different evaluation rules. It doesn't evaluate the first argument, which has to be a symbol- and (unevaluated) (eval 'y), 'y or (symbol "y") aren't symbols, while y is.
You gonna need a macro for that since you need code writing code.
(defmacro mkfn [func]
`(fn ~func [~'a] ...))
There 2 ways of doing it, either function plus eval or macro. If you really want to programatically create a new function with your chosen name, the macro solution is the way to go.
The function + eval solution is instructive, but you'll have to either quote the function name when you call it (via a 2nd eval) or save the created function in another variable when you create it.
If you are interested in writing macros, please see this other question first: How do I write a Clojure threading macro?
For the function + eval, we can start with my favorite template project and add the following:
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test))
(defn maker-eval
[fn-sym]
(let [ll (list 'fn 'anon-fn [] (str "I am " fn-sym))]
(spyx :eval ll)
(eval ll)))
(verify
(let [anon-fn-1 (maker-eval 'aaa)]
(is= "I am aaa" (anon-fn-1))) ; need the temp local variable
(let [anon-fn-2 (maker-eval 'bbb)]
(is= "I am bbb" (anon-fn-2))) ; need the temp local variable
)
and we can see the creation and use of the function, along with printed output:
:eval ll => (fn anon-fn [] "I am aaa")
:eval ll => (fn anon-fn [] "I am bbb")
For the macro version, we type
(defn maker-macro-impl
[fn-sym]
(let [ll `(defn ~fn-sym [] (str "I am " (str (quote ~fn-sym))))]
(spyx :macro ll)
ll))
(defmacro maker-macro
[fn-sym] (maker-macro-impl fn-sym))
(verify
(let [anon-fn-3 (maker-macro-impl 'ccc)]
(is= anon-fn-3 (quote
(clojure.core/defn ccc [] (clojure.core/str "I am " (clojure.core/str (quote ccc)))))))
(maker-macro ddd)
(is= (ddd) "I am ddd"))
and see printed:
:macro ll => (clojure.core/defn ccc [] (clojure.core/str "I am " (clojure.core/str (quote ccc))))
Note that the local variable anon-fn-3 was only used to test the maker-macro-impl function, but was not needed to call the newly-created function ddd
at the end of the unit test.
I have an array of art pieces. I want to find the route length and associate it with each art pieces.
My code will look like:
(defn load-art-routes [art-list ctx]
(doall (map-indexed (fn [index art]
(let [user-location (select-keys (:coords (sub> ctx :geolocation)) [:latitude :longitude])
art-location (:geoLocation art)]
(->> (map-request {:origin (str (:latitude user-location) "," (:longitude user-location))
:destination (str (:lat art-location) "," (:lon art-location))
:mode (name (sub> ctx :transition-mode))})
(p/map (fn [data]
(let [route-length (ocall js/Math "round" (/ (get-in data [:routes 0 :legs 0 :distance :value]) (* 0.621371192 1000)) 2)
route-duration (ocall js/Math "floor" (/ (get-in data [:routes 0 :legs 0 :duration :value]) 60))]
(js/console.log "load-art-routes route-length " route-length")
(assoc art :route-length route-length))))
(p/error (fn [error]
(util/log (str "GOOGLE DIRECTIONS API ERRORS" params) error)
))))) art-list))
art-list)
(defn map-request [params]
(when params
(let [endpoint google-directions-api-endpoint]
(->> (make-get-req (str endpoint "&" (encode-query-params params))
{})
(p/map (fn [data]
(util/log "GOOGLE DIRECTIONS API " data)
data))
(p/error (fn [error]
(util/log (str "GOOGLE DIRECTIONS API ERRORS" params ) error)
))))))
The route length calculation is correct but, assoc is not working. It is not actually associating it. I don't know what the issue is. Can anyone help me?
Please simplify this example! In the process, you will discover the problem.
First, update your question to include the require that shows what p/map, p/error, etc are. Also, put map-request before load-art-routes just like it must be in your source file.
Then, you should start by removing the thread-last ->> operator and use let with names for intermediate values:
(let [aa (map-request ...)
bb (p/map (fn [data] ...) aa)
cc (p/error (fn [error] ...) bb) ]
<bb or cc here?> )
My suspicion is that your p/error call is swallowing the results of p/map and returning nil.
This looks like you are trying to write "mutable" code.
Reformatting the code and fixing one error makes this more obvious:
(defn load-art-routes [art-list ctx]
(doall (map-indexed (fn [index art]
(let [user-location (select-keys (:coords (sub> ctx :geolocation)) [:latitude :longitude])
art-location (:geoLocation art)]
(->> (map-request {:origin (str (:latitude user-location) "," (:longitude user-location))
:destination (str (:lat art-location) "," (:lon art-location))
:mode (name (sub> ctx :transition-mode))})
(p/map (fn [data]
(let [route-length (ocall js/Math "round" (/ (get-in data [:routes 0 :legs 0 :distance :value]) (* 0.621371192 1000)) 2)
route-duration (ocall js/Math "floor" (/ (get-in data [:routes 0 :legs 0 :duration :value]) 60))]
(js/console.log "load-art-routes route-length " route-length)
(assoc art :route-length route-length))))
(p/error (fn [error]
(util/log (str " GOOGLE DIRECTIONS API ERRORS " params) error)
))))) art-list))
art-list)
load-art-routes simply returns the original art-list and kicks of some work in promises. These promises only update their versions of the list but given that we are using immutable data structures the returned art-list themselves remain unchanged. There is also a suspicious second art-list in the p/error call?
You'll probably want to restructure this to something that either returns a Promise or accepts a callback that will be called once all route-length have been computed.
How do I read from an input text file that is 1 column of ids and produce a MySQL query of the format:
SELECT col1,col2,col3 FROM Orders WHERE Id IN ('inputId1','inputId3','inputId3');
The ids in the input file are separated by /n and should be converted into the comma separated list of Ids enclosed in quotes for the MySQL query.
(ns export.core
(:require [clojure.java.jdbc :as j])
(:gen-class))
(defn -main [& args]
;; Get home directory
(def out-file
(str (System/getProperty "user.home") "/Desktop/export.txt"))
(def in-file
(str (System/getProperty "user.home") "/Desktop/orders.txt"))
;; Get string of order-ids
(def order-ids-string (slurp in-file))
???????????
???????????
;; Connect to database
(def db {:subprotocol "mysql"
:subname "XXXXXXXX"
:user "XXXXXXX"
:password "XXXXXXX"})
;; Get headers
(def header-seq
(j/query db ["DESCRIBE Orders"] :row-fn :field))
(def header-str
(str (clojure.string/join "\t" header-seq) "\n"))
;; Get product results and spit data to file
(def header-keys
(into []
(map keyword
(map clojure.string/lower-case header-seq))))
(def data-seq
(j/query db [<needed sql query>]))
(defn select-values [map]
(reduce #(conj %1 (map %2)) [] header-keys))
(spit out-file header-str)
(doseq [row data-seq]
(spit out-file
(str (clojure.string/join "\t" (select-values row)) "\n")
:append true)))
If I've understood your question correctly I would use line-seq, string/join, and format to form the query:
first some test data:
(spit "/tmp/input-file" "id1\nid2\nid3\nid4\n")
then lets read it back and form a string
user> (let [ids (line-seq (clojure.java.io/reader "/tmp/input-file"))
col-names (clojure.string/join "," (map #(str "col" %) (range 1 (inc (count ids)))))
col-ids (clojure.string/join "," (map #(str "'"% "'") ids))]
(format "SELECT %s FROM Orders WHERE Id IN (%s);" col-names col-ids))
"SELECT col1,col2,col3,col4 FROM Orders WHERE Id IN ('id1','id2','id3','id4');"
I'm guessing that the number of order id's matches the number of lines in the file and that they should get sequential numbers in their names.
as amalloy points out it's basicly always better to use query params:
user> (let [ids (line-seq (clojure.java.io/reader "/tmp/input-file"))
col-names (clojure.string/join "," (map #(str "col" %) (range 1 (inc (count ids)))))
question-marks (clojure.string/join "," (repeat (count ids) "?"))]
(list 'exec-raw (format "SELECT %s FROM Orders WHERE Id IN (%s);" col-names question-marks) ids))
(exec-raw "SELECT col1,col2,col3,col4 FROM Orders WHERE Id IN (?,?,?,?);" ("id1" "id2" "id3" "id4"))
(replace list exec-raw with whatever function your use to make the SQL call)
I'm using secretary and reagent. This is my code :
(def view (atom nil))
(defn layout [view]
[:div #view])
(reagent/render-component [layout view] (.getElementById js/document "message"))
(secretary/set-config! :prefix "")
(secretary/defroute home-path "/" [query-params]
(timbre/info "Path : /, query params : " query-params)
(let [warning (:warning query-params)
success (:success query-params)
login-failed (:login_failed query-params)]
(when warning
(timbre/info "Warning found : " warning)
(reset! view [:h4 [:span.label.label-warning warning]]))
(when success
(timbre/info "Success found : " success)
(reset! view [:h4 [:span.label.label-info success]]))
(when login-failed
(timbre/info "Login failed")
(reset! view [:h4 [:span.label.label-warning "Login Failed."]]))))
(let [h (History.)]
(goog.events/listen h EventType.NAVIGATE #(secretary/dispatch! (.-token %)))
(doto h
(.setEnabled true)))
Disregarding the :prefix value (I tried "", "#" and also not setting the :prefix at all) this code only works with routes like :
http://localhost:8080/login#/?success=SuccessMessage
But it doesn't work with routes like :
http://localhost:8080/login?success=SuccessMessage
What I'm actually trying to achieve is to parse the login failure from friend, which in case of failure redirects me to
http://localhost:8080/login?&login_failed=Y&username=someUser
and display login failed message to the user. I don't need to use secretary for this, anything that works to parse the query-parameters would be ok for me.
The hard way would be to parse the query string which I can get with:
(-> js/window .-location .-search)
I believe that this is already done well in some library.
I found it. Using https://github.com/cemerick/url (works for both clojure and clojurescript), one can do :
(require '[cemerick.url :as url])
(:query (url/url (-> js/window .-location .-href)))
From the docs:
If a URI contains a query string it will automatically be extracted to :query-params for string route matchers and to the last element for regular expression matchers.
(defroute "/users/:id" [id query-params]
(js/console.log (str "User: " id))
(js/console.log (pr-str query-params)))
(defroute #"/users/(\d+)" [id {:keys [query-params]}]
(js/console.log (str "User: " id))
(js/console.log (pr-str query-params)))
;; In both instances...
(secretary/dispatch! "/users/10?action=delete")
;; ... will log
;; User: 10
;; "{:action \"delete\"}"