Is there a way to serialize functions at runtime in Clojure? I'd like to be able to send stateless (but not pure) functions over the wire in a serialized format (probably edn, but I'm open to anything).
For example...
If I run prn-str on a function, I don't get what I expected/wanted.
user=> (def fn1 (fn [x] (* x 2)))
#'user/fn1
user=> (def data {:test 1 :key "value"})
#'user/data
user=> (defn fn2 [x] (* x 2))
#'user/fn2
user=> (prn-str fn1)
"#object[user$fn1 0x28b9c6e2 \"user$fn1#28b9c6e2\"]\n"
user=> (prn-str data)
"{:test 1, :key \"value\"}\n"
user=> (prn-str fn2)
"#object[user$fn2 0x206c48f5 \"user$fn2#206c48f5\"]\n"
user=>
I would have wanted/expected something like this:
user=> (prn-str fn2)
"(fn [x] (* x 2))\n"
or, maybe,
user=> (prn-str fn2)
"(defn fn2 [x] (* x 2))\n"
You would have to use quote or ' to prevent evaluation and eval to force evaluation:
(def fn1 '(fn [x] (* x 2)))
(prn-str fn1) ;;=> "(fn [x] (* x 2))\n"
((eval fn1) 1) ;;=> 2
Flambo, a Clojure wrapper for Spark, uses the serializable-fn library to serialize functions (which Spark requires). Sparkling, another wrapper for Spark, uses native Clojure functions through this Java abstract class that implements the Java interface Serializable.
You have basically two choices:
pass source code (s-expressions stored as clojure data)
pass jar files and load them on the other side.
for the first option you save the source at the time the function is compiles (almost always when it is defined) and then pass the same source expression to the other computer and let it compile the same thing. so first you might make a vector of expressions:
(domain-functions '[(defn foo [x] x)
(defn bar [y] (inc y)]
then you can store this into a database and each client can pass it to read and then they will all have the same functions.
The second option depends on the fact that each time you define a function it produces a class file in the /target directory and then loads it. You can then syncronize this directory and load them on the other side. This approach is of course completely crazy, though people do crazy stuff around here. I recommend the first approach
And as a personal note:
I'm doing this now with datomic, and I have adopted the practice of putting the git-hash into the function name using a macro so I know absolutly for certain that when I call a function, I'm getting the same function I see in the editor. This brings peace of mind when running many instances that all pull from the same DB.
At some point it ceases to be Clojure, so the expectation that we can arbitrarily round trip from source to machine instructions and back is a little bit off.
We should be able to serialize a function to a byte array and send that across the wire though. I suspect you'd need to grab the function's java.lang.Class object and then pass that through a java.lang.instrument.ClassFileTransformer to get the bytes. Once you have those you can pass them through to the friendly java.lang.ClassLoader on the remote jvm.
You could use clojure.repl/source.
(with-out-str (source filter))
to get a string, or
(read-string (with-out-str (source filter)))
to get a clojure list.
There really isn't a good way, and for good reason simply shipping a function to another computer or storing it in a DB can cause lots of problems, not the least of which is that that function may require other functions that aren't on the other end.
A much better idea is to stick with data. Instead of writing the function, write the name of the function as a event, and then that even can be translated later by whatever is reading your data. Stick with data, that's the idiomatic way.
Related
I'm studying Clojure, and I've read that in Clojure a function definition is just data, i.e. parameters vector is just an ordinary vector. If that's the case, why can I do this
(def add (fn [a b]
(+ a b)))
but not this
(def vector-of-symbols [a b])
?
I know I normally would have to escape symbols like this:
(def vector-of-symbols [`a `b])
but why don't I have to do it in fn/defn? I assume this is due to fn/defn being macros. I tried examining their source, but they are too advanced for me so far. My attempts to recreate defn also fail, and I'm not sure why (I took example from a tutorial):
(defmacro defn2 [name param & body]
`(def ~name (fn ~param ~#body)))
(defn2 add [a b] (+ a b)) ;;I get "Use of undeclared Var app.core/defn2"
Can someone please explain, how exactly does Clojure turn data structures, especially symbols, into code? And what am I missing about the macro example?
Update Apparently, macro does not work because my project is actually in Clojurescript (in Clojure it does work). I did not think it matters, but as I progress - I discover more and more things that somehow don't work for me in with Clojurescript.
Update 2 This helps: https://www.clojurescript.org/about/differences
A function is a first-class citizen as other data in Clojure.
To define a vector you use (vector ...) or reader has syntaxic sugar [...], for a list it's (list ...) or '(...) the quote not to evaluate the list as a function call, for a set (set ...) or #{...}.
So the factory function for a function is fn (in fact fn*, that comes from Java core of Clojure, fn is a series of macros to manage to destructure and all).
(fn args body)
is a function call that returns a function, where args is a vector of argument(s) event. empty and body is a series of Clojure expressions to be evaluated with args bind to the environment. If nothing is to be evaluated it returns nil. There is also a syntactic sugar #(...) with %x as argument x and % as argument 1.
(fn ...) return a value that is a function. So
(def my-super-function (fn [a b c d] (println "coucou") (+ a b c d)))
binds the symbol my-super-function with the anonymous function returned by (fn [a b c d] (println "coucou") (+ a b c d)).
(def my_vector [1 2 3])
binds the symbol my_vector with the vector [1 2 3]
List of learning resources: https://github.com/io-tupelo/clj-template#documentation
As #jas said, your defn2 macro looks fine.
The main point is that macros are an advanced feature that one almost never needs. A macro is equivalent to a compiler extension, and that is almost never the best solution to a problem. Also keep in mind that functions can do some things macros can't.
Another point: the syntax-quote (aka backquote) ` is very different from a single quote '. In your example you want the single quote for ['a 'b]. Even better would be to quote the entire vector form '[a b].
As to your primary question, it is poorly explained how source-file text is converted into code. This is a 2-step process. The Clojure Reader consumes text string data (from a file or a literal string) and produces data structures like lists, vectors, strings, numbers, symbols. The Clojure compiler takes these data structures as input and produces java byte code that can be executed.
It is confusing because, when printed, one can't tell the difference between the text representation of a vector [1 2 3] and the text string that is input to the reader [1 2 3]. Ideally it would be color-coded or something. This problem doesn't exist in Java, etc since they don't have macros and hence there is no confusion between the source code (text) and the data structures used by a macro (not text).
For a more detailed answer on creating macros in Clojure, please see this answer.
I'm attempting to use a string saved in a variable to call a function like so:
(defn update-product-list [] "Test")
(defn handle-state-change [action]
((resolve (symbol action))))
(handle-state-change "update-product-list")
However, this gives me the following error: Assert failed: Argument to resolve must be a quoted symbol
I've also tried changing the above line to:
((resolve (quote (symbol action))))
But this still gives an error. I also tried changing it just to:
((resolve 'action))
But this gives a different error I don't quite understand: js/action is shadowed by a local. I don't want to override the function just call it. Not sure where I'm going wrong. I've looked at a few examples, but can't see to pin it down.
ClojureScript supports :advanced optimization, in which Google Closure Compiler will rename, inline, or eliminate (unused) functions in order to implement minification. In short, the name of the function you want to look up will, in general, simply no longer exist under :advanced.
Because of this, ClojureScript's resolve is a compile-time facility (a macro requiring a literal quoted symbol).
If you are using :simple or self-hosted ClojureScript, more options are available to you because the support needed persists into runtime. For example Planck has a planck.core/resolve that behave's like Clojure's resolve. A similar approach is possible in Lumo, and similar facilities can be fashioned if using :simple.
In general though, given :advanced, if you need to map strings to a set of functions, you need to somehow arrange to have a static mapping constructed at compile time to support this (the set of functions must be known a priori, at compile time).
If you have a namespace (the name of which is statically known at compile time) which defines functions that need to be dynamically called via strings, you could consider making use of ns-publics:
cljs.user=> (ns foo.core)
foo.core=> (defn square [x] (* x x))
#'foo.core/square
foo.core=> (in-ns 'cljs.user)
nil
cljs.user=> (when-some [fn-var ((ns-publics 'foo.core) (symbol "square"))]
(fn-var 3))
9
This will work under :advanced. The mapping constructed by ns-publics is static; built at compile-time. If you have multiple namespaces that need such treatment, you could merge several calls to ns-publics to build a larger map.
The advantage of this approach is that the code involved is pretty short and requires little maintenance. The disadvantage is that it dumps all of the public vars of the namespace (foo.core in this example) into your generated code (and the generated code for vars is somewhat verbose). Another disadvantage is that you need to statically know the namespace(s) involved at compile time.
If you need to further minimize generated code size, you could just build / maintain a simple static map from string to function value as in
(def fns {"square" foo.core/square})
and use it appropriately, keeping it up to date as your codebase evolves.
Another option would be to mark the functions that you need to access using ^:export meta, and then to call those functions using JavaScript interop. For example if you define the function this way
(defn ^:export square [x] (* x x))
then you can use strings / interop to lookup the function and call it at runtime. Here's an example:
((goog.object/getValueByKeys js/window #js ["foo" "core" "square"]) 3)
The use of ^:export and :advanced is covered here. If you know that you are using :simple or less, then you can simply use JavaScript interop to call the functions of interest, without needn't to use ^:export.
Note that there is no general solution that would let you look up a function by name at runtime under :advanced without somehow putting some aspect of that function into your code at compile time. (In fact, if a function is not referenced in a way that Google Closure Compiler can statically, see, the function implementation will be completely eliminated as dead code.) In the above, ns-publics grabs all the vars for a namespace at compile time, rolling your own lookup map sets up static code to refer to the function value, and using ^:export statically arranges to make the name of the function persist into runtime.
You need to use it like this:
((resolve 'inc) 5)) => 6
or, deconstructed a bit:
(let [the-fn (resolve 'inc)]
(the-fn 7))
=> 8
If you have the function name as a string, use the symbol function to convert from string => symbol (from clojuredocs.org):
user=> ((-> "first" symbol resolve) [1 2 3])
1
And, never forget the Clojure CheatSheet!
In advanced compilation
(js/console.log "HELLO"
js/window.navigator.msSaveBlob
(.. js/window -navigator -msSaveBlob)
(aget js/window "navigator" "msSaveBlob")
js/console.log)
=>
HELLO undefined undefined function function
I think this means that js/console has some provided externs, but navigator does not (or at least not the ms specific stuff).
AFAIK the only way to avoid this is to create some externs? But this seems unnecessarily obtuse; why would you ever want js/anything to be munged?? Wouldn't it make make more sense to never munge js/anything interop?
System functions are not munged; only your own functions are. You probably want (.log js/console ...) ?
For de-munging your own functions, place ^:export between the defn and the function name to export its name intact.
Here is more information.
All see the section called "munging" here.
Context: I have a clojure-based crossword app whose main ui is a JTabbedPane with two tabs, a grid and a clue table. The clue table is a view over a vector of clues, but the vector itself is not the authoritative store of the data, but dynamically generated from a couple of internal data structures via an (active-cluelist) function, triggered by the clue tab being selected.
So this is the implementation of the clue table:
(def cluelist [])
(def update-cluelist)
(def model)
(defn make []
(let [column-names ["Sq" "Word" "Clue"]
column-widths [48 200 600]
table-model (proxy [AbstractTableModel] []
(getColumnCount [] (count column-names))
(getRowCount [] (count cluelist))
(isCellEditable [row col] (= col 2))
(getColumnName [col] (nth column-names col))
(getValueAt [row col] (get-in cluelist [row col]))
(setValueAt [s row col]
(let [word (get-in cluelist [row 1])]
(add-clue word s) ; editing a cell updates the main clue data
(def cluelist (assoc-in cluelist [row 2] s))
(. this fireTableCellUpdated row col))))
table (JTable. table-model)
]
; some pure display stuff elided
(def model table-model)
)
(defn update-cluelist []
(def cluelist (active-cluelist))
(.fireTableDataChanged model))
Someone in another discussion noted that it is a major code smell for (update-cluelist) to be manually calling fireTableDataChanged, because nothing outside the TableModel class should ever be calling that method. However, I feel this is an unavoidable consequence of the table being dynamically generated from an external source. The docs aren't too helpful - they state that
Your custom class simply needs to invoke one the following
AbstractTableModel methods each time table data is changed by an
external source.
which implicitly assumes that the CustomTableModel class is the authoritative source of the data.
Also there is a bit of a clojure/java impedance mismatch here - in java I would have had cluelist and update-cluelist be a private member and method of my TableModel, whereas in clojure cluelist and the table model are dynamically scoped vars that update-cluelist has access to.
My main problem is that there is not a lot of clojure/swing code around that I can look to for best practices. Does anyone have any advice as to the best way to do this?
Suggestion: use an atom for cluelist. Constantly redefining cluelist is not the right way to represent mutable data. Honestly, I would expect it to throw an exception the second time you define cluelist.
If you use an atom for cluelist, you can call the fireTableDataChanged method from a watcher instead of calling it manually. This would mean that anytime (and anywhere) you change the atom, fireTableDataChanged will be called automatically, without an explicit call.
The issue with def is that calling def multiple times doesn't work well in a multi-threaded environment and Clojure tries to make everything default to fairly threadsafe. As I understand it, the "proper" way to use a var is to leave its root binding alone (ie, don't call def again) and use binding if you need to locally change it. def may work the way you are using it, but the language is set up to support atoms, refs, or agents in this sort of situation and these will probably work better most of the time (ie you get watchers). Also, you don't need to worry at all about threads if you add them later.
can i somehow find all functions/macros that take a specific type of parameter ?
for example, what function accepts a Namespace object as parameter ?
(this is because i can create a namespace and store it in a var, but i don't know where i could use that var; what function might i pass that var to ?)
here is some code:
user=> (def working-namespace (create-ns 'my-namespace))
#'user/working-namespace
;i created a namspace and want to use it later
user=> (class working-namespace)
clojure.lang.Namespace
; out of curiosity i found out that "working-namespace" is a Namespace object
user=> (ns working-namespace)
nil
working-namespace=>
; but how do i switch to it ? this didn't do what i wanted...
user=> (refer working-namespace)
java.lang.ClassCastException: clojure.lang.Namespace cannot be cast to clojure.lang.Symbol (NO_SOURCE_FILE:0)
; this did not work either as my expectations
user=> (the-ns working-namespace)
#<Namespace my-namespace>
user=> (class (the-ns working-namespace))
clojure.lang.Namespace
; great, this gave me the same thing, a Namespace
hence the question: how do i use it dynamically (that's why i had put my namespace into a var) ? how do i get something useful for me from a var that points to a namespace ?
i can try look around for functions that make use of a Namespace object or that convert it to something else. i did and only found "intern". searching by hand not seems not that promising
what if i have this problem a million time ? is there an automated way to get me what i'm looking for without having to ask around each time ?
In Clojure 1.2 and previous function arguments dont have types. every function argument is an object. So the question really becomes "how do i find functions that will cast the object I pass them into this type. so searching for type hints will find some of them, though it wont get you everything. I wish it where more possible to answer this in general.
starting with 1.3 (current dev branch 9/2010) function paramerters and return types can have a defined type and will be passed/returned as that type instead of being cast to object and then cast on the other side. This drops one of the zeros from the exacution time of numerical functions with the important limitation that it only works for :static functions and only with direct calls (ie: not through map/reduce/filter/etc.) There is not a lot published on this change yet though it has the important breaking change that integers are no longer boxed by default and integer (actually Long) overflow throws an exception. you can read more here
(defn ^:static fib ^long [^long n]
(if (<= n 1)
1
(+ (fib (dec n)) (fib (- n 2)))))
so after 1.3 is released and widely adopted you will see code with more commonly defined types because they will offer a big speed benefit and then you will be able to find more functions by argument type though still not all of them.
At the same lecture where I learned about function argument types, Rich mentioned plans in the distant Clojure future (after 'Clojure in Clojure') about better support for exposing the compiler internals to tools such as IDEs. So there is hope that someday you will get a real answer to this question.
Dynamic languages make this slightly more difficult in practice and a lot harder in theory.
You already got a good answer from Arthur, so I'll only answer the "how do i get something useful for me from a var that points to a namespace ?". From (doc ns), note that it says unevaluated:
user=> (doc ns)
-------------------------
clojure.core/ns
([name docstring? attr-map? references*])
Macro
Sets *ns* to the namespace named by name (unevaluated), creating it
Now there's something you could do with in-ns if you want (the whole namespace object -> string -> symbol conversion is probably stupid, but enough to illustrate my point):
user=> (in-ns (symbol (str working-namespace)))
#<Namespace my-namespace>
my-namespace=>
I don't think you can do that without a lot of hackery in a dynamic language. If you want to now what function that take namespaces look at the documentation of the namespace stuff.
For example cleaning namespaces or reload them.
You wrote:
user=> (ns working-namespace)
nil
working-namespace=>
; but how do i switch to it ? this didn't do what i wanted...
But you did switch to the working-namespace namespace (that's why the prompt changed), so I'm not clear as to what "you wanted".
As I noted earlier, you need to present the ultimate problem are you trying to solve. It's entirely likely that messing with namespace objects won't be the solution.