I am extremely new to the programming and to the elixir. So I am very exited to learn as much as I can. But I've got a problem. I looking the way how to use my functions in another module. I am building the web-server which stores the key-value maps in the memory. To keep the maps temporary I've decided to use Agent. Here is the part of my code:
defmodule Storage do
use Agent
def start_link do
Agent.start_link(fn -> %{} end, name: :tmp_storage)
end
def set(key, value) do
Agent.update(:tmp_storage, fn map -> Map.put_new(map, key, value) end)
end
def get(key) do
Agent.get(:tmp_storage, fn map -> Map.get(map, key) end)
end
end
So I'm trying to put this functions to the routes of the web server:
defmodule Storage_router do
use Plug.Router
use Plug.Debugger
require Logger
plug(Plug.Logger, log: :debug)
plug(:match)
plug(:dispatch)
post "/storage/set" do
with {:ok, _} <- Storage.set(key, value) do
send_resp(conn, 200, "getting the value")
else
_ ->
send_resp(conn, 404, "nothing")
end
end
end
And I receive:
warning: variable "key" does not exist and is being expanded to "key()", please use parentheses to remove the ambiguity or change the variable name
lib/storage_route.ex:12
warning: variable "value" does not exist and is being expanded to "value()", please use parentheses to remove the ambiguity or change the variable name
lib/storage_route.ex:12
looking for any suggestions\help
I am extremly new to the programming and to the elixir.
I do not think it is wise to begin learning programming with elixir. I would start with python or ruby, and then after a year or two then I would try elixir.
The first thing you need to learn is how to post code. Search google for how to post code on stackoverflow. Then, you have to get your indenting all lined up. Are you using a computer programming text editor? If not, then you have to get one. There are many free ones. I use vim, which comes installed on Unix like computers. You can learn how to use vim by typing vimtutor in a terminal window.
Next, you have a syntax error in your code:
Agent.start_link(fn -> %{} end, name: :tmp_storage
end)
That should be:
Agent.start_link(fn -> %{} end, name: :tmp_storage)
The warning you got is because your code tries to do the equivalent of:
def show do
IO.puts x
end
Elixir and anyone else reading that code would ask, "What the heck is x?" The variable x is never assigned a value anywhere, and therefore the variable x does not exist, and you cannot output something that is non-existent. You do the same thing here:
with {:ok, _} <- Storage.set(key, value) do
send_resp(conn, 200, "getting the value")
else
_->
send_resp(conn, 404, "nothing")
end
You call the function:
Storage.set(key, value)
but the variables key and value were never assigned a value, and elixir (and anyone else reading that code) wonders, "What the heck are key and value?"
This is the way functions work:
b.ex:
defmodule MyFuncs do
def show(x, y) do
IO.puts x
IO.puts y
end
end
defmodule MyWeb do
def go do
height = 10
width = 20
MyFuncs.show(height, width)
end
end
In iex:
~/elixir_programs$ iex b.ex
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
Interactive Elixir (1.6.6) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> MyWeb.go
10
20
:ok
iex(2)>
So, in your code you need to write something like this:
post "/storage/set" do
key = "hello"
value = 10
with {:ok, _} <- Storage.set(key, value) do
send_resp(conn, 200, "Server saved the key and value.")
else
_->
send_resp(conn, 404, "nothing")
end
end
However, that will store the same key/value for every post request. Presumably, you want to store whatever is sent in the body of the post request. Do you know the difference between a get request and a post request? A get request tacks data onto the end of the url, while a post request sends the data in the "body of the request", so there are different procedures for extracting the data depending on the type of the request.
What tutorial are you reading? This tutorial: https://www.jungledisk.com/blog/2018/03/19/tutorial-a-simple-http-server-in-elixir/, shows you how to extract the data from the body of a post request. The data in the body of a post request is just a string. If the string is in JSON format, then you can convert the string into an elixir map using Poison.decode!(), which will allow you to easily extract the values associated with the keys that you are interested in. For example:
post "/storage/set" do
{:ok, body_string, conn} = read_body(conn)
body_map = Poison.decode!(body_string)
IO.inspect(body_map) #This outputs to terminal window where server is running
message = get_in(body_map, ["message"])
send_resp(
conn,
201,
"Server received: #{message}\n"
)
end
Then you can use the following curl command in another terminal window to send a post request to that route:
$ curl -v -H 'Content-Type: application/json' "http://localhost:8085/storage/set" -d '{"message": "hello world" }'
(-v => verbose output, -H => request header, -d => data)
Now, based on what I said was wrong with your code above, you should be wondering about this line:
{:ok, body_string, conn} = read_body(conn)
That line calls:
read_body(conn)
but the variable conn is not assigned a value anywhere. However, Plug invisibly creates the conn variable and assigns a value to it.
Here is a complete example using Agent to store post request data (following the tutorial I linked above):
simple_server
config/
lib/
simple_server/
application.ex
router.ex
storage.ex
test/
An elixir convention is to have a directory in the lib/ directory with the same name as your project, in this case that would be simple_server, then you give the modules you define names that reflect the directory structure. So, in router.ex you would define a module named SimpleServer.Router and in storage.ex you would define a module named SimpleServer.Storage. However, the . in a module name means nothing special to elixir, so you will not get an error if you decide to name your module F.R.O.G.S in the file lib/rocks.ex--and your code will work just fine.
router.ex:
defmodule SimpleServer.Router do
use Plug.Router
use Plug.Debugger
require Logger
plug(Plug.Logger, log: :debug)
plug(:match)
plug(:dispatch)
get "/storage/:key" do
resp_msg = case SimpleServer.Storage.get(key) do
nil -> "The key #{key} doesn't exist!\n"
val -> "The key #{key} has value #{val}.\n"
end
send_resp(conn, 200, resp_msg)
end
post "/storage/set" do
{:ok, body_string, conn} = read_body(conn)
body_map = Poison.decode!(body_string)
IO.inspect(body_map) #This outputs to terminal window where server is running
Enum.each(
body_map,
fn {key, val} -> SimpleServer.Storage.set(key,val) end
)
send_resp(
conn,
201,
"Server stored all key-value pairs\n"
)
end
match _ do
send_resp(conn, 404, "not found")
end
end
The first thing to note in the code above is the route:
get "/storage/:key" do
That will match a path like:
/storage/x
and plug will create a variable named key and assign it the value "x", like this:
key = "x"
Also, note that when you call a function:
width = 10
height = 20
show(width, height)
elixir looks at the function definition:
def show(x, y) do
IO.puts x
IO.puts y
end
and matches the function call to the def like this:
show(width, height)
| |
V V
def show( x , y) do
...
end
and performs the assignments:
x = width
y = height
Then, inside the function you can use the x and y variables. In this line:
Enum.each(
body_map,
# | | | | |
# V V V V V
fn {key, val} -> SimpleServer.Storage.set(key,val) end
)
Elixir will call the anonymous function passing values for key and val, like this:
func("x", "10")
Therefore, in the body of the anonymous function you can use the variables key and val:
SimpleServer.Storage.set(key,val)
because the variables key and val will already have been assigned values.
storage.ex:
defmodule SimpleServer.Storage do
use Agent
def start_link(_args) do #<*** Note the change here
Agent.start_link(fn -> %{} end, name: :tmp_storage)
end
def set(key, value) do
Agent.update(
:tmp_storage,
fn(map) -> Map.put_new(map, key, value) end
)
end
def get(key) do
Agent.get(
:tmp_storage,
fn(map) -> Map.get(map, key) end
)
end
end
application.ex:
defmodule SimpleServer.Application do
# See https://hexdocs.pm/elixir/Application.html
# for more information on OTP Applications
#moduledoc false
use Application
def start(_type, _args) do
# List all child processes to be supervised
children = [
Plug.Adapters.Cowboy.child_spec(scheme: :http, plug: SimpleServer.Router, options: [port: 8085]),
{SimpleServer.Storage, []}
]
# See https://hexdocs.pm/elixir/Supervisor.html
# for other strategies and supported options
opts = [strategy: :one_for_one, name: SimpleServer.Supervisor]
Supervisor.start_link(children, opts)
end
end
mix.exs:
defmodule SimpleServer.MixProject do
use Mix.Project
def project do
[
app: :simple_server,
version: "0.1.0",
elixir: "~> 1.6",
start_permanent: Mix.env() == :prod,
deps: deps()
]
end
# Run "mix help compile.app" to learn about applications.
def application do
[
extra_applications: [:logger],
mod: {SimpleServer.Application, []}
]
end
# Run "mix help deps" to learn about dependencies.
defp deps do
[
{:poison, "~> 4.0"},
{:plug_cowboy, "~> 2.0"}
# {:dep_from_hexpm, "~> 0.3.0"},
# {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"},
]
end
end
Note, if you use the dependencies and versions specified in the tutorial you will get some warnings, including the warning:
~/elixir_programs/simple_server$ iex -S mix
...
...
12:48:57.767 [warn] Setting Ranch options together
with socket options is deprecated. Please use the new
map syntax that allows specifying socket options
separately from other options.
...which is an issue with Plug. Here are the dependencies and versions that I used to get rid of all the warnings:
{:poison, "~> 4.0"},
{:plug_cowboy, "~> 2.0"}
Also, when you list an application as a dependency, you no longer have to enter it in the :extra_applications list. Elixir will automatically start all the applications listed as dependencies before starting your application. See :applications v. :extra_applications.
Once the server has started, you can use another terminal window to send a post request with curl (or you can use some other program):
~$ curl -v -H 'Content-Type: application/json' "http://localhost:8085/storage/set" -d '{"x": "10", "y": "20" }
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8085 (#0)
> POST /storage/set HTTP/1.1
> Host: localhost:8085
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 23
>
* upload completely sent off: 23 out of 23 bytes
< HTTP/1.1 201 Created
< server: Cowboy
< date: Fri, 30 Nov 2018 19:22:23 GMT
< content-length: 34
< cache-control: max-age=0, private, must-revalidate
<
Server stored all key-value pairs
* Connection #0 to host localhost left intact
The > lines are the request, and the < lines are the response. Also, check the output in the terminal window where the server is running.
~$ curl -v http://localhost:8085/storage/z
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8085 (#0)
> GET /storage/z HTTP/1.1
> Host: localhost:8085
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< server: Cowboy
< date: Fri, 30 Nov 2018 19:22:30 GMT
< content-length: 25
< cache-control: max-age=0, private, must-revalidate
<
The key z doesn't exist!
* Connection #0 to host localhost left intact
.
~$ curl -v http://localhost:8085/storage/x
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8085 (#0)
> GET /storage/x HTTP/1.1
> Host: localhost:8085
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< server: Cowboy
< date: Fri, 30 Nov 2018 19:22:37 GMT
< content-length: 24
< cache-control: max-age=0, private, must-revalidate
<
The key x has value 10.
* Connection #0 to host localhost left intact
I'm not entirely sure what you're trying to accomplish, but the error is telling you that the key and value that are passed to the router with statement are not defined. Elixir thinks you are trying to call a function with those arguments because they are not "bound" to a value. That is why you are seeing warning: variable "value" does not exist and is being expanded to "value()"
I suppose this is not really an answer but maybe more an explanation of the error you're seeing.
You need to pull the key/value params out of your %Plug.Conn{} object (conn). The key/value variables have not yet been defined within the scope of your route. The conn object is only available because it is injected by the post macro provided by Plug.
I am not quite aware of what type of requests you're submitting to the router, but I'll assume it's JSON as an example. You can manually parse the body in your connection by doing something like:
with {:ok, raw_body} <- Plug.Conn.read_body(conn),
{:ok, body} <- Poison.decode(raw_body) do
key = Map.get(body, "key")
value = map.get(body, "value")
# ... other logic
end
The Plug project, however, provides a nice convenience plug for you to parse request bodies in a generic way: Plug.Parsers.
To implement this in your router, you just have to add the plug to the top of your router (below Plug.Logger I think):
plug Plug.Parsers,
parsers: [:urlencoded, :json]
json_decoder: Poison,
pass: ["text/*", "application/json"]
The :urlencoded part will parse your query parameters and the :json part will parse the body of the request.
Then below in your route, you can get the key/value params from your conn object in the :params key like so:
%{params: params} = conn
key = Map.get(params, "key")
value = Map.get(params, "value")
Also, I should note that the best JSON decoder at the moment is Jason which is basically a drop-in replacement for Poison, but faster.
Anyway, reading hexdocs really helps with figuring this stuff out and the Plug project has great documentation. I think Elixir is a great language to start programming with (although it's essential to learn object-oriented paradigms as well). Happy coding!
I am new in clojure , and I have one problem with this map.
{:status 200, :headers {Server openresty, Date Thu, 11 Feb 2016 11:35:11 GMT, Content-Type application/json; charset=utf-8, Transfer-Encoding chunked, Connection close, X-Source back, Access-Control-Allow-Origin *, Access-Control-Allow-Credentials true, Access-Control-Allow-Methods GET, POST}, :body {"coord":{"lon":-0.13,"lat":51.51},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10d"}],"base":"stations","main":{"temp":278.36,"pressure":1004,"humidity":65,"temp_min":276.05,"temp_max":280.15},"visibility":10000,"wind":{"speed":2.1,"deg":230},"rain":{"1h":0.2},"clouds":{"all":0},"dt":1455190219,"sys":{"type":1,"id":5091,"message":0.0549,"country":"GB","sunrise":1455175330,"sunset":1455210486},"id":2643743,"name":"London","cod":200}, :request-time 695, :trace-redirects [http://api.openweathermap.org/data/2.5/weather?q=London,uk&appid=44db6a862fba0b067b1930da0d769e98], :orig-content-encoding nil}
I want to read fields but with get-in can not make this job.Only with
(get-in location [:body])
I can read the body of map but when I make
(get-in location ["coord" "log"])
I just get a nil response.
How I can read this fields with clojure?
Thanks and sorry for my bad english.
The body is json, so you need to first use a json library to decode the body string before being able to treat it as a map:
Using cheshire, one of the major JSON libraries for Clojure, you could write this code like:
(require '[cheshire.core])
(-> location
:body
cheshire.core/parse-string
(get-in ["coord" "log"]))
I have been writing a clojure application with ring and compojure. I am using the ring.middleware.json middleware for handling JSON so I don't have to serialise and deserialise it myself in my code.
This middleware only seems to correctly parse nested JSON when given in a flattened format, for example if I want to POST nested data to an API route I have to encode it as:
{"task" : 1,
"success" : false,
"files[0][type]" : "log",
"files[0][sha256]" : "adef5c",
"files[0][url]" : "s3://url"}
Instead of, what seems to me, more standard JSON:
{"task" : 1,
"success" : false,
"files" : [{"type" : "log", "sha256" : "adef5c", "url": "s3://url"}]}
Is this indented or a standard way of posting nested JSON? Here is my middleware stack:
(defn middleware [handler]
(-> handler
(wrap-json-response)
(wrap-with-logger)
(api)))
Here is a full Clojure example of post and get routes that handle JSON or return JSON:
(ns ring-app.core
(:require [ring.adapter.jetty :as jetty]
[compojure.core :as compojure]
[ring.middleware.keyword-params :refer [wrap-keyword-params]]
[ring.middleware.json :refer [wrap-json-response wrap-json-body]]
[ring.util.http-response :as response]
[clojure.pprint :refer [pprint]]
[ring.middleware.reload :refer [wrap-reload]]))
(defn json-get [request]
(response/ok
{"task" 1
"success" false
"files" [{"type" "log" "sha256" "adef5c" "url" "s3://url"}]}))
(defn json-post [request]
(let [bpdy (:body request)]
(prn bpdy)
(response/ok bpdy)))
(compojure/defroutes handler_
(compojure/GET "/get" request json-get)
(compojure/POST "/post" request json-post))
(defn wrap-nocache [handler]
(fn [request]
(-> request
handler
(assoc-in [:headers "Pragma"] "no-cache"))))
(def handler
(-> #'handler_ wrap-nocache wrap-reload wrap-json-response wrap-json-body ) )
The get endpoint returns the nested structure:
curl http://localhost:3000/get
; {"task":1,"success":false,"files":[{"type":"log","sha256":"adef5c","url":"s3://url"}]}
And the POST endpoint parses the json and create a Clojure data structure:
curl -X post -d '{"task":1,"success":false,"files":[{"type":"log","sha256":"adef5c","url":"s3://url"}]}' -H "Accept: application/json" -H "Content-type: application/json" http://localhost:3000/post
; {"task":1,"success":false,"files":[{"type":"log","sha256":"adef5c","url":"s3://url"}]}
This is roundtripping on JSON->CLJ->JSON
The json ring middleware expects the Content-type to be properly set so maybe that was why the structure was not parsed properly in your case ?
I am trying to POST some JSON data to a web service using drakma.
(ql:quickload :st-json)
(ql:quickload :cl-json)
(ql:quickload :drakma)
(defvar *rc* (merge-pathnames (user-homedir-pathname) ".apirc"))
(defvar *user*
(with-open-file (s *rc*)
(st-json:read-json s)))
(defvar api-url (st-json:getjso "url" *user*))
(defvar api-key (st-json:getjso "key" *user*))
(defvar api-email (st-json:getjso "email" *user*))
(setf drakma:*header-stream* *standard-output*)
(defvar *req* '(("date" . "20071001") ("time" . "00") ("origin" . "all")))
(format t "json:~S~%" (json:encode-json-to-string *req*))
(defun retrieve (api request)
(let* ((cookie-jar (make-instance 'drakma:cookie-jar))
(extra-headers (list (cons "From" api-email)
(cons "X-API-KEY" api-key)))
(url (concatenate 'string api-url api "/requests"))
(stream (drakma:http-request url
:additional-headers extra-headers
:accept "application/json"
:method :post
:content-type "application/json"
:external-format-out :utf-8
:external-format-in :utf-8
:redirect 100
:cookie-jar cookie-jar
:content (json:encode-json-to-string request)
:want-stream t)))
(st-json:read-json stream)))
(retrieve "/datasets/tigge" *req*)
Unfortunately, I get an error, although the data seems to be encoded OK to JSON and the headers generated by drakma too, I think. Apparently something is wrong with the :content (the list of integers in the errors message is just the list of ASCII codes of the JSON encoded data).
json:"{\"date\":\"20071001\",\"time\":\"00\",\"origin\":\"all\",\"type\":\"pf\",\"param\":\"tp\",\"area\":\"70\\/-130\\/30\\/-60\",\"grid\":\"2\\/2\",\"target\":\"data.grib\"}"
POST /v1/datasets/tigge/requests HTTP/1.1
Host: api.service.int
User-Agent: Drakma/1.3.0 (SBCL 1.1.5; Darwin; 12.2.0; http://weitz.de/drakma/)
Accept: application/json
Connection: close
From: me#gmail.com
X-API-KEY: 19a0edb6d8d8dda1e6a3b21223e4f86a
Content-Type: application/json
Content-Length: 193
debugger invoked on a SIMPLE-TYPE-ERROR:
The value of CL+SSL::THING is #(123 34 100 97 116 97 115 101 116 34 58 34
...), which is not of type (SIMPLE-ARRAY
(UNSIGNED-BYTE 8)
(*)).
Any idea what's wrong with this code? Many thanks in advance.
Thanks to Kevin and Hans from the General interest list for Drakma and Chunga drakma-devel for helping me out - it turned out that the problem was caused by a bug in a recent version of cl+ssl, already fixed in a development branch. I use quicklisp, and here is what Hans Hübner advised my to do to update my cl+ssl installation, which worked:
You can check out the latest cl+ssl - which contains a fix for the
problem:
cd ~/quicklisp/local-projects/
git clone git://gitorious.org/cl-plus-ssl/cl-plus-ssl.git
Quicklisp will automatically find cl+ssl from that location. Remember
to remove that checkout after you've upgraded to a newer quicklisp
release that has the fix in the future.
EDIT - The source code is on github if you're interested. Thanks
I am a bit confused as to how to access json data that has been posted to a url in clojure; I just can't seem to get it to work.
This is my route:
(cc/POST "/add"
request
(str ": " request))
I am not entirely sure what I have to put in place of request - I just saw some blog online and tried to follow it, but couldn't get it to work.
Here's how I'm trying to post: (from fiddler)
note: request header port is different in the image; it's a mistake, I was trying to mess with that data to see what it says so ignore that part in the above image please
In curl, I'm just doing this:
curl -X POST -H "Content-Type: application/json" -d '{"foo":"bar","baz":5}'
http://localhost:3005/add
It looks like clojure is not receiving the json data that I posted at all.
Here's what the request var contains:
: {:scheme :http, :query-params {}, :form-params {}, :request-method :post,
:query-string nil, :route-params {}, :content-type "\"application/json\"",
:uri "/event", :server-name "localhost", :params {},
:headers {"user-agent" "Fiddler", "content-type" "\"application/json\"",
"content-length" "23", "host" "localhost:3005"},
:content-length 23, :server-port 3005, :character-encoding nil, :body #}
As you can see, all params are empty...
I am using compojure and cheshire - and I can convert data into json and return them just fine for GET routes.. I need to figure out how to pass json and convert it into clojure data..
thanks
That's because :params is filled by a ring middleware which deals with "form-encoded" body.
You can use ring-json to wrap your application into this other middleware. It will parse the JSON body and fill :params accordingly. (https://github.com/ring-clojure/ring-json)
Here is an example that does what you want. The code was taken from this project. In the README you will see some of the access patterns that this supports. The code is a little messy, but it should illustrate how this can be done.
(ns poky.protocol.http.jdbc.text
(:require [poky.kv.core :as kv]
(compojure [core :refer :all]
[route :as route]
[handler :as handler])
[ring.util.response :refer [response not-found]]
(ring.middleware [format-response :as format-response ]
[format-params :as format-params])
[cheshire.core :as json]
[ring.middleware.stacktrace :as trace]))
;
; curl -d"some data" -H'Content-Type: application/text' -v -X PUT http://localhost:8080/xxx
; curl -d'{"bla":"bar"}' -H'Content-Type: application/json' -v -X PUT http://localhost:8080/bla
(def valid-key-regex #"[\d\w-_.,]+")
; FIXME: this should be split- one fn for get, one for mget
(defn- wrap-get
[kvstore ks params headers body]
(response
(let [eks (clojure.string/split ks #",")
nks (count eks)
multi (> nks 1)
ret (if multi (kv/mget* kvstore eks) (kv/get* kvstore ks))]
(condp = (get headers "accept")
"application/json" ret
"text/plain" (if multi (throw (Exception. "Multi get unsupported with Accept: text/plain")) (get ret ks))
ret))))
(defn- wrap-put
[kvstore ks params headers body]
(if (and
(= (get headers "content-type") "application/json")
(get params (keyword ks) nil))
(kv/set* kvstore ks (get params (keyword ks)))
(kv/set* kvstore ks body))
(response ""))
(defn api
[kvstore]
(let [api-routes
(routes
(GET ["/:ks" :ks valid-key-regex] {{:keys [ks] :as params} :params body :body headers :headers}
(wrap-get kvstore ks params headers body))
(PUT ["/:ks" :ks valid-key-regex] {{:keys [ks] :as params} :params
body :body body-params :body-params headers :headers}
(let [body (slurp body)
body (if (empty? body) body-params body)]
(wrap-put kvstore ks params headers body))))]
(-> (handler/api api-routes)
(format-params/wrap-format-params
:predicate format-params/json-request?
:decoder #(json/parse-string % true)
:charset format-params/get-or-guess-charset)
(format-response/wrap-format-response
:predicate format-response/serializable?
:encoders [(format-response/make-encoder json/encode "application/json")
(format-response/make-encoder identity "text/plain")]
:charset "utf-8")
trace/wrap-stacktrace)))
Hope this helps.
Just an update to a previous answers, now is enough to add formats key with list of formats which will be processed by Ring to your handler.
So something like
(def app (noir.util.middleware/app-handler
[your-routes]
:formats [:json-kw]))