I am new in clojure , and I have one problem with this map.
{:status 200, :headers {Server openresty, Date Thu, 11 Feb 2016 11:35:11 GMT, Content-Type application/json; charset=utf-8, Transfer-Encoding chunked, Connection close, X-Source back, Access-Control-Allow-Origin *, Access-Control-Allow-Credentials true, Access-Control-Allow-Methods GET, POST}, :body {"coord":{"lon":-0.13,"lat":51.51},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10d"}],"base":"stations","main":{"temp":278.36,"pressure":1004,"humidity":65,"temp_min":276.05,"temp_max":280.15},"visibility":10000,"wind":{"speed":2.1,"deg":230},"rain":{"1h":0.2},"clouds":{"all":0},"dt":1455190219,"sys":{"type":1,"id":5091,"message":0.0549,"country":"GB","sunrise":1455175330,"sunset":1455210486},"id":2643743,"name":"London","cod":200}, :request-time 695, :trace-redirects [http://api.openweathermap.org/data/2.5/weather?q=London,uk&appid=44db6a862fba0b067b1930da0d769e98], :orig-content-encoding nil}
I want to read fields but with get-in can not make this job.Only with
(get-in location [:body])
I can read the body of map but when I make
(get-in location ["coord" "log"])
I just get a nil response.
How I can read this fields with clojure?
Thanks and sorry for my bad english.
The body is json, so you need to first use a json library to decode the body string before being able to treat it as a map:
Using cheshire, one of the major JSON libraries for Clojure, you could write this code like:
(require '[cheshire.core])
(-> location
:body
cheshire.core/parse-string
(get-in ["coord" "log"]))
Related
I'm new to using compojure, but have been enjoying using it so far. I'm
currently encountering a problem in one of my API endpoints that is generating
a large CSV file from the database and then passing this as the response body.
The problem I seem to be encountering is that the whole CSV file is being kept
in memory which is then causing an out of memory error in the API. What is the
best way to handle and generate this, ideally as a gzipped file? Is it possible
to stream the response so that a few thousand rows are returned at a time? When
I return a JSON response body for the same data, there is no problem returning
this.
Here is the current code I'm using to return this:
(defn complete
"Returns metrics for each completed benchmark instance"
[db-client response-format]
(let [benchmarks (completed-benchmark-metrics {} db-client)]
(case response-format
:json (json-grouped-output field-mappings benchmarks)
:csv (csv-output benchmarks))))
(defn csv-output [data-seq]
(let [header (map name (keys (first data-seq)))
out (java.io.StringWriter.)
write #(csv/write-csv out (list %))]
(write header)
(dorun (map (comp write vals) data-seq))
(.toString out)))
The data-seq is the results returned from the database, which I think is a
lazy sequence. I'm using yesql to perform the database call.
Here is my compojure resource for this API endpoint:
(defresource results-complete [db]
:available-media-types ["application/json" "text/csv"]
:allowed-methods [:get]
:handle-ok (fn [request]
(let [response-format (keyword (get-in request [:request :params :format] :json))
disposition (str "attachment; filename=\"nucleotides_benchmark_metrics." (name response-format) "\"")
response {:headers {"Content-Type" (content-types response-format)
"Content-Disposition" disposition}
:body (results/complete db response-format)}]
(ring-response response))))
Thanks to all the suggestion that were provided in this thread, I was able to create a solution using piped-input-stream:
(defn csv-output [data-seq]
(let [headers (map name (keys (first data-seq)))
rows (map vals data-seq)
stream-csv (fn [out] (csv/write-csv out (cons headers rows))
(.flush out))]
(piped-input-stream #(stream-csv (io/make-writer % {})))))
This differs from my solution because it does not realise the sequence using dorun and does not create a large String object either. This instead writes to a PipedInputStream connection asynchronously as described by the documentation:
Create an input stream from a function that takes an output stream as its
argument. The function will be executed in a separate thread. The stream
will be automatically closed after the function finishes.
Your csv-output function completely realises the dataset and turns it into a string. To lazily stream the data, you'll need to return something other than a concrete data type like a String. This suggests ring supports returning a stream, that can be lazily realised by Jetty. The answer to this question might prove useful.
I was also struggling with the streaming of large csv file. My solution was to use httpkit-channel to stream every single line of the data-seq to the client and then close the channel. My solution looks like that:
[org.httpkit.server :refer :all]
(fn handler [req]
(with-channel req channel (let [header "your$header"
data-seq ["your$seq-data"]]
(doseq [line (cons header data-seq)]
(send! channel
{:status 200
:headers {"Content-Type" "text/csv"}
:body (str line "\n")}
false))
(close channel))))
I was flooded with a primitive json body for fcm:
Body = mochijson2:encode([ {<<"operation">>, <<"create">>},{<<"notification_key_name">>, <<"console group">>},{<<"registration_ids">>, [<<"02aa6XXXX3c9b6d">>,<<"APA91bGtaXXXXXXXXXXXXoi4UH8vIdZk1X67A_9izpSFSHV3BXxdIwG">>]}]).
And send POST-request to create group according to documentation:
httpc:request(post, {Url, [{"Authorization", KeyApi}, {"project_id", ProjectId}], "application/json", Body},[{timeout, 5000}], []).
But I got error BadJsonFormat:
{ok,{{"HTTP/1.1",400,"Bad Request"},
[{"cache-control","private, max-age=0"},
{"date","Fri, 10 Mar 2017 16:19:37 GMT"},
{"accept-ranges","none"},
{"server","GSE"},
{"vary","Accept-Encoding"},
{"content-length","25"},
{"content-type","application/json; charset=UTF-8"},
{"expires","Fri, 10 Mar 2017 16:19:37 GMT"},
{"x-content-type-options","nosniff"},
{"x-frame-options","SAMEORIGIN"},
{"x-xss-protection","1; mode=block"},
{"alt-svc","quic=\":443\"; ma=2592000; v=\"36,35,34\""}],
"{\"error\":\"BadJsonFormat\"}"}}
But mochijson2:decode(Body) works fine, and it looks like properly formed json, but I get the error BadJsonFormat anyway.
What was wrong? How can I fix this?
The function mochijson2:encode doesn't return a string or a binary, but an iolist:
1> Body = mochijson2:encode([ {<<"operation">>, <<"create">>},{<<"notification_key_name">>, <<"console group">>},{<<"registration_ids">>, [<<"02aa6XXXX3c9b6d">>,<<"APA91bGtaXXXXXXXXXXXXoi4UH8vIdZk1X67A_9izpSFSHV3BXxdIwG">>]}]).
[123,
[34,<<"operation">>,34],
58,
[34,<<"create">>,34],
44,
[34,<<"notification_key_name">>,34],
58,
[34,<<"console group">>,34],
44,
[34,<<"registration_ids">>,34],
58,
[91,
[34,<<"02aa6XXXX3c9b6d">>,34],
44,
[34,<<"APA91bGtaXXXXXXXXXXXXoi4UH8vIdZk1X67A_9izpSF"...>>,
34],
93],
125]
There is nothing wrong with that, by itself. Using iolists instead of strings or binaries means that you don't have to create an expensive flat data structure, that you would just write to a file or a socket, after which you'd throw it away. Function like file:write_file and gen_tcp:send handle iolists just as well as strings or binaries.
However, httpc:request doesn't!
Let's test that by starting a server on port 1111 with netcat in a shell:
$ nc -l 1111
And then make a request from the Erlang shell:
3> httpc:request(post, {"http://127.0.0.1:1111", [], "application/json", Body},[{timeout, 5000}], []).
The netcat server shows this output:
POST / HTTP/1.1
content-type: application/json
content-length: 13
te:
host: 127.0.0.1:1111
connection: keep-alive
{"operation":"create",....
Note that the content-length is 13 instead of 159! httpc:request is able to send the iolist, but it uses the function length instead of iolist_size to generate the content-length header, and as a result the server only considers the first 13 bytes of the JSON object, which is not valid JSON by itself.
The solution is to pass iolist_to_binary(Body) to httpc:request instead of just Body.
I am trying to implement a simple GET/POST api via Django REST framework
views.py
class cuser(APIView):
def post(self, request):
stream = BytesIO(request.DATA)
json = JSONParser().parse(stream)
return Response()
urls.py
from django.conf.urls import patterns, url
from app import views
urlpatterns = patterns('',
url(r'^challenges/',views.getall.as_view() ),
url(r'^cuser/' , views.cuser.as_view() ),
)
I am trying to POST some json to /api/cuser/ (api is namespace in my project's urls.py ) ,
the JSON
{
"username" : "abhishek",
"email" : "john#doe.com",
"password" : "secretpass"
}
I tried from both Browseable API page and httpie ( A python made tool similar to curl)
httpie command
http --json POST http://localhost:58601/api/cuser/ username=abhishek email=john#doe.com password=secretpass
but I am getting JSON parse error :
JSON parse error - Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Whole Debug message using --verbose --debug
POST /api/cuser/ HTTP/1.1
Content-Length: 75
Accept-Encoding: gzip, deflate
Host: localhost:55392
Accept: application/json
User-Agent: HTTPie/0.8.0
Connection: keep-alive
Content-Type: application/json; charset=utf-8
{"username": "abhishek", "email": "john#doe.com", "password": "aaezaakmi1"}
HTTP/1.0 400 BAD REQUEST
Date: Sat, 24 Jan 2015 09:40:03 GMT
Server: WSGIServer/0.1 Python/2.7.9
Vary: Accept, Cookie
Content-Type: application/json
Allow: POST, OPTIONS
{"detail":"JSON parse error - Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"}
The problem that you are running into is that your request is already being parsed, and you are trying to parse it a second time.
From "How the parser is determined"
The set of valid parsers for a view is always defined as a list of classes. When request.data is accessed, REST framework will examine the Content-Type header on the incoming request, and determine which parser to use to parse the request content.
In your code you are accessing request.DATA, which is the 2.4.x equaivalent of request.data. So your request is being parsed as soon as you call that, and request.DATA is actually returning the dictionary that you were expecting to parse.
json = request.DATA
is really all you need to parse the incoming JSON data. You were really passing a Python dictionary into json.loads, which does not appear to be able to parse it, and that is why you were getting your error.
I arrived at this post via Google for
"detail": "JSON parse error - Expecting property name enclosed in double-quotes":
Turns out you CANNOT have a trailing comma in JSON.
So if you are getting this error you may need to change a post like this:
{
"username" : "abhishek",
"email" : "john#doe.com",
"password" : "secretpass",
}
to this:
{
"username" : "abhishek",
"email" : "john#doe.com",
"password" : "secretpass"
}
Note the removed comma after the last property in the JSON object.
Basically, whenever you are trying to make a post request with requests lib, This library also contains json argument which is ignored in the case when data argument is set to files or data. So basically when json argument is set with json data. Headers are set asContent-Type: application/json. Json argument basically encodes data sends into a json format. So that at DRF particularly is able to parse json data. Else in case of only data argument it is been treated as form-encoded
requests.post(url, json={"key1":"value1"})
you can find more here request.post complicated post methods
I am trying to POST some JSON data to a web service using drakma.
(ql:quickload :st-json)
(ql:quickload :cl-json)
(ql:quickload :drakma)
(defvar *rc* (merge-pathnames (user-homedir-pathname) ".apirc"))
(defvar *user*
(with-open-file (s *rc*)
(st-json:read-json s)))
(defvar api-url (st-json:getjso "url" *user*))
(defvar api-key (st-json:getjso "key" *user*))
(defvar api-email (st-json:getjso "email" *user*))
(setf drakma:*header-stream* *standard-output*)
(defvar *req* '(("date" . "20071001") ("time" . "00") ("origin" . "all")))
(format t "json:~S~%" (json:encode-json-to-string *req*))
(defun retrieve (api request)
(let* ((cookie-jar (make-instance 'drakma:cookie-jar))
(extra-headers (list (cons "From" api-email)
(cons "X-API-KEY" api-key)))
(url (concatenate 'string api-url api "/requests"))
(stream (drakma:http-request url
:additional-headers extra-headers
:accept "application/json"
:method :post
:content-type "application/json"
:external-format-out :utf-8
:external-format-in :utf-8
:redirect 100
:cookie-jar cookie-jar
:content (json:encode-json-to-string request)
:want-stream t)))
(st-json:read-json stream)))
(retrieve "/datasets/tigge" *req*)
Unfortunately, I get an error, although the data seems to be encoded OK to JSON and the headers generated by drakma too, I think. Apparently something is wrong with the :content (the list of integers in the errors message is just the list of ASCII codes of the JSON encoded data).
json:"{\"date\":\"20071001\",\"time\":\"00\",\"origin\":\"all\",\"type\":\"pf\",\"param\":\"tp\",\"area\":\"70\\/-130\\/30\\/-60\",\"grid\":\"2\\/2\",\"target\":\"data.grib\"}"
POST /v1/datasets/tigge/requests HTTP/1.1
Host: api.service.int
User-Agent: Drakma/1.3.0 (SBCL 1.1.5; Darwin; 12.2.0; http://weitz.de/drakma/)
Accept: application/json
Connection: close
From: me#gmail.com
X-API-KEY: 19a0edb6d8d8dda1e6a3b21223e4f86a
Content-Type: application/json
Content-Length: 193
debugger invoked on a SIMPLE-TYPE-ERROR:
The value of CL+SSL::THING is #(123 34 100 97 116 97 115 101 116 34 58 34
...), which is not of type (SIMPLE-ARRAY
(UNSIGNED-BYTE 8)
(*)).
Any idea what's wrong with this code? Many thanks in advance.
Thanks to Kevin and Hans from the General interest list for Drakma and Chunga drakma-devel for helping me out - it turned out that the problem was caused by a bug in a recent version of cl+ssl, already fixed in a development branch. I use quicklisp, and here is what Hans Hübner advised my to do to update my cl+ssl installation, which worked:
You can check out the latest cl+ssl - which contains a fix for the
problem:
cd ~/quicklisp/local-projects/
git clone git://gitorious.org/cl-plus-ssl/cl-plus-ssl.git
Quicklisp will automatically find cl+ssl from that location. Remember
to remove that checkout after you've upgraded to a newer quicklisp
release that has the fix in the future.
EDIT - The source code is on github if you're interested. Thanks
I am a bit confused as to how to access json data that has been posted to a url in clojure; I just can't seem to get it to work.
This is my route:
(cc/POST "/add"
request
(str ": " request))
I am not entirely sure what I have to put in place of request - I just saw some blog online and tried to follow it, but couldn't get it to work.
Here's how I'm trying to post: (from fiddler)
note: request header port is different in the image; it's a mistake, I was trying to mess with that data to see what it says so ignore that part in the above image please
In curl, I'm just doing this:
curl -X POST -H "Content-Type: application/json" -d '{"foo":"bar","baz":5}'
http://localhost:3005/add
It looks like clojure is not receiving the json data that I posted at all.
Here's what the request var contains:
: {:scheme :http, :query-params {}, :form-params {}, :request-method :post,
:query-string nil, :route-params {}, :content-type "\"application/json\"",
:uri "/event", :server-name "localhost", :params {},
:headers {"user-agent" "Fiddler", "content-type" "\"application/json\"",
"content-length" "23", "host" "localhost:3005"},
:content-length 23, :server-port 3005, :character-encoding nil, :body #}
As you can see, all params are empty...
I am using compojure and cheshire - and I can convert data into json and return them just fine for GET routes.. I need to figure out how to pass json and convert it into clojure data..
thanks
That's because :params is filled by a ring middleware which deals with "form-encoded" body.
You can use ring-json to wrap your application into this other middleware. It will parse the JSON body and fill :params accordingly. (https://github.com/ring-clojure/ring-json)
Here is an example that does what you want. The code was taken from this project. In the README you will see some of the access patterns that this supports. The code is a little messy, but it should illustrate how this can be done.
(ns poky.protocol.http.jdbc.text
(:require [poky.kv.core :as kv]
(compojure [core :refer :all]
[route :as route]
[handler :as handler])
[ring.util.response :refer [response not-found]]
(ring.middleware [format-response :as format-response ]
[format-params :as format-params])
[cheshire.core :as json]
[ring.middleware.stacktrace :as trace]))
;
; curl -d"some data" -H'Content-Type: application/text' -v -X PUT http://localhost:8080/xxx
; curl -d'{"bla":"bar"}' -H'Content-Type: application/json' -v -X PUT http://localhost:8080/bla
(def valid-key-regex #"[\d\w-_.,]+")
; FIXME: this should be split- one fn for get, one for mget
(defn- wrap-get
[kvstore ks params headers body]
(response
(let [eks (clojure.string/split ks #",")
nks (count eks)
multi (> nks 1)
ret (if multi (kv/mget* kvstore eks) (kv/get* kvstore ks))]
(condp = (get headers "accept")
"application/json" ret
"text/plain" (if multi (throw (Exception. "Multi get unsupported with Accept: text/plain")) (get ret ks))
ret))))
(defn- wrap-put
[kvstore ks params headers body]
(if (and
(= (get headers "content-type") "application/json")
(get params (keyword ks) nil))
(kv/set* kvstore ks (get params (keyword ks)))
(kv/set* kvstore ks body))
(response ""))
(defn api
[kvstore]
(let [api-routes
(routes
(GET ["/:ks" :ks valid-key-regex] {{:keys [ks] :as params} :params body :body headers :headers}
(wrap-get kvstore ks params headers body))
(PUT ["/:ks" :ks valid-key-regex] {{:keys [ks] :as params} :params
body :body body-params :body-params headers :headers}
(let [body (slurp body)
body (if (empty? body) body-params body)]
(wrap-put kvstore ks params headers body))))]
(-> (handler/api api-routes)
(format-params/wrap-format-params
:predicate format-params/json-request?
:decoder #(json/parse-string % true)
:charset format-params/get-or-guess-charset)
(format-response/wrap-format-response
:predicate format-response/serializable?
:encoders [(format-response/make-encoder json/encode "application/json")
(format-response/make-encoder identity "text/plain")]
:charset "utf-8")
trace/wrap-stacktrace)))
Hope this helps.
Just an update to a previous answers, now is enough to add formats key with list of formats which will be processed by Ring to your handler.
So something like
(def app (noir.util.middleware/app-handler
[your-routes]
:formats [:json-kw]))