JSON requests within Emacs, Phase 2: Getting a collection of items - json

I'm trying to get a list of sites (a list of a-lists) with the Stack Exchange API using request.el.
I'm making an Emacs major mode for Stack Exchange, so this has some nice potential payoff for you Emacs users out there. ;) (And since I tagged elisp, I'm assuming that's the lot of you.)
To do this, a fundamental necessity would be to make a request for JSON, and then view the list of returned sites. The StackExchange API supplies the /sites resource, and a request for that resource returns a collection of site objects like so:
{
"items": [
{
"site_type": "main_site",
"name": "Stack Overflow",
"logo_url": "http://cdn.sstatic.net/stackoverflow/img/logo.png",
"api_site_parameter": "stackoverflow",
"site_url": "http://stackoverflow.com",
"audience": "professional and enthusiast programmers",
"icon_url": "http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon.png",
"aliases": [
"http://www.stackoverflow.com"
],
"site_state": "normal",
"styling": {
"link_color": "#0077CC",
"tag_foreground_color": "#3E6D8E",
"tag_background_color": "#E0EAF1"
},
"launch_date": 1221436800,
"favicon_url": "http://cdn.sstatic.net/stackoverflow/img/favicon.ico",
"related_sites": [
{
"name": "Stack Overflow Chat",
"site_url": "http://chat.stackoverflow.com",
"relation": "chat"
}
],
"markdown_extensions": [
"Prettify"
],
"high_resolution_icon_url": "http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon#2.png"
},
{
"site_type": "main_site",
"name": "Server Fault",
"logo_url": "http://cdn.sstatic.net/serverfault/img/logo.png",
"api_site_parameter": "serverfault",
"site_url": "http://serverfault.com",
"audience": "professional system and network administrators",
"icon_url": "http://cdn.sstatic.net/serverfault/img/apple-touch-icon.png",
...
},
{
"site_type": "main_site",
"name": "Super User",
...
{
"site_type": "main_site",
"name": "Meta Stack Overflow",
...
}
...
}
I'd like to minimize the number of calls I make to the API and retrieve them all and place them in a data structure in one go for me to be able to make sense of it later on.
I'm trying to adapt one of the lovely solutions I've found here, Making JSON requests within Emacs, to fit what I need to do. It uses the request.el library, from tkf.
The example tkf gave me was able to take the most active question on a site and grab its title and tags properties using json-read, which essentially turns an object into an a-list. This attempt is based off of that solution:
(request
"https://api.stackexchange.com/2.1/sites"
:parser 'buffer-string
:success (function*
(lambda (&key data &allow-other-keys)
(let* ((items (assoc-default 'items data))
(names (mapcar (lambda (item) (assoc-default 'name item)) items))
(launches (mapcar (lambda (item) (assoc-default 'launch-date item)) items)))
(mapcar* (lambda (name launch)
(message "name:`%s` launch:`%s`" name launch))
names
launches)))))
...but seems entirely ineffective. The other examples work perfectly, so it is something wrong with my usage.
request.el can be downloaded from the MELPA package repository and, to my knowledge, requires curl to run correctly (which I have).
I suspect the problem lies in my usage (or preparation thereof) of mapcar*, where the following does work as expected:
(mapcar* (lambda (a b) (insert a) (insert b)) '(1 2 3) '(4 5 6))
I know this post is long, but I tried to provide as much information as I could.

You were almost there. This one works for me:
(request
"https://api.stackexchange.com/2.1/sites"
:parser 'json-read
:success (function*
(lambda (&key data &allow-other-keys)
(let* ((items (assoc-default 'items data))
(names (mapcar (lambda (item) (assoc-default 'name item)) items))
(launches (mapcar (lambda (item) (assoc-default 'launch_date item)) items)))
(mapcar* (lambda (name launch)
(message "name:`%s` launch:`%s`" name launch))
names
launches)))))
Two changes: 1. use json-read instead of buffer-string for parser argument. 2. use launch_date instead of launch-date as the key for alist.

Related

Clojurescript — PersistentArrayMap → Object → PersistentArrayMap — transferring data between web worker

I'm trying to pass a clojurescript map to a webworker.
Before I pass it, it is of type PersistentArrayMap.
cljs.core.PersistentArrayMap {meta: null, cnt: 3, arr: Array(6), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951…}
However, when it gets to the worker, it's just a plain old Object
Object {meta: null, cnt: 3, arr: Array(6), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951…}
with the data seemingly intact. At this point I'd like to turn it back into a PersistentArrayMap so that I can work with it in cljs again.
Using clj->js and js->clj doesn't really work because it doesn't distinguish between keywords and strings, so some data is lost.
What's the best way to handle this situation? It's possible that I'm going about this in the entirely wrong way.
The built-in solution is to serialize the data to EDN and reading it back. clj->js is inherently lossy and should be avoided.
You start by turning the object into a string and send that over to the worker.
(pr-str {:a 1})
;; => "{:a 1}"
In the worker you read it back via cljs.reader/read-string
(cljs.reader/read-string "{:a 1}")
;; => {:a 1}
This will usually be good enough but the transit-cljs library will be slightly faster. Depending on the amount of data you plan on sending it may be worth the extra dependency.
Did you use keywordize-keys in the code?
(def json "{\"foo\": 1, \"bar\": 2, \"baz\": [1,2,3]}")
(def a (.parse js/JSON json))
;;=> #js {:foo 1, :bar 2, :baz #js [1 2 3]}
(js->clj a)
;;=> {"foo" 1, "bar" 2, "baz" [1 2 3]}
(js->clj a :keywordize-keys true)
;;=> {:foo 1, :bar 2, :baz [1 2 3]}
Full documentation is here.

Using request.el to parse json to get location data from ipinfo.io

Request.el appears to be built to easily query web services that return json responses.
I would like to use it to get the city and state results that return from a query to http://ipinfo.io
If, from the command-line, I issue the following:
curl https://ipinfo.io
I get back the following:
{
"ip": "24.xxx.xxx.xxx",
"hostname": "cpe-xx-xxx-xxx-x.xxx.xxx.xx.com",
"city": "Brooklyn",
"region": "New York",
"country": "US",
"loc": "40.6406,-74.0169",
"org": "AS12271 Time Warner Cable Internet LLC",
"postal": "11220"
}%
I would like to pull out the "city" and "state" values and put them into a single variable as a string such as:
(setq my-location "Brooklyn, NY")
While the resolve.el page has a great deal of documentation, I'm not a programmer and I simply don't understand how to get the information I need. I'm pretty sure I'm on the right track, but I just lack the knowlege of how to get where I want to go.
Something like this might work:
(require 'request)
(request "https://ipinfo.io"
:parser 'json-read
:success (cl-function
(lambda (&key data &allow-other-keys)
(message "I sent: %S" (assoc-default 'args data))
(message "Data is %s" data )
(message "Result is %s, %s" (assoc-default 'city data) (assoc-default 'region data)))))
(Not tested because I have an internal IP address and therefore don't have access to city/region data.)

Accessing array in Json Data with Clojurescript

I am new Clojurescript and want to hack arround with clojurescript and electron based on an small json file.
I am doing something like (with transit/cljs)
(def jsondata (t/read (t/reader :json) (.readFileSync fs path_to_file "utf8")))) )
first I check if status is ok, that works fine...
(let [json_status (get jsondata "status")]
(.log js/console "JSON Glossar Status:" json_status))
now, how can I access one of the maps in the pages array, or step through the map?
{"status":"ok",
"pages":[
{
"id":1,
"name":"name1",
"image":"imagename1.png",
"children":[
{
"id":1,
"copytext":"kdjsldjsljfl"
},
{
"id":2,
"copytext":"dksdöfksöfklsöf"
}
]
},
{
"id":2,
"name":"name1",
"image":"imagename1.png",
"children":[
{
"id":4,
"copytext":"kdjsldjsljfl"
},
{
"id":5,
"copytext":"dksdöfksöfklsöf"
}
]
}
]
}
You can use aget (i.e. "array get") for nested ClojureScript / JavaScript interop.
For example, if you wanted to access the second map item in your "pages" array, you could do this:
(def my-js-object
(clj->js {:status "ok"
:pages [{:id 1
:name "foo"
:children []}
{:id 2
:name "bar"
:children []}]}))
(aget my-js-object "pages" 1)
In the above code I'm simply using clj->js to construct a notional (and incomplete) representation of your JSON; I hope this is enough to make sense.
My REPL output was:
#js {:id 2, :name "bar", :children #js []}
If you wanted to do something more complex with each page item, e.g. "map over each page hashmap and pull out the name values", then you could make use of the .- JS property accessor
(->> (.-pages my-js-object)
(map #(.-name %)))
REPL output:
("foo" "bar")
To not answer the question, you could use
js->cljs, https://cljs.github.io/api/cljs.core/js-GTclj, to turn your json into a normal Clojure data structure and use Clojures normal fns to extract the data you want.

How to read json file into Clojure defrecord (to be searched later)

I have created a defrecord in a Clojure REPL:
user=> (defrecord Data [column1 column2 column3])
user.Data
How do I automate adding data to this record by reading in a .json file? Each of the columns in the defrecord corresponds exactly to a key in the json data. If the file contained a single record it would look similar to this:
[
{
"column1" : "value1"
"column2" : "value2"
"column3" : "value3"
}
]
But there are many thousands of such records in the file.
I can slurp the contents of the file like this:
(json/read-json (slurp "path/to/file.json")))
The dependencies for the read-json function are added to the project.clj file found in the directory where I am running lein repl from the command line: :dependencies [org.clojure/data.json "0.2.1"].
I would just like to be able to search the values of the records using a Clojure function, such that the value I am passing to the search function is between the values of a single record's column1 and column2 values (i.e., nth-record.column1.value <= query <= nth-record.column2.value). Once I've found a matching record, I want to return the value of another column in that same record (nth-record.column3.value). The values of columns 1 and 2 will be unique, representing a non-overlapping range of values. The value of column3 is not unique.
This seems like a fairly trivial task, but I can't figure out how to do it using the Clojure documentation or the examples I've found online. It doesn't matter to me how the records are stored internally in Clojure, as long as I can search them and return the value of a related field in the same record.
Using data.json package:
(require '[clojure.data.json :as json])
Read values into memory:
(def all-records (json/read-str (slurp "path/to/file.json")
:key-fn keyword))
;; ==> [ { :column1 "value1", :column2 "value2", :column3 "value3" }, ...]
Find matching records:
(def query "some-value")
(def matching (filter #(and (< (:column1 %) query) (< query (:column2 %))) all-records))
Get column3:
(map :column3 matching)
Collecting it all together (and making it more flexible):
(defn find-matching [select-fn result-fn records]
(map result-fn (filter select-fn records)))
(defn select-within [rec query]
(and (< (:column1 rec) query) (< query (:column2 rec))))
(find-matching #(select-within % "some-value") :column3 all-records)
Should probably use cheshire for speed.
If your queries get sufficiently complex, consider lucene, clojure has a nice wrapper.
I think you're thinking records are somehow more suitable for this than maps, as far as I can tell, you're not using any features that make records special like polymorphism. There might be a way to make cheshire spit out records, but I wouldn't bother.

clojure loop through json data

I have a problem parsing json data in a loop. Iam a clojure beginner and need some hint for looping through json data.
The data looks like this:
{"photoset" {"primary" "8455893107", "total" "2", "pages" 1, "perpage" 500, "page" 1,
"per_page" 500, "photo"
[{"id" "8455893107", "secret" "1a3236df06", "server" "8087",
"farm" 9, "title" "IMG_0137", "isprimary" "1"}
{"id" "8469482476", "secret" "4c1bf59214",
"server" "8235", "farm" 9, "title" "HippieBus", "isprimary" "0"}]
, "owner"
"93029076#N07", "id" "72157632724688181", "ownername" "clojureB5"}, "stat" "ok"}
What I want to do is loop through the two photos and build a new url with the id and farm value like http://www.flickr.com/farm/id
I know that I can get one value like this:
(-> (get-in (cheshire.core/parse-string (:body picList)) ["photoset" "photo"]) first (get "id"))
But I can I now loop through it?
You can simply use map.
(->> (get-in data ["photoset" "photo"])
(map #(str "http://www.flickr.com/" (get % "farm") "/" (get % "id"))))
It will yield the following list:
("http://www.flickr.com/9/8455893107" "http://www.flickr.com/9/8469482476")