I have a single string like with multiples json:
{"operation": {"status": true, "limit": 100}}{"operation1": {"customer": "Jhon", "sum": 20, "time": "2019-02-13T10:00:00.000Z"}}{"operation1": {"customer": "Henry", "sum": 90, "time": "2019-02-13T11:00:00.000Z"}}
I would like to get a list of json to be able to process each json as an individual object.
You can use json library with streaming support. E.g.
(require '[cheshire.core :as json])
(->> "{\"operation\": {\"status\": true, \"limit\": 100}}{\"operation1\": {\"customer\": \"Jhon\", \"sum\": 20, \"time\": \"2019-02-13T10:00:00.000Z\"}}{\"operation1\": {\"customer\": \"Henry\", \"sum\": 90, \"time\": \"2019-02-13T11:00:00.000Z\"}}"
char-array
io/reader
json/parsed-seq
(take 2))
returns
({"operation" {"status" true, "limit" 100}}
{"operation1" {"customer" "Jhon", "sum" 20, "time" "2019-02-13T10:00:00.000Z"}})
I had to clean up your data. You may need to clarify where it is coming from so your question is more understandable (in context).
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test)
(:require
[tupelo.core :as t]
[tupelo.string :as ts]))
; NOTE: I added square-brackets to wrap everyting in an array, and I also
; had to add commas between the array entries
(def data
"[ {'operation': {'status': true, 'limit': 100}},
{'operation1': {'customer': 'Jhon', 'sum': 20, 'time': '2019-02-13T10:00:00.000Z'}},
{'operation1': {'customer': 'Henry', 'sum': 90, 'time': '2019-02-13T11:00:00.000Z'}} ] " )
with result
(t/json->edn (ts/quotes->double data)) =>
({:operation {:status true, :limit 100}}
{:operation1
{:customer "Jhon", :sum 20, :time "2019-02-13T10:00:00.000Z"}}
{:operation1
{:customer "Henry", :sum 90, :time "2019-02-13T11:00:00.000Z"}})
This does not deserve to count as an answer beyond the helpful one by #rmcv, but it is too much for just a comment. If useful, #rmcv should feel free to add this context to their answer and I can then delete this answer...
If you want to return the JSON keywords as Clojure keywords then a tweak to that answer is:
;; Emits map with JSON keys as Clojure keywords...
(-> "{\"operation\": {\"status\": true, \"limit\": 100}}{\"operation1\": {\"customer\": \"Jhon\", \"sum\": 20, \"time\": \"2019-02-13T10:00:00.000Z\"}{\"operation1\": {\"customer\": \"Henry\", \"sum\": 90, \"time\": \"2019-02-13T11:00:00.000Z\"}}"
char-array
io/reader
(json/parsed-seq true)
(->> (take 2)))
which returns
({:operation {:status true, :limit 100}}
{:operation1 {:customer "Jhon", :sum 20, :time "2019-02-13T10:00:00.000Z"}})
The true argument to parsed-seq just applies this function to the JSON keywords
(fn [k] (keyword k))
and you can do more complex things with the keyword processing this way, i.e. (not so complex):
(-> "{\"operation\": {\"status\": true, \"limit\": 100}}{\"operation1\": {\"customer\": \"Jhon\", \"sum\": 20, \"time\": \"2019-02-13T10:00:00.000Z\"}}{\"operation1\": {\"customer\": \"Henry\", \"sum\": 90, \"time\": \"2019-02-13T11:00:00.000Z\"}}"
char-array
io/reader
(chesire/parsed-seq (fn [k] (keyword (clojure.string/upper-case k))))
(->> (take 2)))
returns
({:OPERATION {:STATUS true, :LIMIT 100}}
{:OPERATION1 {:CUSTOMER "Jhon", :SUM 20, :TIME "2019-02-13T10:00:00.000Z"}})
Related
This is how far I've gotten from converting a pandas dataframe into the format I want it.
print(data_frame)
Output:
>>{'Country': ['Croatia', 'Serbia', 'United States', 'Algeria', 'Chile'],
'Value': [5, 5, 4, 2, 2]}
dictlist = []
for key, value in data_frame.items():
temp = [key,value]
dictlist.append(temp)
print(dictlist)
Output:
>>[['Country', ['Croatia', 'Serbia', 'United States', 'Algeria', 'Chile']], ['Value', [5, 5, 4, 2, 2]]]
Now here is where I'm lost. I've tried so many combinations of videos and tutorials that even my brute force ideas are empty.
for key, val in dictlist:
print(key, "->", val)
for item in val:
print(item)
output:
>>Country -> ['Croatia', 'Serbia', 'United States', 'Algeria', 'Chile']
>>Croatia
>>Serbia
>>United States
>>Algeria
>>Chile
>>Value -> [5, 5, 4, 2, 2]
>>5
>>5
>>4
>>2
>>2
I just don't know how to work inside that for loop. I can't iterate any further than that.
This is the format I'm trying to get it converted to:
[
{
"name": "Croatia",
"value": 5,
},
{
"name": "Serbia",
"value": 5,
},
{
"name": "United States",
"value": 4,
},
{
"name": "Algeria",
"value": 2,
},
{
"name": "Chile",
"value": 2,
}
]
Update
Thanks to a friendly fellow in the chat, I've now gotten closer to the goal. This line helped me a lot.
data_frame = frame.set_index('Country').T.to_dict('list')
print(data_frame)
output:
{'Croatia': [5],
'Serbia': [5],
'United States': [4],
'Algeria': [2],
'Chile': [2]}
Not sure why the values got converted into arrays, but at least this should be solvable! Cheers!
This would return dictionary as desired in the question.
# this is the data frame
frame = pd.DataFrame({'Country': ['Croatia', 'Serbia', 'United States', 'Algeria', 'Chile'],
Check the pandas documentation for to_dict to get available parameter options apart from 'list' that was used in the posted code sample.
Now one can settle for 'records' which returns dictionary inside a list:
data_frame = frame.set_index('Country')T.to_dict('records')[0]
# this is the output without [0]
# [{'Croatia': 5, 'Serbia': 5, 'United States': 4, 'Algeria': 2, 'Chile': 2}]
# so you need the [0] to pick the dictionary in the list and get the desired dictionary output.
I try to write this d3.js's code https://bl.ocks.org/d3noob/43a860bc0024792f8803bba8ca0d5ecd with clojurescript because I want to use re-frame framework.
But I could not find any solution because it has the process of changing values in a dictionary such as the function #'collapse.
I have a problem.
Can ClojureScript write changing values in a dictionary?
ex.
function remove_children (d) {
if (d.children) {
d.children = null;
}
}
family_tree = {
name: "John",
children: [
{ name: "Jack",
children: [
{ name: "Michel"}]},
{ name: "Emily",
children: [
{ name: "Mike"}]}]
};
jack = family_tree.children[0]
remove_children(jack)
;; in clojurescript ... I have no solution ...
As mentioned above, you might run into issues later if you are trying to work with mutable data (for D3) and immutable data (for re-frame).
When possible, use ClojureScript's own data structures and only pass JavaScript objects when iteracting with libraries, eg. you can use (clj->js my-map) to convert a CLJS map into a JS object.
If you want to use JS interop to mutate JavaScript objects, that's also possible:
(def family-tree
(clj->js
{
"name" "John",
"children" [
{ "name" "Jack",
"children" [
{ "name" "Michel"}]},
{ "name" "Emily",
"children" [
{ "name" "Mike"}]}]
}))
;; Check if family-tree looks like expected:
cljs.user=> family-tree
#js {:name "John",
:children #js [#js {:name "Jack",
:children #js [#js {:name "Michel"}]}
#js {:name "Emily",
:children #js [#js {:name "Mike"}]}]}
;; Define remove-children and Jack using interop:
(defn remove-children [elem]
(set! (.-children elem) nil))
(def jack
(-> family-tree .-children (get 0)))
;; Test them
cljs.user=> (remove-children jack)
nil
cljs.user=> family-tree
#js {:name "John",
:children #js [#js {:name "Jack", :children nil} ;; <== Children were deleted
#js {:name "Emily",
:children #js [#js {:name "Mike"}]}]}
My question could be a little bit easy to answer but I've just started to learn clojure and I have no idea of how to do this.
I have a JSON file that I have read and parsed. Now I have to get each element of this JSON and treat then according with the type.
For example, I have the JSON:
{
"foo": {
"id": 1,
"name": "foo1"
},
"bar": {
"id": 1,
"name": "bar1"
},
"foo": {
"id": 2,
"name": "foo2"
},
"bar": {
"id": 2,
"name": "bar2"
}
}
So, I want to make a function that iterate on every element of my JSON and then call another function with multiples declarations for each element type of my JSON.
The problem is that I don't know how I can get the JSON element type...
Can someone help me?
You can get all key-value pairs of a map by treating it as a Clojure sequence. Then it behaves as a sequence of two-element vectors, a sequence of MapEntry elements to be more precise. When you call first on a MapEntry you will get the key.
So (first (seq {:a 1})) returns [:a 1] and (first [:a 1]) returns :a, the key. The call to seq in this example is not needed, but is only there to make the example more explicit.
You could then write a case expression to do things according to the key. To make it more extensible you could use multimethods.
(def json-str "{\n \"foo\": {\n \"id\": 1,\n \"name\": \"foo1\"\n },\n \"bar\": {\n \"id\": 1,\n \"name\": \"bar1\"\n },\n \"foo\": {\n \"id\": 2,\n \"name\": \"foo2\"\n },\n \"bar\": {\n \"id\": 2,\n \"name\": \"bar2\"\n }\n}")
(def parsed (cheshire.core/parse-string json-str true))
;; {:foo {:id 2, :name "foo2"}, :bar {:id 2, :name "bar2"}}
;; handle with case:
(defn handle-data
[data]
(doseq [[k v] data]
(case k
:foo (println "this is a foo!" v)
:bar (println "this is a bar!" v)
(throw (ex-info (str "Did not recognize key "
key)
{:key k
:map v})))))
;; handle with multimethods:
(defmulti do-thing-depending-on-key first)
(defmethod do-thing-depending-on-key :foo [[k v]]
(println "this is a foo!" v))
(defmethod do-thing-depending-on-key :bar [[k v]]
(println "this is a bar!" k))
(defmethod do-thing-depending-on-key :default [[k v]]
(throw (ex-info (str "Did not recognize key "
key)
{:key k
:map v})))
(run! do-thing-depending-on-key parsed)
;; this is a foo! {:id 2, :name foo2}
;; this is a bar! {:id 2, :name bar2}
;; => nil
(run! do-thing-depending-on-key {:unknown {:id 3 :name "Unknown"}})
;; =>
;; clojure.lang.ExceptionInfo:
;; Did not recognize key :unknown {:key :unknown, :map {:id 3, :name "Unknown"}}
Multimethods might be overkill for something this simple. I'd just use cond:
(ns tst.demo.core
(:require
[cheshire.core :as cheshire]
[clojure.java.io :as io] ))
(let [json-str (slurp (io/resource "data.json"))
edn-data (cheshire/parse-string json-str true) ; need `true` to emit keywords
]
(doseq [[type id-name-map] edn-data]
(cond
(= type :foo) (println :foo-proc id-name-map)
(= type :bar) (println :bar-proc id-name-map)
:else (println :other-proc id-name-map) )))
with results:
:foo-proc {:id 2, :name foo2}
:bar-proc {:id 2, :name bar2}
which requires the following in project.clj:
:dependencies [
[cheshire "5.8.0"]
....
and where your data is in resources/data.json:
~/expr/demo > cat resources/data.json
{
"foo": {
"id": 1,
"name": "foo1"
},
"bar": {
"id": 1,
"name": "bar1"
}, ....
The ARRAY expression works similar to the Array.prototype.map method in JS in that it translates elements from one array to a new array. One thing missing from ARRAY is the ability to get the index of the current element.
For example, is there a way to do this succinctly with ARRAY in n1ql?
// returns [{x: 7, i: 0}, {x: 3, i: 1}]
[{x: 7}, {x: 3}].map((obj, i) => ({...obj, i}));
Something like this would be a nice API:
SELECT ARRAY OBJECT_PUT(obj, "i", i) FOR obj, i IN [{x: 7}, {x: 3}] END
where you can define two variables after FOR: the first being the current array element and the optional second variable being the current array index.
Is there a valid way to do this easily?
You can use ARRAY_POSITION() function to get the index. https://developer.couchbase.com/documentation/server/4.6/n1ql/n1ql-language-reference/arrayfun.html#story-h2-15
And, yes multiple FOR expressions can be used in the ARRAY construct, which can be iterated together (i.e they should be of same size).
https://developer.couchbase.com/documentation/server/4.6/n1ql/n1ql-language-reference/collectionops.html
For ex:
SELECT ARRAY {i, j} FOR i IN [10, 11, 12], j IN [0, 1, 2] END;
[
{
"$1": [
{
"i": 10,
"j": 0
},
{
"i": 11,
"j": 1
},
{
"i": 12,
"j": 2
}
]
}
]
SELECT ARRAY {"x":v.x, "i": ARRAY_POS([{"x": 7}, {"x": 3}],v)} FOR v IN [{"x": 7}, {"x": 3}] END;
There is a large (does not fit in memory) .json file with the following content:
[{
"doc_number": "xxx",
"other": "data"
}, {
"doc_number": "yyy",
"other": "data"
}, {
"doc_number": "zzz",
"other": "data"
}]
I would like to read it as fast as possible using as little memory as possible. In other languages I usually create a lazy sequence of the file and read only when necessary. I was wondering if Erlang has an idiomatic way of achieving that.
jsx can be used as incremental parser, but for your format of data you have to write your own callback module:
-module(jsx_increment).
-export([parse_file/1]).
-export([init/1, handle_event/2]).
parse_file(FN) ->
{ok, File} = file:open(FN, [read, raw, binary]),
read(File, jsx:decoder(?MODULE, [], [stream, return_tail])),
file:close(File).
read(File, JSX) ->
{ok, Data} = file:read(File, 8), %% eof should raise error
case JSX(Data) of
{incomplete, F} ->
read(File, F);
{with_tail, _, Tail} ->
Tail =/= <<>> andalso io:format("Surplus content: ~s~n", [Tail])
end.
init(_) ->
start.
handle_event(start_array, start) ->
[];
handle_event(_, start) ->
error(expect_array);
handle_event(start_object, L) ->
[start_object|L];
handle_event(start_array, L) ->
[start_array|L];
handle_event(end_object, L) ->
check_out(collect_object(L));
handle_event(end_array, []) ->
stop;
handle_event(end_array, L) ->
check_out(collect_array(L));
handle_event(E, L) ->
check_out([event(E)|L]).
check_out([X]) ->
io:format("Collected object: ~p~n", [X]),
[];
check_out(L) -> L.
event({_, X}) -> X;
event(X) -> X.
collect_object(L) ->
collect_object(L, #{}).
collect_object([start_object|T], M) ->
[M|T];
collect_object([V, K|T], M) ->
collect_object(T, M#{K => V}).
collect_array(L) ->
collect_array(L, []).
collect_array([start_array|T], L) ->
[L|T];
collect_array([H|T], L) ->
collect_array(T, [H|L]).
And your example:
1> io:put_chars(element(2, file:read_file("data.json"))).
[{
"doc_number": "xxx",
"other": "data"
}, {
"doc_number": "yyy",
"other": "data"
}, {
"doc_number": "zzz",
"other": "data"
}]
ok
2> jsx_increment:parse_file("data.json").
Collected object: #{<<"doc_number">> => <<"xxx">>,<<"other">> => <<"data">>}
Collected object: #{<<"doc_number">> => <<"yyy">>,<<"other">> => <<"data">>}
Collected object: #{<<"doc_number">> => <<"zzz">>,<<"other">> => <<"data">>}
ok
It is proof of concept code which you have to adapt to your use case anyway, handle errors and so. (Used maps handling works only from R18. Use maps:put(K, V, M) for R17 and proplist for pre R17.)