I have a data structure that I want to convert to json and preserve the key order.
For example:
%{ x: 1, a: 5} should be converted to "{\"x\": 1, \"a\": 5}"
Poison does it without any problem. But when I upgrade to Jason, it changes to "{\"a\": 5, \"x\": 1}".
So I use JasonHelpers json_map to preserve the order like this:
Jason.Helpers.json_map([x: 1, a: 5])
It creates a fragment with correct order.
However, when I use a variable to do this:
list = [x: 1, a: 5]
Jason.Helpers.json_map(list)
I have an error:
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:list, [line: 15], nil} of type Tuple.
....
QUESTION: How can I pass a pre-calculated list into Jason.Helpers.json_map ?
The calculation is complicated, so I don't want to repeat the code just to use json_map, but use the function that returns a list.
json_map/1 is a macro, from its docs:
Encodes a JSON map from a compile-time keyword.
It is designed for compiling JSON at compile-time, which is why it doesn't work with your runtime variable.
Support for encoding keyword lists was added to the Jason library a year ago, but it looks like it hasn't been pushed to hex yet. I managed to get it work by pulling the latest code from github:
defp deps do
[{:jason, git: "https://github.com/michalmuskala/jason.git"}]
end
Then by creating a struct that implements Jason.Encoder (adapted from this solution by the Jason author):
defmodule OrderedObject do
defstruct [:value]
def new(value), do: %__MODULE__{value: value}
defimpl Jason.Encoder do
def encode(%{value: value}, opts) do
Jason.Encode.keyword(value, opts)
end
end
end
Now we can encode objects with ordered keys:
iex(1)> Jason.encode!(OrderedObject.new([x: 1, a: 5]))
"{\"x\":1,\"a\":5}"
I don't know if this is part of the public API or just an implementation detail, but it appears you have some control of the order when implementing the Jason.Encoder protocol for a struct.
Let's say you've defined an Ordered struct:
defmodule Ordered do
#derive {Jason.Encoder, only: [:a, :x]}
defstruct [:a, :x]
end
If you encode the struct, the "a" key will be before the "x" key:
iex> Jason.encode!(%Ordered{a: 5, x: 1})
"{\"a\":5,\"x\":1}"
Let's reorder the keys we pass in to the :only option:
defmodule Ordered do
#derive {Jason.Encoder, only: [:x, :a]}
defstruct [:a, :x]
end
If we now encode the struct, the "x" key will be before the "a" key:
iex> Jason.encode!(%Ordered{a: 5, x: 1})
"{\"x\":1,\"a\":5}"
I run following command in erlang,
os:cmd("curl -k -X GET http://10.210.12.154:10065/iot/get/task").
It gives a JSON output like this,
{"data":[
{"id":1,"task":"Turn on the bulb when the temperature in greater than 28","working_condition":1,"depending_value":"Temperature","action":"123"},
{"id":2,"task":"Trun on the second bulb when the temperature is greater than 30","working_condition":0,"depending_value":"Temperature","action":"124"}
]}
I want to categorize this data to Id, task, depending_value, action. It is like putting them in to a table. I want to easily find what is the depending value, working condition & action for Id=1. How can I do this?
It gives a JSON output like this.
{"data":[{"id":1,"t ...
Highly doubtful. The docs say that os:cmd() returns a string, which does not start with a {. Note also that a string is not even an erlang data type, rather double quotes are a shortcut for creating a list of integers, and a list of integers is not terribly useful in your case.
Here are two options:
Call list_to_binary() on the list of integers returned by os:cmd() to covert to a binary.
Instead of os:cmd(), use an erlang http client, like hackney, which will return the json as a binary.
The reason you want a binary is because then you can use an erlang json module, like jsx, to convert the binary into an erlang map (which might be what you are after?).
Here's what that will look like:
3> Json = <<"{\"data\": [{\"x\": 1, \"y\": 2}, {\"a\": 3, \"b\": 4}] }">>.
<<"{\"data\": [{\"x\": 1, \"y\": 2}, {\"a\": 3, \"b\": 4}] }">>
4> Map = jsx:decode(Json, [return_maps]).
#{<<"data">> =>
[#{<<"x">> => 1,<<"y">> => 2},#{<<"a">> => 3,<<"b">> => 4}]}
5> Data = maps:get(<<"data">>, Map).
[#{<<"x">> => 1,<<"y">> => 2},#{<<"a">> => 3,<<"b">> => 4}]
6> InnerMap1 = hd(Data).
#{<<"x">> => 1,<<"y">> => 2}
7> maps:get(<<"x">>, InnerMap1).
1
...putting them in to a table. I want to easily find what is the
depending value, working condition & action for Id=1.
Erlang has various table implementations: ets, dets, and mnesia. Here is an ets example:
-module(my).
-compile(export_all).
get_tasks() ->
Method = get,
%See description of this awesome website below.
URL = <<"https://my-json-server.typicode.com/7stud/json_server/db">>,
Headers = [],
Payload = <<>>,
Options = [],
{ok, 200, _RespHeaders, ClientRef} =
hackney:request(Method, URL, Headers, Payload, Options),
{ok, Body} = hackney:body(ClientRef),
%{ok, Body} = file:read_file('json/json.txt'), %Or, for testing you can paste the json in a file (without the outer quotes), and read_file() will return a binary.
Map = jsx:decode(Body, [return_maps]),
_Tasks = maps:get(<<"data">>, Map).
create_table(TableName, Tuples) ->
ets:new(TableName, [set, named_table]),
insert(TableName, Tuples).
insert(_Table, []) ->
ok;
insert(Table, [Tuple|Tuples]) ->
#{<<"id">> := Id} = Tuple,
ets:insert(Table, {Id, Tuple}),
insert(Table, Tuples).
retrieve_task(TableName, Id) ->
[{_Id, Task}] = ets:lookup(TableName, Id),
Task.
By default, an ets set type table ensures that the first position in the inserted tuple is the unique key (or you can explicitly specify another position in the tuple as the unique key).
** If you have a github account, I discovered a really cool website that allows you to place a json file in a new repository on github, and the website will serve up that file as json. Check it out at https://my-json-server.typicode.com:
How to
Create a repository on GitHub (<your-username>/<your-repo>)
Create a db.json file [in the repository].
Visit https://my-json-server.typicode.com/<your-username>/<your-repo> to
access your server
You can see the url I'm using in the code, which can be obtained by clicking on the link at the provided server page and copying the url in your web browser's address bar.
In the shell:
.../myapp$ rebar3 shell
===> Verifying dependencies...
===> Compiling myapp
src/my.erl:2: Warning: export_all flag enabled - all functions will be exported
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [kernel-poll:false]
Eshell V9.3 (abort with ^G)
1> ===> The rebar3 shell is a development tool; to deploy applications in production, consider using releases (http://www.rebar3.org/docs/releases)
===> Booted unicode_util_compat
===> Booted idna
===> Booted mimerl
===> Booted certifi
===> Booted ssl_verify_fun
===> Booted metrics
===> Booted hackney
1> Tasks = my:get_tasks().
[#{<<"action">> => <<"123">>,
<<"depending_value">> => <<"Temperature">>,<<"id">> => 1,
<<"task">> =>
<<"Turn on the bulb when the temperature in greater than 28">>,
<<"working_condition">> => 1},
#{<<"action">> => <<"124">>,
<<"depending_value">> => <<"Temperature">>,<<"id">> => 2,
<<"task">> =>
<<"Trun on the second bulb when the temperature is greater than 30">>,
<<"working_condition">> => 0}]
2> my:create_table(tasks, Tasks).
ok
3> my:retrieve_task(tasks, 1).
#{<<"action">> => <<"123">>,
<<"depending_value">> => <<"Temperature">>,<<"id">> => 1,
<<"task">> =>
<<"Turn on the bulb when the temperature in greater than 28">>,
<<"working_condition">> => 1}
4> my:retrieve_task(tasks, 2).
#{<<"action">> => <<"124">>,
<<"depending_value">> => <<"Temperature">>,<<"id">> => 2,
<<"task">> =>
<<"Trun on the second bulb when the temperature is greater than 30">>,
<<"working_condition">> => 0}
5> my:retrieve_task(tasks, 3).
** exception error: no match of right hand side value []
in function my:retrieve_task/2 (/Users/7stud/erlang_programs/old/myapp/src/my.erl, line 58)
6>
Note that the id is over to the right at the end of one of the lines. Also, if you get any errors in the shell, the shell will automatically restart a new process and the ets table will be destroyed, so you have to create it anew.
rebar.config:
{erl_opts, [debug_info]}.
{deps, [
{jsx, "2.8.0"},
{hackney, ".*", {git, "git://github.com/benoitc/hackney.git", {branch, "master"}}}
]}.
{shell, [{apps, [hackney]}]}. % This causes the shell to automatically start the listed apps. See https://stackoverflow.com/questions/40211752/how-to-get-an-erlang-app-to-run-at-starting-rebar3/45361175#comment95565011_45361175
src/myapp.app.src:
{application, 'myapp',
[{description, "An OTP application"},
{vsn, "0.1.0"},
{registered, []},
{mod, {'myapp_app', []}},
{applications,
[kernel,
stdlib
]},
{env,[]},
{modules, []},
{contributors, []},
{licenses, []},
{links, []}
]}.
But, according to the rebar3 dependencies docs:
You should add each dependency to your app or app.src files:
So, I guess src/myapp.app.src should look like this:
{application, 'myapp',
[{description, "An OTP application"},
{vsn, "0.1.0"},
{registered, []},
{mod, {'myapp_app', []}},
{applications,
[kernel,
stdlib,
jsx,
hackney
]},
{env,[]},
{modules, []},
{contributors, []},
{licenses, []},
{links, []}
]}.
My project parses JSONs, with a read/write library, called:
cheshire.core
I was having problems, trying to get the decode (func) to work, so I started messing around with:
data.json
My JSON contains data that consists of a field named "zone" this contains a vector with :keys inside, like so {:zone : [:hand :table]} that is stored into strings within the vector stored like so: {"zone" : ["hand" "table"]}
So I figured out how to convert the sample data using:
(mapv keyword {"zone" : ["hand"]})
which was great, I then needed to figure out how to implement a decoder for cheshire, I couldn't do this with my logic, I only spent like an hour working on this, but I had been using data.json, and the decoder function is relatively easy I think.
I got my project to work, here is some sample code:
(ns clojure-noob.core (:require
[cheshire.core :refer [decode]]
[clojure.data.json :as j-data]
) (:gen-class))
(defn -main
"I don't do a whole lot ... yet."
[& args]
)
this is using cheshire:
(let [init (decode "{\"zone\" : [\"hand\"]}" true
(fn [field-name]
(if (= field-name "zone")
(mapv keyword [])
[])))]
(println (str init)))
this is using data.json:
(defn my-value-reader [key value]
(if (= key :zone)
(mapv keyword value)
value))
(let [init (j-data/read-str
"{\"zone\" : [\"hand\"]}"
:value-fn my-value-reader
:key-fn keyword)]
(println (str init)))
I want the bottom result of these two from the console:
{:zone ["hand"]}
{:zone [:hand]}
The problem is I would like to do this using cheshire 😎
p.s. I am reading the factory section of cheshire? maybe this easier?
I would agree with #TaylorWood. Don't mess with the decoder, just do a bite in a time. First, parse json. Second, transform the result.
(def data "{\"zone\" : [\"hand\"]}")
(-> data
(cheshire.core/decode true)
(update-in ["zone"] (partial mapv keyword)))
#=> {:zone [:hand]}
I recommend you use a tool like schema.tools to coerce the input. You can add a second pass that attempts to coerce JSON strings into richer clojure types.
Here's some sample code!
;; require all the dependencies. See links below for libraries you need to add
(require '[cheshire.core :as json])
(require '[schema.core :as s])
(require '[schema.coerce :as sc])
(require '[schema-tools.core :as st])
;; your data (as before)
(def data "{\"zone\" : [\"hand\"]}")
;; a schema that wants an array of keywords
(s/defschema MyData {:zone [s/Keyword]})
;; use `select-schema` along with a JSON coercion matcher
(-> data
(json/decode true)
(st/select-schema MyData sc/json-coercion-matcher))
;; output: {:zone [:hand]}
Using defschema to define the shape of data you want gives you a general solution for serializing into JSON while getting the full benefit of Clojure's value types. Instead of explicitly "doing" the work of transforming, your schema describes the expected outcome, and hopefully coercions can do the right thing!
Links to libraries:
- https://github.com/plumatic/schema
- https://github.com/metosin/schema-tools#coercion
Note: you can do a similar thing with clojure.spec using metosin/spec-tools. Check out their readme for some help.
I'm using Cheshire to generate some JSON for data structures like this:
(require '[cheshire.core :refer [generate-string])
(generate-string {:id 123, :foo "something", :bar nil})
Which produces JSON like this:
{"id": 123, "foo": "something", "bar": null}
What I'd like is for the JSON to omit the keys without values; e.g.
{"id": 123, "foo": "something"}
Can Cheshire do this? I can certainly pre-filter the map before calling generate-string, but since Cheshire has to traverse my data structure anyway, I thought it would be more performant to instruct Cheshire to do the filtering.
No, null is a valid JSON value, so you should filter nil values yourself.
See this question for more info.
You may propose this feature to Cheshire team.