I'm using Cheshire to generate some JSON for data structures like this:
(require '[cheshire.core :refer [generate-string])
(generate-string {:id 123, :foo "something", :bar nil})
Which produces JSON like this:
{"id": 123, "foo": "something", "bar": null}
What I'd like is for the JSON to omit the keys without values; e.g.
{"id": 123, "foo": "something"}
Can Cheshire do this? I can certainly pre-filter the map before calling generate-string, but since Cheshire has to traverse my data structure anyway, I thought it would be more performant to instruct Cheshire to do the filtering.
No, null is a valid JSON value, so you should filter nil values yourself.
See this question for more info.
You may propose this feature to Cheshire team.
Related
Using the JSONiq to JavaScript implementation of JSONiq, say I have an array
let $a := [1,2,3]
I'd like to get the elements as a sequence, but all of these return the array itself -
return $a()
return $a[]
return members($a)
What is the correct way to extract the members of the array?
My ultimate goal is to convert objects in an array to strings, like so -
let $updates := [
{"address": "%Q0.1", "keys": ["OUT2", "output.2"], "value": 0},
{"address": "%Q0.7", "keys": ["OUT8", "output.8"], "value": 1}
]
for $update in $updates()
return "<timestamp>|address|" || $update.address
in order to convert an array of JSON objects to a set of strings like <timestamp>|address|%Q0.7, etc
Edit: Using Zorba the $a() syntax seems to work okay - is it an issue with the node jsoniq parser?
e.g.
jsoniq version "1.0";
let $updates := [
{"address": "%Q0.1", "keys": ["OUT2", "output.2"], "value": 0},
{"address": "%Q0.7", "keys": ["OUT8", "output.8"], "value": 1}
]
for $update in $updates()
return current-dateTime() || "|address|" || $update.address
returns
2021-02-19T23:10:13.434273Z|address|%Q0.1 2021-02-19T23:10:13.434273Z|address|%Q0.7
In the core JSONiq syntax, an array is turned into a sequence (i.e., its members are extracted) with an empty pair or square brackets, like so:
$array[]
Example:
[1, 2, 3, 4][]
returns the sequence:
(1, 2, 3, 4)
This means that the query would be:
let $updates := [
{"address": "%Q0.1", "keys": ["OUT2", "output.2"], "value": 0},
{"address": "%Q0.7", "keys": ["OUT8", "output.8"], "value": 1}
]
for $update in $updates[]
return "<timestamp>|address|" || $update.address
The function-call-like notation with an empty pair of parenthesis dates back to JSONiq's early days, as it was primarily designed as an extension to XQuery and maps and arrays were navigated with function calls ($object("foo"), $array(), $array(2)). As JSONiq started having its own life, though, more user-friendly and intuitive syntax for JSON navigation was introduced:
$array[[1]]
for array member lookup given a position
$object.foo
for object lookup given a key and
$array[]
for array unboxing.
While the JSONiq extension to XQuery still exists for scenarios in which users need both JSON and XML support (and is supported by Zorba 3.0, IBM Websphere, etc), the core JSONiq syntax is the main one for all engines that specifically support JSON, like Rumble.
Some engines (including Zorba 3.0) support both the core JSONiq syntax and the JSONiq extension to XQuery, and you can pick the one you want with a language version declaration:
jsoniq version "1.0";
[1, 2, 3, 4][]
vs.
xquery version "3.0";
[1, 2, 3, 4]()
Zorba is relatively lenient and will probably even accept both () and [] in its core JSONiq implementation.
(Warning: Zorba 2.9 doesn't support the latest core JSONiq syntax, in particular the try.zorba.io page still runs on Zorba 2.9. You need to download Zorba 3.0 and run it locally if you want to use it).
A final note: JSON navigation works in parallel, on sequences of arrays and objects, too:
(
{"foo":1},
{"foo":2},
{"foo":3},
{"foo":4}
).foo
returns
(1, 2, 3, 4)
while
(
[1, 2],
[3, 4, 5],
[6, 7, 8]
)[]
returns
(1, 2, 3, 4, 5, 6, 7, 8)
This makes it very easy and compact to navigate large sequences:
$collection.foo[].bar[[1]].foobar[].foo
I'm trying to pass a clojurescript map to a webworker.
Before I pass it, it is of type PersistentArrayMap.
cljs.core.PersistentArrayMap {meta: null, cnt: 3, arr: Array(6), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951…}
However, when it gets to the worker, it's just a plain old Object
Object {meta: null, cnt: 3, arr: Array(6), __hash: null, cljs$lang$protocol_mask$partition0$: 16647951…}
with the data seemingly intact. At this point I'd like to turn it back into a PersistentArrayMap so that I can work with it in cljs again.
Using clj->js and js->clj doesn't really work because it doesn't distinguish between keywords and strings, so some data is lost.
What's the best way to handle this situation? It's possible that I'm going about this in the entirely wrong way.
The built-in solution is to serialize the data to EDN and reading it back. clj->js is inherently lossy and should be avoided.
You start by turning the object into a string and send that over to the worker.
(pr-str {:a 1})
;; => "{:a 1}"
In the worker you read it back via cljs.reader/read-string
(cljs.reader/read-string "{:a 1}")
;; => {:a 1}
This will usually be good enough but the transit-cljs library will be slightly faster. Depending on the amount of data you plan on sending it may be worth the extra dependency.
Did you use keywordize-keys in the code?
(def json "{\"foo\": 1, \"bar\": 2, \"baz\": [1,2,3]}")
(def a (.parse js/JSON json))
;;=> #js {:foo 1, :bar 2, :baz #js [1 2 3]}
(js->clj a)
;;=> {"foo" 1, "bar" 2, "baz" [1 2 3]}
(js->clj a :keywordize-keys true)
;;=> {:foo 1, :bar 2, :baz [1 2 3]}
Full documentation is here.
I'm having trouble serializing my ruby object to json, more specifically the format of the numbers.
I have written an rspec test to illustrate my issue more precisely.
expected = '{ "foo": 1.00, "bar": 4.50, "abc": 0.00, "xyz": 1.23 }'
it 'serializes as expected' do
my_hash = { "foo": 1, "bar": 4.5, "abc": 0, "xyz": 1.23}
expect(my_to_json_method(my_hash)).to eq expected
end
This is the case that I am having trouble with. I can use the sprintf but how do I get the string output as shown in the above example?
First of all, you should not use floats to represent monetary values. So instead, let's use a more appropriate type: (there's also the Ruby Money gem)
require 'bigdecimal'
my_hash = {
foo: BigDecimal.new('1.00'),
bar: BigDecimal.new('4.50'),
abc: BigDecimal.new('0.00'),
xyz: BigDecimal.new('1.23')
}
There are several options to represent monetary values. All of the following JSON strings are valid according to the JSON specification and all require special treatment upon parsing. It's up to you to choose the most appropriate.
Note: I'm implementing a custom to_json method to convert the BigDecimal instances to JSON using Ruby's default JSON library. This is just for demonstration purposes, you should generally not patch core (or stdlib) classes.
1. Numbers with fixed precision
This is what you asked for. Note that many JSON libraries will parse these numbers as floating point values by default.
class BigDecimal
def to_json(*)
'%.2f' % self
end
end
puts my_hash.to_json
Output:
{"foo":1.00,"bar":4.50,"abc":0.00,"xyz":1.23}
2. Numbers as strings
This will work across all JSON libraries, but storing numbers as strings doesn't look quite right to me.
class BigDecimal
def to_json(*)
'"%.2f"' % self
end
end
puts my_hash.to_json
Output:
{"foo":"1.00","bar":"4.50","abc":"0.00","xyz":"1.23"}
3. Numbers as integers
Instead of representing monetary values as fractional numbers, you simply output the cents as whole numbers. This is what I usually do.
class BigDecimal
def to_json(*)
(self * 100).to_i.to_s
end
end
puts my_hash.to_json
Output:
{"foo":100,"bar":450,"abc":0,"xyz":123}
User Sprintf
sprintf('%.2f', 5.5)
And simply interpolate into your JSON as an ERB template.
You can use, sprintf and can take as many decimal points as you needed by mentioning %.(number)f.
Eg: For two decimals, %.2f
Here is a real implementation,
2.2.2 :019 > test = { "foo": (sprintf "%.2f","1.11"), "bar": (sprintf "%.2f","4.55"), "abc": (sprintf "%.2f","0.2") }
=> {:foo=>"1.11", :bar=>"4.55", :abc=>"0.20"}
Here is the reference
puts '{' << my_hash.map { |k, v| %Q|"#{k}": #{"%.2f" % v}| }.join(', ') << '}'
#⇒ {"foo": 1.00, "bar": 4.50, "abc": 0.00, "xyz": 1.23}
I have a JSON file that, for now, is validated by hand prior to being placed into production. Ideally, this is an automated process, but for now this is the constraint.
One thing I found helpful in Eclipse were the JSON tools that would highlight duplicate keys in JSON files. Is there similar functionality in Sublime Text or through a plugin?
The following JSON, for example, could produce a warning about duplicate keys.
{
"a": 1,
"b": 2,
"c": 3,
"a": 4,
"d": 5
}
Thanks!
There are plenty of JSON validators available online. I just tried this one and it picked out the duplicate key right away. The problem with using Sublime-based JSON linters like JSONLint is that they use Python's json module, which does not error on extra keys:
import json
json_str = """
{
"a": 1,
"b": 2,
"c": 3,
"a": 4,
"d": 5
}"""
py_data = json.loads(json_str) # changes JSON into a Python dict
# which is unordered
print(py_data)
yields
{'c': 3, 'b': 2, 'a': 4, 'd': 5}
showing that the first a key is overwritten by the second. So, you'll need another, non-Python-based, tool.
Even Python documentation says that:
The RFC specifies that the names within a JSON object should be
unique, but does not mandate how repeated names in JSON objects should
be handled. By default, this module does not raise an exception;
instead, it ignores all but the last name-value pair for a given name:
weird_json = '{"x": 1, "x": 2, "x": 3}'
json.loads(weird_json) {'x': 3}
The object_pairs_hook parameter can be used to alter this behavior.
So as pointed from docs:
class JsonUniqueKeysChecker:
def __init__(self):
self.keys = []
def check(self, pairs):
for key, _value in pairs:
if key in self.keys:
raise ValueError("Non unique Json key: '%s'" % key)
else:
self.keys.append(key)
return pairs
And then:
c = JsonUniqueKeysChecker()
print(json.loads(json_str, object_pairs_hook=c.check)) # raises
JSON is very easy format, not very detailed so things like that can be painful. Detection of doubled keys is easy but I bet it's quite a lot of work to forge plugin from that.
I have created a defrecord in a Clojure REPL:
user=> (defrecord Data [column1 column2 column3])
user.Data
How do I automate adding data to this record by reading in a .json file? Each of the columns in the defrecord corresponds exactly to a key in the json data. If the file contained a single record it would look similar to this:
[
{
"column1" : "value1"
"column2" : "value2"
"column3" : "value3"
}
]
But there are many thousands of such records in the file.
I can slurp the contents of the file like this:
(json/read-json (slurp "path/to/file.json")))
The dependencies for the read-json function are added to the project.clj file found in the directory where I am running lein repl from the command line: :dependencies [org.clojure/data.json "0.2.1"].
I would just like to be able to search the values of the records using a Clojure function, such that the value I am passing to the search function is between the values of a single record's column1 and column2 values (i.e., nth-record.column1.value <= query <= nth-record.column2.value). Once I've found a matching record, I want to return the value of another column in that same record (nth-record.column3.value). The values of columns 1 and 2 will be unique, representing a non-overlapping range of values. The value of column3 is not unique.
This seems like a fairly trivial task, but I can't figure out how to do it using the Clojure documentation or the examples I've found online. It doesn't matter to me how the records are stored internally in Clojure, as long as I can search them and return the value of a related field in the same record.
Using data.json package:
(require '[clojure.data.json :as json])
Read values into memory:
(def all-records (json/read-str (slurp "path/to/file.json")
:key-fn keyword))
;; ==> [ { :column1 "value1", :column2 "value2", :column3 "value3" }, ...]
Find matching records:
(def query "some-value")
(def matching (filter #(and (< (:column1 %) query) (< query (:column2 %))) all-records))
Get column3:
(map :column3 matching)
Collecting it all together (and making it more flexible):
(defn find-matching [select-fn result-fn records]
(map result-fn (filter select-fn records)))
(defn select-within [rec query]
(and (< (:column1 rec) query) (< query (:column2 rec))))
(find-matching #(select-within % "some-value") :column3 all-records)
Should probably use cheshire for speed.
If your queries get sufficiently complex, consider lucene, clojure has a nice wrapper.
I think you're thinking records are somehow more suitable for this than maps, as far as I can tell, you're not using any features that make records special like polymorphism. There might be a way to make cheshire spit out records, but I wouldn't bother.