Using django query set values() to index into JSONField - json

I am using django with postgres, and have a bunch of JSON fields (some of them quite large and detailed) within my model. I'm in the process of switching from char based ones to jsonb fields, which allows me to filter on a key within the field, and I'm wondering if there is any way to get the equivalent benefit out of a call to the query set values method.
Example:
What I would like to do, given a Car model with options JSONField, is something like
qset = Car.objects.filter(options__interior__color='red')
vals = qset.values('options__interior__material')
Please excuse the lame toy problem, but hopefully it gets the idea across. Here the filter call does exactly what I want, but the call to values does not seem to be aware of the special nature of the JSON field. I get an error because values can't find the field called "interior" to join on. Is there some other syntax or option that I am missing that will make this work?
Seems like a pretty obvious extension to the existing functionality, but I have so far failed to find any reference to something similar in the docs or through stack overflow or google searches.
Edit - a workaround:
After playing around, looks like this could be fudged by inserting the following in between the two lines of code above:
qset=qset.annotate(options__interior__material=RawSQL("SELECT options->'interior'->'material'",()))
I say "fudged" because it seems like an abuse of notation and would require special treatment for integer indices.
Still hoping for a better answer.

I can suggest a bit cleaner way with using:
django's Func
https://docs.djangoproject.com/en/2.0/ref/models/expressions/#func-expressions
and postgres function jsonb_extract_path_text https://www.postgresql.org/docs/9.5/static/functions-json.html
from django.db.models import F, Func, CharField, Value
Car.objects.all().annotate(options__interior__material =
Func(
F('options'),
Value('interior'),
Value('material'),
function='jsonb_extract_path_text'
),
)

Perhaps a better solution (for Django >= 1.11) is to use something like this:
from django.contrib.postgres.fields.jsonb import KeyTextTransform
Car.objects.filter(options__interior__color='red').annotate(
interior_material=KeyTextTransform('material', KeyTextTransform('interior', 'options'))
).values('interior_material')
Note that you can nest KeyTextTransform expressions to pull out the value(s) you need.

Car.objects.extra(select={'interior_material': "options#>'{interior, material}'"})
.filter(options__interior__color='red')
.values('interior_material')
You can utilize .extra() and add postgres jsonb operators
Postgres jsonb operators: https://www.postgresql.org/docs/9.5/static/functions-json.html#FUNCTIONS-JSON-OP-TABLE

Related

Using cts query to retrieve collections associated with the given document uri- Marklogic

I need to retrieve the collections to which a given document belongs in Marklogic.
I know xdmp command does that. But I need to use it in cts query to retrieve the data and then filter records from it.
xdmp:document-get-collections("uri of document") can't be run inside cts-query to give appropriate data.
Any idea how can it be done using cts query?
Thanks
A few options come to mind:
Option One: Use cts:values()
cts:values(cts:collection-reference())
If you check out the documentation, you will see that you can also restrict this to certain fragments by passing a query as one of the parameters.
**Update: [11-10-2017]
The comment attached to this asked for a sample of restricting the results of cts:values() to a single document(for practical purposes, I will say fragment == document)
The documentation for cts:values explains this. It is the 4th parameter - a query to restrict the results. Get to know this pattern as it is part of many features of MarkLogic. It is your friend. The query I would use for this problem statement would be a cts:document-query();
An Example:
cts:values(
cts:collection-reference(),
(),
(),
cts:document-query('/path/to/my/document')
)
Full Example:
cts:search(
collection(),
cts:collection-query(
cts:values(
cts:collection-reference(),
(),
(),
cts:document-query('/path/to/my/document')
)
)
)[1 to 10]
Option two: use cts:collection-match()
Need more control over returning just some of the collections from a document, then use cts:colection-match(). Like the first option, you can restrict the results to just some fragments. However, it has the benefit of having an option for a pattern.
Attention:
They both return a sequence - perfect for feeding into other parts of your query. However, under the hood, I believe they work differently. The second option is run against a lexicon. The larger the list of unique collection names and the more complex your pattern match, the longer for resolution. I use collection-match in projects. However, I usually use it when I can limit the possible choices by restricting the results to a smaller number of documents.
You can't do this in a single step. You have to run code first to retrieve collections associated with a document. You can use something like xdmp:document-get-collections for that. You then have to feed that into a cts query that you build dynamically:
let $doc-collections := xdmp:document-get-collections($doc-uri)
return
cts:search(collection(), cts:collection-query($doc-collections))[1 to 10]
HTH!
Are you looking for cts:collection-query()?
Insert two XML files to the same collection:
xquery version "1.0-ml";
xdmp:document-insert("/a.xml", <root><sub1><a>aaa</a></sub1></root>,
map:map() => map:with("collections", ("coll1")));
xdmp:document-insert("/b.xml", <root><sub2><a>aaa</a></sub2></root>,
map:map() => map:with("collections", ("coll1")));
Search the collection:
xquery version "1.0-ml";
let $myColl:= xdmp:document-get-collections("/a.xml")
return
cts:search(/root,
cts:and-query((cts:collection-query($myColl),cts:element-query(xs:QName("a"),"aaa")
)))

Rails advanced search query

I have a database with Lab models. I want to be able to search them using multiple different methods.
I chose to use one input field and separate query into words array:
search = search.split(/[^[[:word:]]]+/).map{|val| val.downcase}
I use Acts-as-taggable gem so it would be nice to include those tags in search to:
tag_results = self.tagged_with(search, any: true, wild: true)
For methods down below it seemed to be necessary to use:
search = search.map{|val| "%#{val}%"}
Sunspot seemed also a great way to go for full-text search so
full_text_search = self.search {fulltext search}
full_text_results = full_text_search.results
I decided to go also with simple database query searching for a Lab name:
name_results = self.where("LOWER(name) ILIKE ANY ( array[?] )", search)
Lastly I need all of the results in one array so:
result = (tag_results + name_results + full_text_results).uniq
It works perfectly (what I mean is that the result is what I expect) but it returns a simple array and not ActiveRecord::Relation so there is no way for me to use method like .select() or .order() on the results.
I want to ask is there is some better way to implement such search? I was searching for search engines but it seems like there is nothing that would fit my idea.
If there is not - is there a way to convert an array into ActiveRecord::Relation? (SO says there is no way)
Answering this one:
is there a way to convert an array into ActiveRecord::Relation? (SO
says there is no way)
You can convert an array ob ActiveRecord objects into ActiveRecord::Relation by fetching ids from array and querying your AR model for objects with these ids:
Model.where(id: result.map(&:ids)) # returns AR, as expected.
It is the only way I am aware of.

Custom `returnFormat` in ColdFusion 10 or 11?

I've a function which is called from different components, .cfms or remotely. It returns the results of a query.
Sometimes the response from this function is manually inspected - a person may want to see the ID of a specific record so they can use it elsewhere.
The provided return formats, being wddx, json, plain all aren't very easily readable for a layman.
I'd love to be able to create a new return format: dump, where the result first writeDumped and then returned to the caller.
I know there'd be more complicated ways of solving this, like writing a function dump, and calling that like a proxy by providing the component, function and parameters so it can call that function and return the results.
However I don't think it's worth going that far. I figured it'd be great if I could just write a new return format, because that's just... intuitive and nice, and I may also be able to use that technique to solve different problems or improve various workflows.
Is there a way to create custom function returnFormats in ColdFusion 10 or 11?
(From comments)
AFAIK, you cannot add a custom returntype to a cffunction, but take a look at OnCFCRequest. Might be able to use it to build something more generic that responds differently whenever a custom URL parameter is passed, ie url.returnformat=yourType. Same net effect as dumping and/or manipulating the result manually, just a little more automated.
From the comments, the return type of the function is query. That being the case, there is simply no need for a custom return format. If you want to dump the query results, do so.
queryVar = objectName.nameOfFunction(arguments);
writeDump (queryVar);

Get rails ActiveRecord instead of Array of objects for multiple queries

I have a table that stores queries that return a list of users.
I then have a method "get_public" to a "Banana" model that execute multiple queries using logic AND between them.
So, when I do
Banana.find(x).get_public I receive an Array of users (the ones suitable to that banana object).
The get_public method is like this:
def get_public
pb = []
banana_queries.each do |q|
pb << User.find_by_sql(q.query)
end
pb.inject(:'&')
end
But, would be great if I could get ActiveRecord::Relation instead. I want to do something like this after: Banana.find(x).get_public.where(...)
Any way to modify get_public and achieve this?
I am not sure I correcly undestood the problem, but I will try to help anyway.
As especified here
where returns an ActiveRecord::Relation
find (and its related dynamic methods) returns a single model object
So I suggest divide your queries into: 'joins' and 'where' fields. Your new code should look like something like this:
pb << User.joins(q.query_joins).where(q.query_where)
Also find methods will are deprecated in rails 4, so using where is recommended.
Hope I haven't missed the point too much :-)

Is it okay to rely on automatic pass-by-reference to mutate objects?

I'm working in Python here (which is actually pass-by-name, I think), but the idea is language-agnostic as long as method parameters behave similarly:
If I have a function like this:
def changefoo(source, destination):
destination["foo"] = source
return destination
and call it like so,
some_dict = {"foo": "bar"}
some_var = "a"
new_dict = changefoo(some_var, some_dict)
new_dict will be a modified version of some_dict, but some_dict will also be modified.
Assuming the mutable structure like the dict in my example will almost always be similarly small, and performance is not an issue (in application, I'm taking abstract objects and changing into SOAP requests for different services, where the SOAP request will take an order of magnitude longer than reformatting the data for each service), is this okay?
The destination in these functions (there are several, it's not just a utility function like in my example) will always be mutable, but I like to be explicit: the return value of a function represents the outcome of a deterministic computation on the parameters you passed in. I don't like using out parameters but there's not really a way around this in Python when passing mutable structures to a function. A couple options I've mulled over:
Copying the parameters that will be mutated, to preserve the original
I'd have to copy the parameters in every function where I mutate them, which seems cumbersome and like I'm just duplicating a lot. Plus I don't think I'll ever actually need the original, it just seems messy to return a reference to the mutated object I already had.
Just use it as an in/out parameter
I don't like this, it's not immediately obvious what the function is doing, and I think it's ugly.
Create a decorator which will automatically copy the parameters
Seems like overkill
So is what I'm doing okay? I feel like I'm hiding something, and a future programmer might think the original object is preserved based on the way I'm calling the functions (grabbing its result rather than relying on the fact that it mutates the original). But I also feel like any of the alternatives will be messy. Is there a more preferred way? Note that it's not really an option to add a mutator-style method to the class representing the abstract data due to the way the software works (I would have to add a method to translate that data structure into the corresponding SOAP structure for every service we send that data off too--currently the translation logic is in a separate package for each service)
If you have a lot of functions like this, I think your best bet is to write a little class that wraps the dict and modifies it in-place:
class DictMunger(object):
def __init__(self, original_dict):
self.original_dict = original_dict
def changefoo(source)
self.original_dict['foo'] = source
some_dict = {"foo": "bar"}
some_var = "a"
munger = DictMunger(some_dict)
munger.changefoo(some_var)
# ...
new_dict = munger.original_dict
Objects modifying themselves is generally expected and reads well.