STR_TO_DATE(string_time,'%Y-%m-%d %H:%i:%s') > (select timestamp from table)
this line causes the error 'Function str_to_date not registered' in Athena. is there any way to work around this problem?
All databases have their own set of functions, even though some are common and exist in more than one. STR_TO_DATE is not available in Athena, but there are lots of other date and time functions that can be used to achieve the same goal.
You can find links to all functions supported by Athena here: https://docs.aws.amazon.com/athena/latest/ug/presto-functions.html
In your case I think you can use either parse_datetime, which looks like it works like STR_TO_DATE in your example.
Alternatively I think you could cast the string to a timestamp since the format you are using matches Athena's, try CAST(string_time AS TIMESTAMP)
I'm using bookshelf with postgresql database
Information is a column of type json.
I want to retrieve all column that are like '%pattern%'
With sql query i use
select * from table where information::text like '%pattern%';
I want to do that with bookshelf query builder
model.query(function(qb) {
qb.where('information', 'LIKE', '%pattern%')
}).fetch()
But it didn't work and i can't find how to do it in bookshelf docs
Any idea?
The tricky part here is, although you might think that JSON (and JSONB) columns are text, they aren't! So there's no way to do a LIKE comparison on one. Well, there is, but you'd have to convert it to a string first:
SELECT * FROM wombats WHERE information #>> '{}' LIKE '%pattern%';
which is a really terrible idea, please don't do that! As #GMB points out in the comments, JSON is a structured format that is far more powerful. Postgres is great at handling JSON, so just ask it for what you need. Let's say your value is in a JSON property named description:
SELECT * FROM wombats
WHERE (information->'description')::TEXT
LIKE '%pattern%';
Here, even though we've identified the correct property in our JSON object, it comes out as type JSON: we still have to cast it to ::TEXT before comparing it with a string using LIKE. The Bookshelf/Knex version of all this would look like:
model
.query(function(qb) {
const keyword = "pattern";
qb.whereRaw(`(information->'description')::TEXT LIKE '%${keyword}%'`)
})
.fetch();
Apparently this part of the raw query cannot be parameterized (in Postgres, at least) so the string substitution in JavaScript is required. This means you should be extra careful with where that string comes from (ie only use a limited subset, or sanitise before use) as you're bypassing Knex's usual protections.
So, I have a list of "whitelisted" countries (eg. ['PT', 'US', 'UK', 'ES']) on a column of a table, and I want to check if the user's country (eg. 'US') is whitelisted, in order to show the user the row's content.
I searched and the best and easiest way I found was to compile the whitelisted list into a JSON array, make the column JSON type and use the JSON_CONTAINS function present on MySQL 5.7+
However, I can't figure out how to implement that with CI's database library.
How can I use CI's DB lib to use MySQL's functions? Would there be a better way to achieve this instead of JSON array?
You can use
$this->db->where("(JSON_CONTAINS(field,'[\"US\"]')) > ",0);
The codeigniter where clause required an operator. If we provide an operator codeigniter will ignore IS NULL
I'm trying to convert a date type column into a nice human readable string like so: 25/11/2016 (or any other masks I'd like to use)
Does Big Query supports masks when using dates? When I use the Date() Functions it returns something like "2016-05-05" but that's not the standard pattern in many countries.
I've searched for a lot of different things the closes thing I got is this doc: https://cloud.google.com/bigquery/query-reference but I didn't see anythin that would help me
check STRFTIME_UTC_USEC
SELECT STRFTIME_UTC_USEC(CURRENT_DATE(), '%d/%m/%Y')
I'm using Django 1.9 with its built-in JSONField and Postgres 9.4.
In my model's attrs json field I store objects with some values, including numbers. And I need to aggregate over them to find min/max values.
Something like this:
Model.objects.aggregate(min=Min('attrs__my_key'))
Also, it would be useful to extract specific keys:
Model.objects.values_list('attrs__my_key', flat=True)
The above queries fail with
FieldError: "Cannot resolve keyword 'my_key' into field. Join on 'attrs' not permitted."
Is it possible somehow?
Notes:
I know how to make a plain Postgres query to do the job, but am searching specifically for an ORM solution to have the ability to filter etc.
I suppose this can be done with a (relatively) new query expressions/lookups API, but I haven't studied it yet.
From django 1.11 (which isn't out yet, so this might change) you can use django.contrib.postgres.fields.jsonb.KeyTextTransform instead of RawSQL .
In django 1.10 you have to copy/paste KeyTransform to you own KeyTextTransform and replace the -> operator with ->> and #> with #>> so it returns text instead of json objects.
Model.objects.annotate(
val=KeyTextTransform('json_field_key', 'blah__json_field'))
).aggregate(min=Min('val')
You can even include KeyTextTransforms in SearchVectors for full text search
Model.objects.annotate(
search=SearchVector(
KeyTextTransform('jsonb_text_field_key', 'json_field'))
)
).filter(search='stuff I am searching for')
Remember you can also index in jsonb fields, so you should consider that based upon your specific workload.
For those who interested, I've found the solution (or workaround at least).
from django.db.models.expressions import RawSQL
Model.objects.annotate(
val=RawSQL("((attrs->>%s)::numeric)", (json_field_key,))
).aggregate(min=Min('val')
Note that attrs->>%s expression will become smth like attrs->>'width' after processing (I mean single quotes). So if you hardcode this name you should remember to insert them or you will get error.
/// A little bit offtopic ///
And one more tricky issue not related to django itself but that is needed to be handled somehow. As attrs is json field and there're no restrictions on its keys and values you can (depending on you application logic) get some non-numeric values in, for example, width key. In this case you will get DataError from postgres as a result of executing the above query. NULL values will be ignored meanwhile so it's ok. If you can just catch the error then no problem, you're lucky. In my case I needed to ignore wrong values and the only way here is to write custom postgres function that will supress casting errors.
create or replace function safe_cast_to_numeric(text) returns numeric as $$
begin
return cast($1 as numeric);
exception
when invalid_text_representation then
return null;
end;
$$ language plpgsql immutable;
And then use it to cast text to numbers:
Model.objects.annotate(
val=RawSQL("safe_cast_to_numeric(attrs->>%s)", (json_field_key,))
).aggregate(min=Min('val')
Thus we get quite solid solution for such a dynamic thing as json.
I know this is a bit late (several months) but I came across the post while trying to do this. Managed to do it by:
1) using KeyTextTransform to convert the jsonb value to text
2) using Cast to convert it to integer, so that the SUM works:
q = myModel.objects.filter(type=9) \
.annotate(numeric_val=Cast(KeyTextTransform(sum_field, 'data'), IntegerField())) \
.aggregate(Sum('numeric_val'))
print(q)
where 'data' is the jsonb property, and 'numeric_val' is the name of the variable I create by annotating.
Hope this helps somebody!
It is possible to do this using a Postgres function
https://www.postgresql.org/docs/9.5/functions-json.html
from django.db.models import Func, F, FloatField
from django.db.models.expressions import Value
from django.db.models.functions import Cast
text = Func(F(json_field), Value(json_key), function='jsonb_extract_path_text')
floatfield = Cast(text, FloatField())
Model.objects.aggregate(min=Min(floatfield))
This is much better than using the RawQuery because it doesn't break if you do a more complex query, where Django uses aliases and where there are field name collisions. There is so much going on with the ORM that can bite you with hand written implementations.
Since Django 3.1 the KeyTextTransform function on a JSON field works for all database backends. It maps to the ->> operator in Postgres.
It can be used to annotate a specific JSON value inside a JSONField on the queryset results before you aggregate it. A more clear example how to utilize this:
First we need to annotate the key you want to aggregate. So if you have a Django model with a JSONField named data and the JSON containing looks like this:
{
"age": 43,
"name" "John"
}
You would annotate the queryset as following:
from django.db.models import IntegerField
from django.db.models.fields.json import KeyTextTransform
qs = Model.objects.annotate(
age=Cast(
KeyTextTransform("age", "data"), models.IntegerField()
)
The Cast is needed to stay compatible with all database backend.
Now you can aggregate to your liking:
from django.db.models import Min, Max, Avg, IntegerField
from django.db.models.functions import Cast, Round
qs.aggregate(
min_age=Round(Min("age")),
max_age=Round(Max("age")),
avg_age=Cast(Round(Avg("age")), IntegerField()),
)
>>> {'min_age': 25, 'max_age' 82:, 'avg_age': 33}
Seems there is no native way to do it.
I worked around like this:
my_queryset = Product.objects.all() # Or .filter()...
max_val = max(o.my_json_field.get(my_attrib, '') for o in my_queryset)
This is far from being marvelous, since it is done at the Python Level (and not at the SQL level).
from django.db.models.functions import Cast
from django.db.models import Max, Min
qs = Model.objects.annotate(
val=Cast('attrs__key', FloatField())
).aggregate(
min=Min("val"),
max=Max("val")
)