I want to add a timestamp to my macro log that shows the current date and time of the local place. I tried to place the run_started_at function in and out of the log, but it doesn't work. I prefer to have inside the log. ie,'Operation Completed 01:10:2023 11:30' Below are the works I have done;
{% macro create_local_database(username, reset_environment = false) %}
{% set setup_script1 %}
{%- if reset_environment == false -%}
CREATE DATABASE ANALYTICS_LOCAL_MART_{{username}}
{% do log('Operation Completed', True) %} {{ run_started_at.strftime("%Y-%m-%d") }}
{%- endif -%}
{% endset %}
{% endmacro %}
I would like to have a macro log that includes the message and timestamp.
Just append the timestamp to the string you send to log()
{%- macro logt(msg) -%}
{{ log(this ~ " (" ~ run_started_at.strftime('%Y-%m-%d') ~ "): " ~ msg, true) }}
{%- endmacro -%}
Related
I want to develop a macro that will loop rows from a seeding query's result and create a dynamic query for another task.
Let's assume my row would be similar to
<agate.Row: ('product_available', Decimal('0.6'), 'Positive')>
<agate.Row: ('product_quality', Decimal('0.5'), 'Negative')>
I intend to generate an array selection for downstream queries which have my_udf that will take arguments from my seeding query's result. For example,
my_udf("product_available", 0.6, 'Positive')
my_udf("product_quality", 0.5, 'Negative')
The problem that I have is some arguments are actual column names, while others are values. Hence, column names should have double quotes, while value must have single quote. For instance, "product_available" vs 'Positive'
My code is
{% macro generate_list_select_from(seeding_query) %}
{# ... other code to execute my seeding query ... #}
{# loop query goes here #}
{% for i in results_list %}
{% set item = "my_udf( {{ i[0] }} , {{ i[1] }}, '{{ i[2] }}' )" %}
{{items.append(item)}}
{% endfor %}
{{ return(items) }}
{% endmacro %}
Below is output when I use my macro
select foo_column,
my_udf( {{ i[0] }} , {{ i[1] }}, '{{ i[2] }}' ),
my_udf( {{ i[0] }} , {{ i[1] }}, '{{ i[2] }}' )
from foo_table
My question is how to create a such string?
Update:
Tried other way, {% set item = "my_udf(" + {{i[0]}} + ")" %}, I end up with a compilation error expected token ':', got '}'
Don't nest your curlies.
~ is the string concatenation operator in jinja. You could use that to build up your function args, including the single quotes:
{% set item = "my_udf(" ~ i[0] ~ ", " ~ i[1] ~ ", '" ~ i[2] ~ "' )" %}
This is pretty hard to read, though. I'd probably add a macro called quoted:
{% macro quoted(s) %}
'{{ s }}'
{% endmacro %}
And then use the join filter to concatenate the items of a list:
{% set args = [i[0], i[1], quoted(i[2])] %}
{% set item = "my_udf(" ~ args | join(", ") ~ ")" %}
Lastly, if the purpose of this macro is to template SQL, then you don't need to use return(items). You should just have the body of the macro template the string you want (see the quoted macro above). That dramatically simplifies things:
{% macro quoted(s) %}
'{{ s }}'
{% endmacro %}
{% macro generate_list_select_from(seeding_query) %}
{# ... other code to execute my seeding query ... #}
{# loop query goes here #}
{% for i in results_list %}
{% set args = [i[0], i[1], quoted(i[2])] %}
my_udf({{ args | join(", "}}){% if not loop.last %},{% endif %}
{% endfor %}
{% endmacro %}
I have a use case where I would like to define the name of a macro and then apply it to one column.
A simplified example could be as follows. I have two macros defined that I want to call dynamically in my model (both take one column as an input):
cast_to_string
convert_empty_string_to_null_value
Now, I want to call them dynamically. See the example below
{%- set macro_names = ["cast_to_string", "convert_empty_string_to_null_value"] -%}
select
{% for macro_name in macro_names %}
-- this should dynamically be evaluated to `{{ cast_to_string(my_column) }}`
-- and `{{ convert_empty_string_to_null_value(my_column) }}`
{{ macro_name(my_column) }}
{% endfor %}
from my_model
However, this will throw an error saying that a string is not callable.
I also tried using {% raw %} {{ {% endraw %} to escape brackets, but that didn’t work either.
So, my question is, if there is a way to dynamically call macros in jinja/dbt?
I think it should work if you remove the quotes :
{%- set macro_names = [cast_to_string, convert_empty_string_to_null_value] -%}
So that jinja doesn't interpret it as string and you can use it as a Callable
I achieve it using this example :
{%- set macro_names = [print_lower, print_upper] -%}
{% for macro_name in macro_names %}
{{ macro_name("test") }}
{% endfor %}
and
{% macro print_lower(string) %}
{{ print(string|lower) }}
{% endmacro %}
{% macro print_upper(string) %}
{{ print(string|upper) }}
{% endmacro %}
I can't figure out how to remove the last empty line from the output generated by a for and an if Jinja2 statments on SaltStack. I couldn't find a post pointing out the solution neither, as the solutions on internet that I could find don't involve for and if nested statements.
Any help is welcome, thanks.
This is what I tried so far:
{% for user in list_users -%}
{% if user.enabled == True -%}
{{ user.name }}:{{ user.passwd }}
{%- endif %}
{% endfor %}
user1:encrypted_pass
user2:encrypted_pass
user3:encrypted_pass
I also tried adding - at the beginning of the {endfor}:
{% for user in list_users -%}
{% if user.enabled == True -%}
{{ user.name }}:{{ user.passwd }}
{%- endif %}
{%- endfor %}
user1:encrypted_passuser2:encrypted_passuser3:encrypted_pass
Edit: this is the output that I am trying to achieve:
user1:encrypted_pass
user2:encrypted_pass
user3:encrypted_pass
PS: I had a look to the Jinja's whitespace control without success.
I want to check if a specific version of a program is already installed.
Therefore, I got a state file:
{% set rvs = ['1113','1278'] %}
{% for rv in rvs %}
{% if ('r{{ rv }}' not in grains.get('cat12', [])) %}
... install it ...
{% else %}
... do nothing ...
{% endif %}
{% endfor %}
In my grains I have:
cat12:
- r1113
I would expect that '1278' is installed and list item '1113' triggers nothing, but even that is installed again...
There's no such syntax as using {{ and }} inside {% and %}. What's inside {% and %} is already Jinja. Just concatenate the string literal and the string variable with the ~ operator.
{% if 'r' ~ rv not in grains.get('cat12', []) %}
Or you can use the format filter:
{% if 'r%s'|format(rv) not in grains.get('cat12', []) %}
How can I recursively merge dicts in Jinja2 without having to define a custom filter?
So
{ a: { b: 'ab', d: 'ad' } }
+ { a: { b: 'aX', e: 'ae' } }
becomes
{ a: { b: 'aX', d: 'ad', e: 'ae' } }
I'm migrating some Mako-templates to Jinja2. With Mako I could simply use a small python-code snippet doing this for me... The reason I cannot use a custom filter is, that I simply cannot register custom filters in the template engine as I cannot directly access it.
I finally found a solution to my problem:
{%- macro deep_merge(a, b): %}
{%- for k,v in b.items(): %}
{%- if v is not defined: %}
{%- do a.pop(k) %}
{%- else: %}
{%- if v is mapping: %}
{%- if a[k] is not mapping: %}
{%- do a.update({ k: { } }) %}
{%- endif %}
{%- do deep_merge(a[k], v) %}
{%- else: %}
{%- do a.update({ k: v }) %}
{%- endif %}
{% endif %}
{%- endfor %}
{%- endmacro %}
{%- do deep_merge(a, b) %}
This snippet of Jinja2-Code recursively merges b into a, overriding duplicate keys (b takes precedence) and removing keys from a that are set to null in b.
It's ugly, but it sounds like your situation is a bit Abby Normal:
{# Untested code - may not work #}
{% macro merge(destination) %}
{% for provider in varargs %}
{% for key, value in provider.items() %}
{% if value is mapping %}
{% set intermediate = {} %}
{% do merge(intermediate, destination.get(key, {}), value) %}
{% do destination[key] = intermediate %}
{% else %}
{% do destination[key] = value %}
{% endif %}
{% endfor %}
{% endfor %}
{% endmacro %}
And you can use it like this:
{% set result = {} %}
{% do merge(result, first_values, second_values) %}
{# result is now the merger of first_values and second_values #}