I'm trying to invoke a macro as a post-hook. The trouble is (I think) is that I'd like to pass a list to this macro... any idea what's going on here? My theory is that I'm passing a list type argument.
-- models/table.sql
{{
config(
materialized = 'table',
post-hook = "{{ my_macro(this,'my_str', ['foo', 'bar']) }}"
)
}}
SELECT * FROM muh_tayble;
-- macros/my_macro.sql
{% macro my_macro(relation, string, list) %}
BLAH
{% endmacro %}
error message
Encountered an error:
Compilation Error in model table (models/table.sql)
invalid syntax for function call expression
line 2
Rookie mistake folks. I had post-hook instead of post_hook. Problem solved
Related
i followed the document https://docs.getdbt.com/guides/best-practices/writing-custom-generic-tests and was trying to write the generic test "is_even" from here but when i am running the below generic test called is_even using dbt test command,it is giving me the error - "Database Error in test is_even_my_first_dbt_model_id (models/example/schema.yml)
Syntax error: Unexpected "%" at [25:23]"
test_is_even.sql -
{% test is_even(model, column_name) %}
with validation as (
select
{{ column_name }} as even_field
from {{ model }}
),
validation_errors as (
select
even_field
from validation
-- if this is true, then even_field is actually odd!
where (even_field % 2) = 1
)
select *
from validation_errors
{% endtest %}
The datawarehouse for me was bigquery and to find the remainder, we use MOD function not %
Hi all I'm currently writing Jinja macro that runs a sql query based on some parameters passed to it.
I'm having some issue with the macro definition itself. I cannot find a way to use the macros paramters within the set block. It interprets the value as literally 'col1' not the value stored in it.
How can I run a SQL query within the macro that uses my parameters stored value?
How I run the macro:
WITH DRIVER
AS(
SELECT
'apple' as col1
)
SELECT
{{ select_test('col1') }} as output
FROM
DRIVER
The macro definition is shown below:
{% macro select_test(val1) %}
{% set query_to_run%}
select
concat({{val1}},'banana')
{% endset %}
{% set results = run_query(query_to_run) %}
{% if execute %}
{# Return the first column #}
{% set results_list = results.columns[0].values() %}
{% else %}
{% set results_list = [] %}
{% endif %}
{{ return(results_list) }}
{% endmacro %}
I currently get the following output:
col1banana
My expected output:
applebanana
note: edited for clarity
You're very close -- I think you're confused by the nested nature of this query.
I suspect when you created this example, you didn't include a set of quotes that is in your original query. This query will return col1banana if col1 is double-quoted (either in the macro or the model), and will return a Database Error otherwise:
select
concat('{{ val1 }}','banana')
-- {{ select_test('col1') }} returns 'col1banana'
select
concat({{ val1 }}, 'banana')
-- {{ select_test("'col1'") }} returns 'col1banana'
select
concat({{ val1 }}, 'banana')
-- {{ select_test('col1') }} raises a Database Error: column "col1" does not exist
The last example is always going to be a database error, since it compiles to this:
select
concat(col1, 'banana')
and there is no from clause in your query_to_run, so col1 can't exist.
Most macros don't use run_query and are just shortcuts for snippets of sql. Those then get templated back into the model query and executed against your database when you build your model.
A macro to concatenate a string onto a column name is as simple as:
{% macro cat_banana(column_name) %}
concat({{ column_name }}, 'banana')
{% endmacro %}
And then you call it from your model:
WITH DRIVER
AS(
SELECT
'apple' as col1
)
SELECT
{{ cat_banana('col1') }} as output
FROM
DRIVER
Then after dbt run -s my_model, if you select * from my_model you'll get applebanana.
I'd like to pass a column divided by a value as a parameter in a jinja macro.
I'm using the macro in a dbt model like this {{ pmt('rate'/1200, 'nper', 'pv', 'fv') }}
However, this gives the error message
"Encountered an error: unsupported operand type(s) for /: 'str' and 'int'"
Most likely you have to treat the whole argument as a string literal (quote the whole thing):
{{ pmt('rate/1200', 'nper', 'pv', 'fv') }}
The reason this works is because it is likely that the macro templates this string into SQL code, e.g.,
{% macro pmt(arg1, arg2, arg3, arg4) %}
...
select {{ arg1 }}
...
{% endmacro %}
In this toy example, {{ arg1 }} will take on the value {{ 'rate/1200' }}, which enters the template (unquoted) as
...
select rate/1200
...
which is valid sql (if you have a field called rate).
It's possible this won't work with all macros, though! In dbt, since the macros are typically templating SQL code, you usually want to pass in arguments that contain field or table references as string literals. However, the argument to the macro could stay inside the jinja context, in which case, you'll need to keep the argument unquoted, or modify a variable before it's passed into the jinja macro. As another toy example:
{% macro print_value(val) %}
{{ log(val, info=True) }}
{% endmacro %}
-- if val is a string literal:
{{ print_value('rate/1200') }}
-- $ rate/1200
-- if val is unquoted:
{% set rate = 2400 %}
{{ print_value(rate/1200) }}
-- $ 2
I'm using Airflow 2.2.2 with the latest providers installed as appropriate.
I'm trying to use the Azure and MySQL hooks and have created custom operators with templates defined for what variables can be templated.
When I do so, I get an error saying that conn or var cannot be found
e.g. my passed parameter is
{{ conn.<variable_name> }}
or
{{ var.json.value.<variable_name> }}
I believe this should be possible in > v2.0 but not working for me, any ideas why?
EDIT: Below are snippets of code with some sensitive information removed, let me know if anything else is needed?
DAG error -
Broken DAG: [/home/dags/dag.py] Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/dags/dag.py", line 52, in <module>
wasb_conn_id = {{ conn.wasb }},
NameError: name 'conn' is not defined
task in dag.py
t1 = WasbLogBlobsToCSVOperator(
task_id='task_xyz',
wasb_conn_id = {{ conn.wasb }},
Custom Operator using an extended version of the Microsoft Azure wasb hook , used by dag.py -
class WasbLogBlobsToCSVOperator(BaseOperator):
template_fields = (
'wasb_conn_id',
)
def __init__(
self,
*,
wasb_conn_id: str = 'wasb',
**kwargs,
) -> None:
super().__init__(**kwargs)
self.wasb_conn_id = wasb_conn_id
self.hook = ExtendedWasbHook(wasb_conn_id=self.wasb_conn_id)
There looks to be a few things going on here that should help.
Jinja templates are string expressions. Try wrapping your wasb_conn_id arg in quotes.
wasb_conn_id = "{{ conn.wasb }}",
Templated fields are not rendered until the task runs meaning the Jinja expression won't be evaluated until an operator's execute() method is called. This is why you are seeing an exception from your comment below. The literal string "{{ conn.wasb }}" is being evaluated as the conn_id. If you want to use a template field in the custom operator, you need to move that logic to be in the scope of the execute() method.
Why do you need to use a Jinja expression here? Since the format of accessing the Connection object via Jinja is {{ conn.<my_conn_id> }}, you could just use the value "wasb" directly.
I am trying to read a JSON file inside a folder. using import_json.
Here is my code
{% set instance_id = grains['INSTANCE_ID'] %}
INSTANCE_ID Env Var:
environ.setenv:
- name: INSTANCE_ID
- value: {{ grains['INSTANCE_ID'] }}
- update_minion: True
{% import_json "/tmp/$INSTANCE_ID/conf.json" as config_properties %}
But I am getting this error
Data failed to compile:
Rendering SLS 'base:cloud.steps.conf' failed: Jinja error: /tmp/$INSTANCE_ID/conf.json.
Although when I insert the INSTANCE_ID manually it works as expected.
What I want is to be able to insert either $INSTANCE_ID or directly the grain value {{ grains['INSTANCE_ID'] }}
Can someone please help me with this?
Thanks.
{% import_json "/tmp/$INSTANCE_ID/conf.json" as config_properties %}
I imagine you are trying to evaluate the variable $INSTANCE_ID in the above statement. Jinja template evaluates the variables in expression statements.
In this case, the variable is set in the first line, using set
{% set instance_id = grains['INSTANCE_ID'] %}
So, you can use it in expression along with string appends, like
{% import_json "/tmp/" ~ instance_id ~ "/conf.json" as config_properties %}
The above statement should resolve your error.
Also, I would suggest using a variable to evaluate the value of the string expression above, like
{% set conf_json_path = "/tmp/" ~ instance_id ~ "/conf.json" %}
and use it like this
{% import_json conf_json_path as config_properties %}
Hope this help!
In case, you wish to use grains dictionary directly, you can use the value like so
{% set conf_json_path = "/tmp/" ~ grains['INSTANCE_ID'] ~ "/conf.json" %}