i followed the document https://docs.getdbt.com/guides/best-practices/writing-custom-generic-tests and was trying to write the generic test "is_even" from here but when i am running the below generic test called is_even using dbt test command,it is giving me the error - "Database Error in test is_even_my_first_dbt_model_id (models/example/schema.yml)
Syntax error: Unexpected "%" at [25:23]"
test_is_even.sql -
{% test is_even(model, column_name) %}
with validation as (
select
{{ column_name }} as even_field
from {{ model }}
),
validation_errors as (
select
even_field
from validation
-- if this is true, then even_field is actually odd!
where (even_field % 2) = 1
)
select *
from validation_errors
{% endtest %}
The datawarehouse for me was bigquery and to find the remainder, we use MOD function not %
Related
Hi all I'm currently writing Jinja macro that runs a sql query based on some parameters passed to it.
I'm having some issue with the macro definition itself. I cannot find a way to use the macros paramters within the set block. It interprets the value as literally 'col1' not the value stored in it.
How can I run a SQL query within the macro that uses my parameters stored value?
How I run the macro:
WITH DRIVER
AS(
SELECT
'apple' as col1
)
SELECT
{{ select_test('col1') }} as output
FROM
DRIVER
The macro definition is shown below:
{% macro select_test(val1) %}
{% set query_to_run%}
select
concat({{val1}},'banana')
{% endset %}
{% set results = run_query(query_to_run) %}
{% if execute %}
{# Return the first column #}
{% set results_list = results.columns[0].values() %}
{% else %}
{% set results_list = [] %}
{% endif %}
{{ return(results_list) }}
{% endmacro %}
I currently get the following output:
col1banana
My expected output:
applebanana
note: edited for clarity
You're very close -- I think you're confused by the nested nature of this query.
I suspect when you created this example, you didn't include a set of quotes that is in your original query. This query will return col1banana if col1 is double-quoted (either in the macro or the model), and will return a Database Error otherwise:
select
concat('{{ val1 }}','banana')
-- {{ select_test('col1') }} returns 'col1banana'
select
concat({{ val1 }}, 'banana')
-- {{ select_test("'col1'") }} returns 'col1banana'
select
concat({{ val1 }}, 'banana')
-- {{ select_test('col1') }} raises a Database Error: column "col1" does not exist
The last example is always going to be a database error, since it compiles to this:
select
concat(col1, 'banana')
and there is no from clause in your query_to_run, so col1 can't exist.
Most macros don't use run_query and are just shortcuts for snippets of sql. Those then get templated back into the model query and executed against your database when you build your model.
A macro to concatenate a string onto a column name is as simple as:
{% macro cat_banana(column_name) %}
concat({{ column_name }}, 'banana')
{% endmacro %}
And then you call it from your model:
WITH DRIVER
AS(
SELECT
'apple' as col1
)
SELECT
{{ cat_banana('col1') }} as output
FROM
DRIVER
Then after dbt run -s my_model, if you select * from my_model you'll get applebanana.
I'm trying to use an xcom_pull inside an SQL phrase executed by a Snowflake operator in Airflow.
I need the task_id name to use a variable since I want to support different tasks.
I tried this syntax but seems it is not being rendered ok.
Anyone has an idea how to do it?
This is the Python code:
for product, val in PRODUCTS_TO_EXTRACT_INC.items():
product_indicator, prefix = val
params['product_prefix'] = prefix
calculate_to_date = SnowflakeOperator(
dag=dag,
task_id=f'calculate_to_date_{prefix}',
snowflake_conn_id = SF_CONNECTION_ID,
warehouse=SF_WAREHOUSE,
database=BI_DB,
schema=STG_SCHEMA,
role=SF_ROLE,
sql= [ """
{SQL_FILE}
""".format(SQL_FILE="{% include '" + QUERIES_DIR + ETL + "/calculate_to_date.sql'" + " %}")
],
params=params
)
This is the SQL code for calculate_to_date.sql:
select '{{{{ (ti.xcom_pull(key="return_value", task_ids=["calculate_from_date_{}"])[0][0]).get("FROM_DATE") }}}}'.format(params.product_prefix) AS TO_DATE
This is the error message:
File "/home/airflow/gcs/dags/Test/queries/fact_subscriptions_events/calculate_to_date.sql", line 11, in template
select '{{{{ (ti.xcom_pull(key="return_value", task_ids=["calculate_from_date_{}"])[0][0]).get("FROM_DATE") }}}}'.format(params.product_prefix)
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
the correct syntax is
select '{{ (ti.xcom_pull(key="return_value", task_ids="calculate_from_date_{}".format(params.product_prefix))[0]).get("FROM_DATE") }}' AS TO_DATE
it works like a charm
I'd like to pass a column divided by a value as a parameter in a jinja macro.
I'm using the macro in a dbt model like this {{ pmt('rate'/1200, 'nper', 'pv', 'fv') }}
However, this gives the error message
"Encountered an error: unsupported operand type(s) for /: 'str' and 'int'"
Most likely you have to treat the whole argument as a string literal (quote the whole thing):
{{ pmt('rate/1200', 'nper', 'pv', 'fv') }}
The reason this works is because it is likely that the macro templates this string into SQL code, e.g.,
{% macro pmt(arg1, arg2, arg3, arg4) %}
...
select {{ arg1 }}
...
{% endmacro %}
In this toy example, {{ arg1 }} will take on the value {{ 'rate/1200' }}, which enters the template (unquoted) as
...
select rate/1200
...
which is valid sql (if you have a field called rate).
It's possible this won't work with all macros, though! In dbt, since the macros are typically templating SQL code, you usually want to pass in arguments that contain field or table references as string literals. However, the argument to the macro could stay inside the jinja context, in which case, you'll need to keep the argument unquoted, or modify a variable before it's passed into the jinja macro. As another toy example:
{% macro print_value(val) %}
{{ log(val, info=True) }}
{% endmacro %}
-- if val is a string literal:
{{ print_value('rate/1200') }}
-- $ rate/1200
-- if val is unquoted:
{% set rate = 2400 %}
{{ print_value(rate/1200) }}
-- $ 2
I'm trying to invoke a macro as a post-hook. The trouble is (I think) is that I'd like to pass a list to this macro... any idea what's going on here? My theory is that I'm passing a list type argument.
-- models/table.sql
{{
config(
materialized = 'table',
post-hook = "{{ my_macro(this,'my_str', ['foo', 'bar']) }}"
)
}}
SELECT * FROM muh_tayble;
-- macros/my_macro.sql
{% macro my_macro(relation, string, list) %}
BLAH
{% endmacro %}
error message
Encountered an error:
Compilation Error in model table (models/table.sql)
invalid syntax for function call expression
line 2
Rookie mistake folks. I had post-hook instead of post_hook. Problem solved
I am trying to read a JSON file inside a folder. using import_json.
Here is my code
{% set instance_id = grains['INSTANCE_ID'] %}
INSTANCE_ID Env Var:
environ.setenv:
- name: INSTANCE_ID
- value: {{ grains['INSTANCE_ID'] }}
- update_minion: True
{% import_json "/tmp/$INSTANCE_ID/conf.json" as config_properties %}
But I am getting this error
Data failed to compile:
Rendering SLS 'base:cloud.steps.conf' failed: Jinja error: /tmp/$INSTANCE_ID/conf.json.
Although when I insert the INSTANCE_ID manually it works as expected.
What I want is to be able to insert either $INSTANCE_ID or directly the grain value {{ grains['INSTANCE_ID'] }}
Can someone please help me with this?
Thanks.
{% import_json "/tmp/$INSTANCE_ID/conf.json" as config_properties %}
I imagine you are trying to evaluate the variable $INSTANCE_ID in the above statement. Jinja template evaluates the variables in expression statements.
In this case, the variable is set in the first line, using set
{% set instance_id = grains['INSTANCE_ID'] %}
So, you can use it in expression along with string appends, like
{% import_json "/tmp/" ~ instance_id ~ "/conf.json" as config_properties %}
The above statement should resolve your error.
Also, I would suggest using a variable to evaluate the value of the string expression above, like
{% set conf_json_path = "/tmp/" ~ instance_id ~ "/conf.json" %}
and use it like this
{% import_json conf_json_path as config_properties %}
Hope this help!
In case, you wish to use grains dictionary directly, you can use the value like so
{% set conf_json_path = "/tmp/" ~ grains['INSTANCE_ID'] ~ "/conf.json" %}