Incremental query in DBT based on current month with Jinja - jinja2

I am trying to implement a incremental query in DBT using Jinja.
Considering there are tables getting created every month in warehouse with year and month suffix and I need to write a logic to union the new table which gets created every month to execute the DBT model.
Below is the code which I have started with
#initialize the months in a list
{% set months= ['03','04','05','06','07','08','09','10','11','12','01','02'] %}
#first select query for Feb month of 2022
SELECT *, '2022-02-01' AS ref_month
FROM source_table_2022_02
#initilalize year variable to 2022
{% set year= namespace(items=2022) %}
#loop through the months to generate dynamic query for upcoming months
{% for month in months %}
#if month is Jan increment the year
{% if month == '01' %}
{% set year.items = year.items + 1 %}
{% endif %}
UNION ALL
SELECT *, '{{ year.items }}-{{ month }}-01' AS ref_month
FROM source_table_{{ year.items }}_{{ month }}
{% endfor %}
output of above logic is as below
SELECT *, '2022-02-01' AS ref_month
FROM source_table_2022_02
UNION ALL
SELECT *, '2022-03-01' AS ref_month
FROM source_table_2022_03
UNION ALL
SELECT *, '2022-04-01' AS ref_month
FROM source_table_2022_04
.
.
.
UNION ALL
SELECT *, '2023-02-01' AS ref_month
FROM source_table_2023_02
I need help in stopping the for loop when we reach the current month i.e Dec(because there is no current_month method in Jinja and I need to implement this logic in DBT models.sql file and not a python file), instead of looping through the upcoming months.
Note: as mentioned earlier the source table gets created every month with year and month suffix
I also want to continue the loop after 2023 Feb in the upcoming months. Current logic stops immediately after the list iteration ends i.e 2023 Feb

Would you consider installing the commonly used dbt-utils package?
If so, they have a macro called get_relations_by_pattern and another called union_relations.
These could be used to solve your problem as follows:
{% set monthly_relations = dbt_utils.get_relations_by_pattern('my_schema', 'source_table_%') %}
SELECT *
FROM {{ dbt_utils.union_relations(relations = monthly_relations) }}
Note that a new field _dbt_source_relation will be added, listing the original table name. You'll be able to parse the month and year from this.

Related

Retrieve ALL row data from TWO different months in SAME table?

Re-do Times 2 for clarity
For simplicity and clarity, the data I am aggregating is in a database I will refer to as "BaseA".
Normally, when comparing Month over month data, I can use the following query:
select date_trunc('month',hour) as date,
sum(a) as total_a,
sum(b) as available_b,
sum(c) as c,
sum(d) as net_d
from BaseA where id=12345 and hour >='2022-01-01'
group by date order by date desc
Instead of looking back and collecting ALL months from 2021-2022 for the duration I wish to view, I want to collect ONLY two months of data, those being the following:
October 2021
vs.
April 2022
I'd like the data to be visualized in the Month over month format, like so:
example
However, I would like to:
Select all BaseA columns (aka, select *)
Only include Two rows: April 2022 & October 2021
So, should come out like so:
example 2
This query is what Im trying to do (in word form since I can't write it)
Select *
from BaseA
where date
is in
April 2022
&
October 2022
The result of the above should result in 2 rows of data (one for each month referenced)
Is there a place in the below query where DISTINCT would make that actualized?
select * from BaseA
where id=12345 and
(
(month(month) = '04' and year(month) = '2022')
OR
(month(month) = '10' and year(month) = '2021')
)
--DISTINCT go where?
Appreciate the help!
To get data for the two months try something like this:
select * from BaseA
where _id=12345 and
(
(month(month) = '04' and year(month) = '2022')
OR
(month(month) = '10' and year(month) = '2021')
)
I have no idea what you mean by "return data consolidated (grouped) by month (so multiple lines occur per month)". Can you provide dummy data that illustrates the data that you have and the result that you want to achieve?

Angular date pipe subtracts a day from the actual date value

I am not sure If I need to add anything here. I have a table with a date column. When the date column gets processed through the Angular date pipe, just to change the format to display, it subtracts a day from the actual date. I checked it without adding the date pipe, I get the actual date. What could be wrong?
<div *ngSwitchCase="'pending'" style="color:#367db9">
{{ workOrder.postDate | date: 'MMM d, y'}}
</div>
Have you tried it without the custom format?
{{ workOrder.postDate | date: 'MMM d, y'}}
should be the same result as
{{ workOrder.postDate | date }}
and the same as
{{workOrder.postDate | date:'mediumDate'}}
example: https://jsfiddle.net/Scotty3/bdmqxr5y/4757/
and https://next.angular.io/api/common/DatePipe
If not you'll need to check your source date variable.

Get last "active" day from last week

In my tool users can set their workdays (example: monday to friyday).When they save their settings, it will stored in the mysql user table.
In the next week (this week) i will get their "last workday". In this case it will be the friday. How can i get the last workday from every user with mysql?
Currently i save the workdays in csv (2,3,4,5,6 - 2 = Monday...) but i can change that.
I tried some stuff with WEEKDAY(), but it doesnt work.
Can anybody help me?
DAYNAME(CONCAT('1970-09-2', SUBSTRING_INDEX(workdays, ',', -1)))
Explanation:
This MySQL string function SUBSTRING_INDEX:
SUBSTRING_INDEX(workdays, ',', -1)
...will give you the number of the last workday listed. (What it does is find all the positions where there's a comma and returns everything after the last comma, e.g., SUBSTRING_INDEX('1,2,3', ',', -1) returns 3.)
And this trick from here, using DAYNAME and CONCAT:
DAYNAME(CONCAT('1970-09-2', dayIndex))
...gives you the day name. (Since DAYNAME requires a date, this technique picks a date in the past that is a Sunday and ends in 0, in this case 1970-09-20, and replaces the last digit with the returned index to determine the corresponding day of week, e.g., 20 = Sunday, 21 = Monday, etc.)
Put them together:
SELECT DAYNAME(CONCAT('1970-09-2', SUBSTRING_INDEX(workdays, ',', -1))) AS lastday FROM ...
...and you get the corresponding name of the weekday returned in the lastday variable.
Below sql query might give some help to you. Hope you are asking for this only. It will list the last working day (friday) for each user.
SELECT * FROM tablename
WHERE user_id, workday_id IN
(SELECT user_id, MAX(workday_id) FROM tablename GROUP BY user_id );

Foxpro complex sub query

I'm having a hard time doing a query in foxpro 2.6 (no other choice)
note from some reason union queries do not work.
stockw (item, qte, dat) item is key
shist (item, qte, date) each row is a purchase for item and qte and date
the below query will calculate the first purchase for an item in shist to know how many months we've been selling every item since the beginning of the year
select shist.item as s_item, MIN(shist.date),stockw.qte as qte_remain,;
IIF( YEAR(MIN(shist.date)) < 2013, MONTH(DATE()), MONTH(DATE())-MONTH(shist.date)+1 ) as months;
FROM shist,stockw;
WHERE (shist.item LIKE 'LF-TK%' OR shist.item LIKE 'PL%' OR shist.item LIKE 'LF-A%') AND stockw.item = shist.item;
GROUP BY shist.item;
into CURSOR x
i want to:
- calculate the average qte from the beginning of the year 2013 till now for each item SUM(shist.qte)/months (months is calculated from the above query for each item)
maybe in a different query i want the below:
- select the max of sum of shist.qte for every month for each item (get highest month) from the beginning of the year 2013 till now.
Since I'm not quite following where you would be applying a union, and appears you have multiple times you are running the query, here is a VERY OLD trick that was usable in the old days of VFP...
When you do a select into cursor, the cursor is typically read-only and not read-write to append to... BUT, if you use the result of the first time run query AGAIN as a different alias, you CAN append to it. Something along the lines of...
use in select( "C_ThisIsMaster" )
for i = 1 to 10
use in select( "C_ThisRun" )
select blah, blah2, blah3 ;
from YourTables;
where SomeKey = SomeIDForCyclei;
into cursor C_ThisRun
if not used( "C_ThisIsMaster" )
select 0
use ( dbf( "C_ThisRun") ) again alias C_ThisIsMaster
else
select C_ThisIsMaster
append from dbf( "C_ThisRun" )
endif
endfor
The for[i] loop is just a sample, but if you are running for many cycles that you would do the UNION for, you can just pull that into a query and simulate the same, such as via
select *
from SomeTable
where SomeCondition
into cursor C_DoTheseItems
use in select( "C_ThisIsMaster" )
select C_DoTheseItems
do while not eof()
Do the same thing as inside the FOR loop above
select C_DoTheseItems
skip
enddo
I know VFP has scan/endscan, but don't remember how far back that originated, so using the do while NOT eof() was used here...
At the end of either option would give you a simulated UNION of each item you were trying to process into one "alias" result "C_ThisIsMaster" to work with

Group records by both month and year in Rails

I'm on Ruby 1.9.2, Rails 3.0.x and I use MySQL DB. I have a Messages Model, where one can post new messages every day.
I have an index action where I want to display these messages grouped by both month and year. This basically is used to filter messages.
My query looks like this:-
#active_msgs = Messages.where('expiry > ?', Time.now).order('created_at DESC')
I used the group_by function in Rails, after referring to this post
My Group By Query is:-
#msgs_by_month = #active_msgs.all.group_by {|u| u.created_at.month }
I was returned a Hash called #msgs_by_month. This helped me group the messages by month, but I need to taken into account the year, to form a unique key , as it would help me differentiate between for e.g., Sept 2011 and Sept 2012.
My current key only is of type Hash[Month] ( e.g. Hash[9] => for month of September, I can display all appropriate values ). How can i get a unique key of type month, year which I can easily loop though to display all records grouped by Month and Year of creation.
Thank you
EDIT:-
I'm guessing one way to do this, is to make use the created_at.to_date api, provided here
. I only wonder, how I can strip out the day from created_at.to_date to get a unique month and year combination key which could work with the group_by function.
In SQL:
select * from messages group by year(created_at), month(created_at);
In Rails:
Message.all.group_by { |m| m.created_at.beginning_of_month }
Use Group Date.
It supports time zones, so if you ever need to - the code will easily evolve.
https://github.com/ankane/groupdate
Message.select("date_trunc('month', created_at) as month").group("month")
To group by year and by month (e.g. not having december 2020 with december 2021):
Message.group("date_trunc('year', created_at), date_trunc('month', created_at)")
Tested on PostgreSQL
ModelName.all.group_by { |m| m.created_at.month }
This will work, however month will be returned as the index.
Ex. Nov would refer to 11