I have an SSIS package that runs each morning to pull the previous days file from an FTP server. I am using the code below to create the file name using the previous date. Everything works great with this except when today's date is the first day of the month. for example, if ran today (3/1/2021) this returns name_of_file_20210328.xml.gz, however yesterday's date is 2/28/2021 not 3. How do i update this to say if today's date is beginning of month return mm - 1?
"name_of_file_" + (DT_STR,4,1252)(DATEPART("yyyy",GETDATE())) + (LEN((DT_STR,2,1252)(DATEPART("MM",GETDATE()))) == 2 ? (DT_STR,2,1252)(DATEPART("MM",GETDATE())) : "0" + (DT_STR,2,1252)(DATEPART("MM",GETDATE()))) + (LEN((DT_STR,2,1252)(DATEPART("dd",DATEADD( "day",-1, GETDATE())))) == 2 ? (DT_STR,2,1252)(DATEPART("dd",DATEADD( "day",-1, GETDATE()))) : "0" + (DT_STR,2,1252)(DATEPART("dd",DATEADD( "day",-1, GETDATE())))) +
".xml.gz"
Create a variable, Yesterday of type DateTime. Specify that it uses an expression and use the following expression. This provides a consistent reference point you can test against and if you disable the expression, allows you to specify a date for boundary/special case checking (like a leap year 2020-03-01)
DATEADD("DAY", -1, #[System::StartTime])
The next steps, especially if you're starting out, is to build the date parts in separate variables. It doesn't cost any extra to use lots of variables in your package and makes troubleshooting so much easier.
Add a new variable, YearString of type String.
(DT_WSTR, 4)datepart("YYYY", #[User::Yesterday])
That was easy.
Now we need to deal with create a zero, left padded string. Right now, your expression looks like it's trying to determine if day or month has 2 digits. I have a cleaner expression.
We're going to convert the day/month to a string and then prepend a zero to it. For Jan-Sep, we'll end up with a 2 character expression, Oct-Dec, we'll have a three character expression e.g. 011. For Day, similar bit where it's 01-09 or 010-031. We will then take the last two characters from the string. For the two character strings, it's a no-operation and for the three character variant, it gets us what we want.
Add Variable MonthString, as type string, to your package
RIGHT("0" + (DT_WSTR, 2)datepart("MONTH", #[User::Yesterday]), 2)
Add Variable DayString, as type string, to your package
RIGHT("0" + (DT_WSTR, 2)datepart("DAY", #[User::Yesterday]), 2)
Take a moment and look at your Variable collection. You can see that you have all the building blocks needed to properly construct your YYYYMMDD string. If something is wrong, it's small enough snippet to play with it. If not, break it up into smaller Variables.
Now that we have defined #Yesterday and then built YearString, MonthString and DayString off of Yesterday, all we have to do is bring it all together with the concatenation + operator
Back to the Variable well, creating #CurrentFileName of type string
"name_of_file_" + #[User::YearString] + #[User::MonthString] + #[User::DayString] + ".xml.gz"
Results in a value of name_of_file_20210216.xml.gz
Not addressed in this answer but things you should think about
What happens when the job doesn't run at all today (server fails horrifically, the data source is down, etc)? To pick up and process 2 days ago file, you would need to edit this package, run it through whatever change review process is applicable, deploy to prod, run it and then go back to the process yesterday file package.
That's not fun to type much less consider. You certainly aren't going to change the server time to trick the expression into being yesterday.
I am an advocate for passing the run date to the package (mechanism depends on how you deploy/run packages). In that situation, it's been my experience that it's a far easier bureaucracy fight to change the calling parameter (because no "code" has changed) than to get an emergency code change run through.
Related
I have build an ETL package with two sequence containers.
One for a full load and one for a incremental load.
I want the package to excecute the incremental load each "normal" day
When it is a sunday to execute the full load
Do i need to use a script task for this?
There are a number of ways to accomplish this. One fairly straight forward way would be to add an Execute SQL Task ahead of your two Sequence Containers with some code to determine the day of the week.
This is pretty clear in it's intention:
SELECT
CASE
WHEN DATENAME(WEEKDAY, GETDATE()) = 'Sunday' THEN 1
ELSE 0
END;
Set the result set to Single Row, and on the Result Set tab, assign the query output to a variable.
Create a Precedence Constraint from the task to the incremental container. Edit the constraint to be an Expression and Constraint and specify that #[User::MyVariable]==0.
Create another constraint to the full refresh container, but specify the variable value as 1.
There are a few different ways to solve this.
First approach
The first way is through an Expression on each of your Sequence Containers. The Expressions will control the Disable state of the container and the expression you are looking for is DATEPART for a Weekday dw
Is it Sunday?
DATEPART("weekday", getdate()) == 1
Is it not Sunday?
DATEPART("weekday", getdate()) != 1
Next approach
The problem with using the getdate approach is testing - how are you going to test all the possible scenarios? Are you going to change the clock on the computer to verify the Sunday/nonSunday branch works?
I would create an SSIS Variable called CurrentDate of type DateTime and initialize it to today.
I would create a second SSIS Variable called IsSunday of type Boolean and set an expression on this variable to be DATEPART("weekday", #[User::CurrentDate] ) == 1
I would then modify my two Sequence Containers to have a new, simplified expression on their Disabled state
Is it Sunday
#[User::IsSunday]
Is it not Sunday
!#[User::IsSunday]
Run the package and verify the non-Sunday branch lights up (today is Thursday). Stop execution and then change the value of #[User::CurrentDate] to last Sunday (2019-11-17) and restart the package. Verify the Sunday branch lights up and then stop execution.
Now that you have the logic sound, the remaining step is to modify the SSIS Variable CurrentDate to have an expression on it of getdate(). Restart the package and verify the non-Sunday branch has once again lit up for execution.
Next, Next approach
Congratulations, your package is implemented and it runs well. Until it doesn't. The server maintenance window runs long and this package doesn't actually start until Monday morning at 12:15 a.m. How are the operators going to be able to tell the package "Run the Sunday logic?" You aren't, unless you edit the package and nobody wants to do that.
In this situation, I could use the basic construct I've built above but now, I would add a Parameter to the package, of type Boolean and call it something like ForceSunday and the default value is False. Only in situations where I need to force the Sunday logic would that be flipped to True (and only for the one run-time, not a design-time change).
The logic for our IsSunday gets a touch more complicated as we need to OR || the logic for the parameter name.
DATEPART("weekday", getdate()) == 1 || #[$MyPackage::ForceSunday]
I think I have the parameter name syntax correct but the dollar sign could be elsewhere (this is all from memory)
Next, Next, Next approach
This is a really complex package. Make life easier for everyone - Copy & paste into a package that has Sunday logic and one that has Non-Sunday logic.
Final notes
Regardless of the approach you take, the next problem you will run into is the precedence constraint between both of these containers feeding into the container on the right of the screen. The solid green line is a precedence constraint of Success and Logical AND - so the final box will only ever start if both the Sunday and non-Sunday workflows execute. Logically, that can't happen so you need to double click the line and change it to a Logical OR
2 Data Flow Tasks Linking to one Execute SQL Task
first off all, thank you for your answer.
I might have found an other way which would be a lot easier but I would like to check it with you.
In the Server Manager I can create jobs with build in schedule which can be executed on weekdays and weekend days. So I made two jobs, weekend with value false and week with value true.
Would it work if I make the following parameters on the sequence containers:
In SSDT I have set a parameter for the full load sequence: ForceExecutionValue = false
In SSDT I have set a parameter for the incremental load sequence: ForceExecutionValue = true
Will this work or will it still execute both sequences?
I want to make a simple timer for processes that I run with a Macro. I'm using SetTempVar to record the start and end times, and a simple query to calculate the elapsed time.
The macro is:
Then the query is simply:
SELECT [tempvars]![ProcessStart] AS Start, [TempVars]![ProcessEnd] AS [End], DateDiff("s",[start],[end]) AS Seconds;
But the output is strange:
The 2 fields from SetTempVar display in some strange font. However, the elapsed time of 84 seconds is correct.
How can I display the start and end times correctly?
I can reproduce this. It's an odd issue.
The TempVar gets interpreted as a string, though it contains a date. The binary date data gets interpreted as UTF-16 characters, displaying random characters (often Chinese since there are many Chinese characters in UTF-16).
I'd consider this a bug in Access. Queries should correctly determine variable type, and that's apparently somehow going wrong.
To display the date value, use either Format or CDate.
If you're interested in a time difference, I recommend formatting it as Long Time:
SELECT Format([tempvars]![ProcessStart], "Long Time") AS Start, Format([TempVars]![ProcessEnd], "Long Time") AS [End], DateDiff("s",[start],[end]) AS Seconds;
This is specifically for MS-Access Web Databases (requires Sharepoint hosting) which has many limitations compared to their client counterparts, like no VBA, instead you get form macros and data macros to manage data.
I've run into a weird bug on one of my applications. I have a query used to check stock levels against a "minimum stock level" also saved in the table. The query is pretty intense and there are over 4,000 records now to check against. These querys normally take about 75s. So I have made a little label that gets updated every time the form is loaded showing the time and date the query was last run, and the duration in seconds it took. (so users can see how fresh the data is and decide if it needs to be run again)
Now, the weird thing is it works fine in my Access client, but when I sync my changes to the server and try it in a web browser I get a "type mismatch" error. A small table is used to store the start and end times whenever the query is run, that's how I get the timestamp data. These fields are in a "Date/Time" format, obviously. But it seems the problem here is changing the date format to a string format so it can be put in a label on the form. The Access client seems perfectly capable of doing this, while the web client stumbles and falls.
My problem is, how do I change data in a date/time format to a string format in a Web database? I can't figure out how to do this. The tools are so limited. I may have to end up answering my own question here but I'm posting this for others just in case.
To return a value from a data macro as string, you have to format the internal date/time format as a string. In Access an internal date/time value is a double number with the integer part as number of days since 1900, and the “decimal” time part is a fraction of 24 hours. Unfortunately if you simply wrap the date/time in the str$() function we had for 20+ years, then you get something JUST like if you type this into the debug window:
? cdbl(now())
41955.5478587963
The solution is to simply pull out each part. And “nice” is while in few cases a data macro will cast the data type, it does in this case and thus the STR$() command is not required.
The expression you thus can use is this:
Month([d]) & "/" & Day([d]) & " Time = " & Hour([d]) & ":" & Minute([d])
So say to pluck out the VERY LAST start time column from say a invoice table, since we don’t have a dmax(), then we simply sort the table in the order we want and pull out the first row.
Our data macro will thus look like:
Note how in above I simply typed in the SQL and SET the order on the date/time column. I want the MOST recent invoice start date and time. For those new to SQL, then I suggest you build a query in the query builder and specify a query in above lookup feature, since many are not "comfortable" typing in free hand SQL as I did above.
Now, in your browser side (UI) macro, you can use this code:
The above returns a formatted string that you can stuff into a text box, or as per above code change the caption of a label.
Unfortunately with silly problems like this, it becomes a path-of-least resistance thing.
Since my intended result was simply to get "a timedatestamp from a table to show up on a form (so users could see when a query was last run)", this became redesigning my form in Access to be a text field instead of a label. Text fields can be adjusted to accept "Time/Date" formats, so this is exactly what I did, it now pulls the timestamp data directly from the last record of the table and requires no extra formatting to appear in the web browser. I redesigned the text field to appear and function more like a label, and my desired function was achieved.
However, since my question specifically asks, "how do you change a time/date format into a string format in a Web db?", I will leave it here in case someone actually does solve it.
Evening,
I have created a query that is suppose to change ONLY the year part of a Date/Time field to 1900 when a person is 89 years or older. The query that follows compiles fine but when run it complains about a Type Conversion failure and removes the entire value from the records affected.
The query:
UPDATE tblTestForDOB
SET tblTestForDOB.[PT_BirthDate] = DateValue( (day([PT_BirthDate])/month([PT_BirthDate])/1900) )
WHERE Year(tblTestForDOB.[PT_BirthDate]) <= Year(Date())-"89";
According to the MS Help (F1 over the function):
The required date argument is normally a string expression representing a date from January 1, 100 through December 31, 9999. However, date can also be any expression that can represent a date, a time, or both a date and time, in that range.
Is that not what I'm doing? I also tried placing the " " & before the values inside the DateValue function and that did the same thing
(to ensure that it was a string that was passed)
So how do I go about it? Should I use CDate to convert the value to a Date and then proceed that way? If so what is the correct syntax for this?
Thanks
P.S The field is of Short Date format. Also note that I don't want to take the long way around and use VBA for the whole thing as that would involve opening record sets and so on...
It appears you're trying to give DateValue a string, but that's not what's happening. There may be more going on that I don't understand, so I'll just show you an Immediate window session which may contain something you can build on.
PT_BirthDate = #1923-6-1#
? PT_BirthDate
6/1/1923
? DateDiff("yyyy", PT_BirthDate, Date())
90
' this throws error #13: Type mismatch ...
? DateValue( (day([PT_BirthDate])/month([PT_BirthDate])/1900) )
' it will work when you give DateValue a string ...
? DateValue("1900-" & Month(PT_BirthDate) & "-" & Day(PT_BirthDate))
6/1/1900
' or consider DateSerial instead ...
? DateSerial(1900, Month(PT_BirthDate), Day(PT_BirthDate))
6/1/1900
Hi I am trying to use the expression column in flat file connection manager to generate file with dynamic naming e.g
File-ddmmyyyyhhmmss.txt
I understand that I can use getdatepart but what is the exact method call to get the hh, mm and ss? Can anyone advise?
Pragmatic works SSIS Expression Cheat Sheet is a good start. You can expand to your situation by using (DT_WSTR, 2) datepart("HH",getdate()) to get the hour portion, (DT_WSTR, 2) datepart("n",getdate()) to get the minutes and (DT_WSTR, 2) datepart("s",getdate()) to get the seconds. You will need to add logic in to ensure that values less than 10 still get two digits (i.e. 01 05) etc. For Day, month and Year, there are function DAY() MONTH() and YEAR() that shortcut the equivalent datepart() functions.