I have a getdate() field and I want to convert it into 20210211T172650Z this format how do I do it in SSIS expression?
In SSIS, we have data types for strings, numbers and dates. Dates have no format and when it is converted to a string value, you're getting whatever format the localization rules dictate.
If you have a particular format you want, then you need to control that and the only way you can control it, is by using a string data type.
The pattern we're going to use here, for each element,
extract the digit(s)
convert the digits to string
left pad/prepend a leading zero
extract the last 2 characters from our string
When we extract digits, they're numbers and numbers don't have leading zeroes. We convert to string which will allow us to then add the character zero in front of it because we're just concatenating strings. If the number was less than 10, then this prepending of a zero will result in exactly what we want. 9 -> 09 If it was greater than 9, then we have an extraneous value in there. 11 -> 011. We don't care that we went too big because we're then going to take the right 2 most characters making 09 -> 09 and 011 -> 11. This is the shortest logic to making a leading zero string in SSIS.
Using that logic, we're going to create a variable for each element of our formatted string: year, month, day, hour, minute, second.
What's the starting date?
I created a variable called StartDate of type DateTime and hard coded it to a starting point. This is going to allow me to test various conditions. If I used getdate, then I'd either have to adjust my computer's clock to ensure my code works on 2001-01-01 at 01:01:01 as well as 2021-12-31 at 23:59:59. When you're satisfied your code passes all the tests, you can then specify that StartDate property EvaluateAsExpression is True and then use GetDate(). But I wouldn't use GetDate().
GetDate is going to evaluate every time you inspect it. When your package starts, it will show 2021-02-12 # 11:16 AM But your package takes 5 minutes to run, so when you go to re-use the value that is built on GetDate, you will now get 2021-02-12 # 11:21 AM.
In your case, those keys won't match if you send it more than once to your Amazon thing. Instead, use a System scoped variable like #[System::StartTime] That is updated to the time the package starts executing and remains constant for the duration of the SSIS package execution. So when you're satisfied the expression you've build matches the business rules, then change #[User::StartDate] over to use #[System::StartTime]. It provides the updated time but without the challenges of drifting time.
Extract the digit(s)
The SSIS expression language has YEAR, MONTH and DAY defined but no shorthand methods for time components. But, it does have the DATEPART function in which you can ask for any named date part. I'm going to use that for all of my access methods as it makes it nice and consistent.
As an example, this is how I get the Hour. String literal HOUR and we use our variable
DATEPART("HOUR",#[User::StartDate])
Convert the digits to string
The previous step gave us a number but we've got that leading zero problem to solve so convert that to a string
(DT_WSTR, 2)DATEPART("HOUR",#[User::StartDate])
Cast to string, two characters wide max, the number we generated
left pad/prepend a leading zero
String concatenation is the + operator and since we can't concatenate a string to a number, we make sure we have the correct operand types on both sides
"0" + (DT_WSTR, 2)DATEPART("HOUR",#[User::StartDate])
extract the last 2 characters from our string
Since we might have a 2 or 3 character string at this point, we're going to use the RIGHT function to only get the last N characters.
RIGHT("0" + (DT_WSTR, 2)DATEPART("HOUR",#[User::StartDate]), 2)
Final concatenation
Now that we have our happy little variables and we've checked our boundary conditions, the only thing left is to make one last variable, DateAsISO8601 type of string, EvaulateAsExpression = True
#[User::Year] + #[User::Month] +#[User::Day] + "T" +#[User::Hour] +#[User::Minute] +#[User::Second] + "Z"
Should I take precaution when comparing real numbers in MySQL?
# Get completly paid bills
select * from bills left join payments on bills.id = payments.bill
where bills.price = payments.price;
In usual programming languages (like C/C++), such comparisions between real numbers (floats) must be done using a delta value acting as comparision precision:
if (abs(bill_price - payments_bill) < 0.00001)
// do something
But I don't know if MySQL has such precision problems.
MySQL's internal floating point format is 64-bit IEEE.
Yes, you need to worry about machine epsilon (roundoff errors) when doing that kind of arithmetic in MySQL.
The round function sometime doesn't work very well. I have in my db a row like this:
field_1= 375
field_2= 0.65
field_3= 0.1
field_4= 11
So we know that: field_1*field_2*field_3*field_4 = 268.125 so if I round it to 2 decimals -> 268.13.
But in mysql I got 268.12 -> Select round(field_1*field_2*field_3*field_4) from my table -> 268.12
This situation just happens with these values, I tried with other numbers and no problem the round works.
Any workaround about it. I tried in mysql 4.1.22, and 5.1.44 and I get the same issue. I read in other forum http://bugs.mysql.com/bug.php?id=6251 that it is not a bug, they said that it depends on the C library implementation.
What data type are you using for those columns?
If you want exact precision then you should use NUMERIC or DECIMAL, not FLOAT, REAL, or DOUBLE.
From the manual:
The DECIMAL and NUMERIC types store exact numeric data values.
These types are used when it is important to preserve exact precision,
for example with monetary data.
If applicable you can use FLOOR()
If not applicable you will be better off handling this on the application side. I.e. do the entire calculation application side and then round, or just round application side.
SELECT (field_1*field_2*field_3*field_4) AS myCalculation FROM my table
I'm currently storing various metadata about videos and one of those bits of data is the length of a video.
So if a video is 10 minutes 35 seconds long, it's saved as "10:35" in the database.
But what I'd like to do is retrieve a listing of videos by length (longest first, shortest last).
The problem I'm having is that if a video is "2:56", it's coming up as longest because the number 2 is more than the number 1 in.
So, how can I order data based on that length field so that "10:35" is recognized as being longer than "2:56" (as per my example)?
SELECT * FROM table ORDER BY str_to_date(meta_time,'%l:%i')
You can find the specific formatters on the MySQL Website.
For example:
%k -> Hour (0..23)
%l -> Hour (1..12)
The easiest choice is to store a integer (seconds) or a float (minutes) instead of a string. So 10:35 would be 635 in seconds or 10.583 in minutes. You can sort by these numerically very easily. And you can output them in the format you'd like with some simple math and string functions.
Some options:
Save it as an integer representing the total number of seconds. "10:35" => 635
Save it as a timestamp object with no date component. "10:35" => MAKETIME(0, 10, 34)
Save it with leading decimals or spaces. "2:25" => " 2:25"
My preference would be for the first option.
You could try to see if
ORDER BY TIME_TO_SEC(timefield)
would parse it correctly, however it is not an optimal approach to store time as strings in the database, and I suggest that you store them as TIME if you are able to. Then you can use standard formatting functions to present them as you like.
I had the same problem - storing videos length in database.
I solved it by using TIME mysql type - it solves all ordering and converting issues.
EDIT: Now a Major Motion Blog Post at http://messymatters.com/sealedbids
The idea of rot13 is to obscure text, for example to prevent spoilers. It's not meant to be cryptographically secure but to simply make sure that only people who are sure they want to read it will read it.
I'd like to do something similar for numbers, for an application involving sealed bids. Roughly I want to send someone my number and trust them to pick their own number, uninfluenced by mine, but then they should be able to reveal mine (purely client-side) when they're ready. They should not require further input from me or any third party.
(Added: Note the assumption that the recipient is being trusted not to cheat.)
It's not as simple as rot13 because certain numbers, like 1 and 2, will recur often enough that you might remember that, say, 34.2 is really 1.
Here's what I'm looking for specifically:
A function seal() that maps a real number to a real number (or a string). It should not be deterministic -- seal(7) should not map to the same thing every time. But the corresponding function unseal() should be deterministic -- unseal(seal(x)) should equal x for all x. I don't want seal or unseal to call any webservices or even get the system time (because I don't want to assume synchronized clocks). (Added: It's fine to assume that all bids will be less than some maximum, known to everyone, say a million.)
Sanity check:
> seal(7)
482.2382 # some random-seeming number or string.
> seal(7)
71.9217 # a completely different random-seeming number or string.
> unseal(seal(7))
7 # we always recover the original number by unsealing.
You can pack your number as a 4 byte float together with another random float into a double and send that. The client then just has to pick up the first four bytes. In python:
import struct, random
def seal(f):
return struct.unpack("d",struct.pack("ff", f, random.random() ))[0]
def unseal(f):
return struct.unpack("ff",struct.pack("d", f))[0]
>>> unseal( seal( 3))
3.0
>>> seal(3)
4.4533985422978706e-009
>>> seal(3)
9.0767582382536571e-010
Here's a solution inspired by Svante's answer.
M = 9999 # Upper bound on bid.
seal(x) = M * randInt(9,99) + x
unseal(x) = x % M
Sanity check:
> seal(7)
716017
> seal(7)
518497
> unseal(seal(7))
7
This needs tweaking to allow negative bids though:
M = 9999 # Numbers between -M/2 and M/2 can be sealed.
seal(x) = M * randInt(9,99) + x
unseal(x) =
m = x % M;
if m > M/2 return m - M else return m
A nice thing about this solution is how trivial it is for the recipient to decode -- just mod by 9999 (and if that's 5000 or more then it was a negative bid so subtract another 9999). It's also nice that the obscured bid will be at most 6 digits long. (This is plenty security for what I have in mind -- if the bids can possibly exceed $5k then I'd use a more secure method. Though of course the max bid in this method can be set as high as you want.)
Instructions for Lay Folk
Pick a number between 9 and 99 and multiply it by 9999, then add your bid.
This will yield a 5 or 6-digit number that encodes your bid.
To unseal it, divide by 9999, subtract the part to the left of the decimal point, then multiply by 9999.
(This is known to children and mathematicians as "finding the remainder when dividing by 9999" or "mod'ing by 9999", respectively.)
This works for nonnegative bids less than 9999 (if that's not enough, use 99999 or as many digits as you want).
If you want to allow negative bids, then the magic 9999 number needs to be twice the biggest possible bid.
And when decoding, if the result is greater than half of 9999, ie, 5000 or more, then subtract 9999 to get the actual (negative) bid.
Again, note that this is on the honor system: there's nothing technically preventing you from unsealing the other person's number as soon as you see it.
If you're relying on honesty of the user and only dealing with integer bids, a simple XOR operation with a random number should be all you need, an example in C#:
static Random rng = new Random();
static string EncodeBid(int bid)
{
int i = rng.Next();
return String.Format("{0}:{1}", i, bid ^ i);
}
static int DecodeBid(string encodedBid)
{
string[] d = encodedBid.Split(":".ToCharArray());
return Convert.ToInt32(d[0]) ^ Convert.ToInt32(d[1]);
}
Use:
int bid = 500;
string encodedBid = EncodeBid(bid); // encodedBid is something like 54017514:4017054 and will be different each time
int decodedBid = DecodeBid(encodedBid); // decodedBid is 500
Converting the decode process to a client side construct should be simple enough.
Is there a maximum bid? If so, you could do this:
Let max-bid be the maximum bid and a-bid the bid you want to encode. Multiply max-bid by a rather large random number (if you want to use base64 encoding in the last step, max-rand should be (2^24/max-bid)-1, and min-rand perhaps half of that), then add a-bid. Encode this, e.g. through base64.
The recipient then just has to decode and find the remainder modulo max-bid.
What you want to do (a Commitment scheme) is impossible to do client-side-only. The best you could do is encrypt with a shared key.
If the client doesn't need your cooperation to reveal the number, they can just modify the program to reveal the number. You might as well have just sent it and not displayed it.
To do it properly, you could send a secure hash of your bid + a random salt. That commits you to your bid. The other client can commit to their bid in the same way. Then you each share your bid and salt.
[edit] Since you trust the other client:
Sender:
Let M be your message
K = random 4-byte key
C1 = M xor hash(K) //hash optional: hides patterns in M xor K
//(you can repeat or truncate hash(K) as necessary to cover the message)
//(could also xor with output of a PRNG instead)
C2 = K append M //they need to know K to reveal the message
send C2 //(convert bytes to hex representation if needed)
Receiver:
receive C2
K = C2[:4]
C1 = C2[4:]
M = C1 xor hash(K)
Are you aware that you need a larger 'sealed' set of numbers than your original, if you want that to work?
So you need to restrict your real numbers somehow, or store extra info that you don't show.
One simple way is to write a message like:
"my bid is: $14.23: aduigfurjwjnfdjfugfojdjkdskdfdhfddfuiodrnfnghfifyis"
All that junk is randomly-generated, and different every time.
Send the other person the SHA256 hash of the message. Have them send you the hash of their bid. Then, once you both have the hashes, send the full message, and confirm that their bid corresponds to the hash they gave you.
This gives rather stronger guarantees than you need - it's actually not possible from them to work out your bid before you send them your full message. However, there is no unseal() function as you describe.
This simple scheme has various weaknesses that a full zero-knowledge scheme would not have. For example, if they fake you out by sending you a random number instead of a hash, then they can work out your bid without revealing their own. But you didn't ask for bullet-proof. This prevents both accidental and (I think) undetectable cheating, and uses only a commonly-available command line utility, plus a random number generator (dice will do).
If, as you say, you want them to be able to recover your bid without any further input from you, and you are willing to trust them only to do it after posting their bid, then just encrypt using any old symmetric cipher (gpg --symmetric, perhaps) and the key, "rot13". This will prevent accidental cheating, but allow undetectable cheating.
One idea that poped into my mind was to maybe base your algorithm on the mathematics
used for secure key sharing.
If you want to give two persons, Bob and Alice, half a key each so
that only when combining them they will be able to open whatever the key locks, how do you do that? The solution to this comes from mathematics. Say you have two points A (-2,2) and B (2,0) in a x/y coordinate system.
|
A +
|
C
|
---+---+---+---|---+---B---+---+---+---
|
+
|
+
If you draw a straight line between them it will cross the y axis at exactly one single point, C (0,1).
If you only know one of the points A or B it is impossible to tell where it will cross.
Thus you can let the points A and B be the shared keys which when combined will reveal the y-value
of the crossing point (i.e. 1 in this example) and this value is then typically used as
a real key for something.
For your bidding application you could let seal() and unseal() swap the y-value between the C and B points
(deterministic) but have the A point vary from time to time.
This way seal(y-value of point B) will give completely different results depending on point A,
but unseal(seal(y-value of point B)) should return the y-value of B which is what you ask for.
PS
It is not required to have A and B on different sides of the y-axis, but is much simpler conceptually to think of it this way (and I recommend implementing it that way as well).
With this straight line you can then share keys between several persons so that only two of
them are needed to unlock whatever. It is possible to use curve types other then straight lines to create other
key sharing properties (i.e. 3 out of 3 keys are required etc).
Pseudo code:
encode:
value = 2000
key = random(0..255); // our key is only 2 bytes
// 'sealing it'
value = value XOR 2000;
// add key
sealed = (value << 16) | key
decode:
key = sealed & 0xFF
unsealed = key XOR (sealed >> 16)
Would that work?
Since it seems that you are assuming that the other person doesn't want to know your bid until after they've placed their own, and can be trusted not to cheat, you could try a variable rotation scheme:
from random import randint
def seal(input):
r = randint(0, 50)
obfuscate = [str(r)] + [ str(ord(c) + r) for c in '%s' % input ]
return ':'.join(obfuscate)
def unseal(input):
tmp = input.split(':')
r = int(tmp.pop(0))
deobfuscate = [ chr(int(c) - r) for c in tmp ]
return ''.join(deobfuscate)
# I suppose you would put your bid in here, for 100 dollars
tmp = seal('$100.00') # --> '1:37:50:49:49:47:49:49' (output varies)
print unseal(tmp) # --> '$100.00'
At some point (I think we may have already passed it) this becomes silly, and because it is so easy, you should just use simple encryption, where the message recipient always knows the key - the person's username, perhaps.
If the bids are fairly large numbers, how about a bitwise XOR with some predetermined random-ish number? XORing again will then retrieve the original value.
You can change the number as often as you like, as long as both client and server know it.
You could set a different base (like 16, 17, 18, etc.) and keep track of which base you've "sealed" the bid with...
Of course, this presumes large numbers (> the base you're using, at least). If they were decimal, you could drop the point (for example, 27.04 becomes 2704, which you then translate to base 29...)
You'd probably want to use base 17 to 36 (only because some people might recognize hex and be able to translate it in their head...)
This way, you would have numbers like G4 or Z3 or KW (depending on the numbers you're sealing)...
Here's a cheap way to piggyback off rot13:
Assume we have a function gibberish() that generates something like "fdjk alqef lwwqisvz" and a function words(x) that converts a number x to words, eg, words(42) returns "forty two" (no hyphens).
Then define
seal(x) = rot13(gibberish() + words(x) + gibberish())
and
unseal(x) = rot13(x)
Of course the output of unseal is not an actual number and is only useful to a human, but that might be ok.
You could make it a little more sophisticated with words-to-number function that would also just throw away all the gibberish words (defined as anything that's not one of the number words -- there are less than a hundred of those, I think).
Sanity check:
> seal(7)
fhrlls hqufw huqfha frira afsb ht ahuqw ajaijzji
> seal(7)
qbua adfshua hqgya ubiwi ahp wqwia qhu frira wge
> unseal(seal(7))
sueyyf udhsj seven ahkua snsfo ug nuhdj nwnvwmwv
I know this is silly but it's a way to do it "by hand" if all you have is rot13 available.