I'm loading a bunch of semi-structured data (JSON) into my database through Snowflake. The timestamp values in the entries are javascript timestamps that look like this:
"time": 1621447619899
Snowflake automatically converts this into a timestamp variable that looks like this:
53351-08-15 22:04:10.000.
All good so far. However, I think that the new timestamp is wrong. The actual datetime should by May 19, 2021 around 12pm MDT. Am I reading it wrong? Is it dependent on the timezone that my Snowflake instance is in?
When comparing the following options manually in SQL:
with x as (
SELECT parse_json('{time:1621447619899}') as var
)
SELECT var:time,
var:time::number,
var:time::varchar::timestamp,
1621447619899::timestamp,
'1621447619899'::timestamp,
var:time::timestamp
FROM x;
It appears that what you want to do is execute the following:
var:time::varchar::timestamp
Reviewing the documentation it does look like the to_timestamp is looking for the number as a string, so you need to cast to varchar first, and then cast to timestamp, otherwise you get what you are getting.
The question says that Snowflake transforms it to "53351-08-15 22:04:10.000" looks right, but it doesn't look right to me.
When I try the input number in Snowflake I get this:
select '1621447619899'::timestamp;
-- 2021-05-19T18:06:59.899Z
That makes a lot more sense.
You'll need to provide more code or context for further debugging - but if you tell Snowflake to transform that number to a timestamp, you'll get the correct timestamp out.
See the rules that Snowflake uses here:
https://docs.snowflake.com/en/sql-reference/functions/to_timestamp.html#usage-notes
The ::timestamp handles strings and numeric inputs differently. I.e. a string is added to 1970-01-01 as milliseconds (correct) whereas the numeric value is added in seconds which returns a date way in the future "53351-08-18 20:38:19.000".
SELECT TO_VARCHAR(1621447619899::timestamp) AS numeric_input
,'1621447619899'::timestamp AS string_input
numeric_input = 53351-08-18 20:38:19.000
string_input = 2021-05-19 18:06:59.899
Solutions are to convert to a string or divide by 1000:
SELECT TO_TIMESTAMP(time::string)
SELECT TO_TIMESTAMP(time/1000)
I am working with Mysql workbench.
I have a huge database in a csv, that contains among other things, 3 columns with different formats of dates.
To be able of loading this csv file into my database, I have to set the 3 date columns as text, otherwise, it doesn't upload them properly.
Here an example of my data:
inDate, outDAte
19-01-10, 02-02-10
04-01-11 12:02, 2011-01-11 11:31
29-01-11 6:57, 29-03-2010
30-03-10, 01-04-2010
2012-12-03 05:39:27.040, 12-12-12 17:04
2012-12-04 13:47:01.040, 29-11-12
I want to homogenize them and to make 2 columns of each one of those one only with "date" and other only with "time".
I have tried working with "regular expressions" and with "case".
When I used "reg expressions" gave me nulls and with "case" gave me "truncated incorrect value".
I have tried to find something similar situations in the web. I have found that people got similar issues but with two date formats not with so many different formats as I do:
Convert varchar column to date in mysql at database level
Converting multiple datetime formats to single mysql datetime format
Format date in SELECT * query.
I am really new in this and I do not know how to write so many exceptions in mysql.
Load the CSV into a temporary table; massage the values in that table; finally copy to the 'real' table.
Have 2 columns in that table for each date; one for the raw value coming from the CSV; the other being a DATETIME(3) (or whatever the sanitized version will be).
Do one of these for each distinctly different format:
UPDATE tmp SET inDate = ...
WHERE raw_inDate REGEXP '...';
The WHERE may need things like AND LENGTH(inDate) = 8 and other ways to test other than REGEXP.
SUBSTRING_INDEX(inDate, '-', ...) may be a handy function for splitting up a date.
But, really, I would rather write the code in Perl or some other real programming language.
I want to add a BINARY(35) column to a table in which values are written whose bits are each assigned to a specific meaning.
i.e. "000110001010..."
1st bit: day 1,
2nd bit: day 2,
etc..
I've found out how to write the value into the table
INSERT INTO MYTABLE VALUES(x'03011...');
but how do I retrieve it from the database?
If I cast the column as a character string, I'll loose everything past the first x'00' (NULL) in the value. In my application, its entirely possible that they'll still be '1's past this.
Because I'm using the C++ connector, I've only its API functions to retrieve the data so I'll need to know the type of the data retrieved. The API does not have a getBinary() function. If any of you can tell me which function to use, I'd really appreciate it.
Got the answer from another Q&A site.
SELECT HEX(mycolumn) FROM MYTABLE;
If anyone wants to read more about this:
Hexidecimal Literals: https://dev.mysql.com/doc/refman/5.7/en/hexadecimal-literals.html
Bit-Field Literals: https://dev.mysql.com/doc/refman/5.7/en/bit-field-literals.html
Substring(cast column as varchar), 1,1)
I've a table on ArcGis which contains nummbers and dates. I need to filter these via a sql-query. I just have the possibility to change the where clause.
See here: https://services3.arcgis.com/rKOPqLnqVBkPP9th/arcgis/rest/services/Arbeitsmappe1/FeatureServer/0//query
Just type in the where clause 1=1 and outfield * then you will get all results.
I have to filter installierte_leistung which contains numbers in the following formats:
1.050,20 ; 18; 0,1 ; 1.230
and dates of following format: 11.04.08
wished filters:
installierte_leistung: I want to execute a sql-statement like: where (installierte_leistung BETWEEN '1' AND '2'). In the result there is also the 18. Or if I ask for values greater 10 it shows me also the 1.050,20.
I tried to convert with cast and convert to decimal, signed, unsigned, integer and so on, but the query has been always invalid. I tried with 'number' and with number and with "number". lowercase and uppercase and almost all thinkable possibilities. I get no results with cast or convert.
Same issue with the Date. I want to filter monthly. so means between 01.2008 and 09.2009 for instance.
Could someone please help me? Thanks a lot!
Falk
I had a similar problem in the past with nested query. The more database specific queries (like cast and so) don't work because ArcGIS server is by default configured to work only with standardized queries. If you need to use more specific queries you have to change "standardizedQueries": "false" in server setting, check here how (bottom of the page): http://resources.arcgis.com/en/help/main/10.2/index.html#//015400000641000000. Should work for you. Good luck.
I am using a 3rd party .NET library (Rhino Security) that stores it's identifiers as guids in binary(16) fields in my mysql db. Everything works perfectly from the application but when I attempt to manually run a query via a query editor (TOAD for mysql) no rows are returned for identifiers I know to exist. For instance, if i run the following query, i get no results:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = '02a36462-49b7-406a-a3b6-d5accd6695e5'
Running the same query with no filter returns many results, including one with the above GUID in the EntitySecurityKey field. Is there another way to write a query to search on a guid/binary field?
Thanks!
EDIT
I found it interesting that TOAD returned a string and not an ugly blob. Using a different editor to return the results (for the unfiltered query) I get the raw binary data. I would have assume that my query would work using the binary keyword but neither of the following worked:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = BINARY '02a36462-49b7-406a-a3b6-d5accd6695e5'
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where BINARY EntitySecurityKey = '02a36462-49b7-406a-a3b6-d5accd6695e5'
I am not familiar with Rhino Security, but it uses NHibernate to store to the database as far as I know.
The .net Guid uses 1 Int32, 2 Int16 and 8 Bytes internally to store the 128-bit Guid value.
When NHibernate (3.2) stores/retrieves a Guid value to/from a BINARY(16) column, it uses the .Net ToByteArray() method and Guid(Byte[] ...) constructor respectively. These methods basically swap the order of the bytes for the Int32 and Int16 values. You can reflect on the methods to see the exact code, but here is a simple example of the first 4 bytes:
guidBytes[0] = (byte)this._int32member;
guidBytes[1] = (byte)(this._int32member >> 8),
guidBytes[2] = (byte)(this._int32member >> 16),
guidBytes[3] = (byte)(this._int32member >> 24);
This may be the cause of your issue.
For example, your Guid is stored in MySql differently.
02a36462-49b7-406a-a3b6-d5accd6695e5 - Your Guid from Guid.ToString()
6264a302-b749-6a40-a3b6-d5accd6695e5 - Your Guid in MySql using MySql hex(Guid)
Note: hex(x) does not add the dashes; I added dashes manually for comparison purposes.
Notice that the order of the first 8 bytes is different.
To test if this is your issue, run the following query:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = UNHEX(REPLACE(6264a302-b749-6a40-a3b6-d5accd6695e5,'-',''));
If so, some other items I have noticed in my own travels:
Toad for MySql complicates this too, since it will display the Guid as the .net equivalent in a select statements output. It does not apply the same conversion if you include that Guid in a select's where clause (at least from my experience). You would need to use the Guid as stored by MySql.
You will either need to manually convert the Guids when performing DB queries or you may need to introduce a custom NHibernate type to convert the Guid differently.
You can refer to:
How to read a .NET Guid into a Java UUID
http://pastebin.com/M07wnK5K (I haven't used this code, but found it useful for reference)
Update:
I just tried the custom NH type on the pastebin link above. The byte order is incorrect, at least for my environment. I would need:
private static readonly int[] ByteOrder = new[] { 3, 2, 1, 0, 5, 4, 7, 6, 8, 9, 10, 11, 12, 13, 14, 15 };
to make by .Net output match the select hex(Guid column) from table output.
What strikes me as interesting is that you're storing your GUIDs in fields that are binary(16), emphasis on the 16. Per the manual, a binary field's length is in bytes and will truncate anything that goes over it (only in strict mode though). Is it possible that your GUID's are being truncated? With your sample GUID, 02a36462-49b7-406a-a3b6-d5accd6695e5, try querying the database with the first 16 characters:
WHERE EntitySecurityKey = '02a36462-49b7-40'
Per the accepted answer to this question, a field should be char(16) binary to store a GUID, not just binary(16). However, I couldn't get this to work in my sample table.
What did work for me were using char(36) and also binary(36). Try updating your field-lengths to 36 instead of 16 (I'd do this on a test-table first, just to be safe).
Here was my test table:
CREATE TABLE test_security_keys (
test_key1 char(16) not null,
test_key2 char(16) binary not null,
test_key3 char(36) not null,
test_key4 binary(16) not null,
test_key5 binary(36) not null
);
Then, I ran a script to insert numerous GUIDs (the same one for each column in a row). You can test it with your sample GUID:
INSERT INTO test_security_keys
VALUES ('02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5', '02a36462-49b7-406a-a3b6-d5accd6695e5');
Using a simple SELECT * FROM test_security_keys will show all of the columns except the ones with size 36 to be truncated. Also, binary or not, I was able to successfully query the columns with a regular string-comparison:
SELECT * FROM test_security_keys WHERE test_key3 = '02a36462-49b7-406a-a3b6-d5accd6695e5';
SELECT * FROM test_security_keys WHERE test_key5 = '02a36462-49b7-406a-a3b6-d5accd6695e5';
If you've confirmed that your current columns with binary(16) aren't truncating, I would then-suggest to use a CAST() in your WHERE clause. The following should work (with your sample query):
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
WHERE
EntitySecurityKey = CAST('02a36462-49b7-406a-a3b6-d5accd6695e5' AS binary(16));
If you CAST(.. AS binary(16)) and the input-data is longer than 16 bytes, MySQL should issue a warning (should be unseen and not affect anything) stating that it had to truncated the data (try SHOW WARNINGS; if you get them). This is to be expected, but also means that you can't use binary(16) to store the GUIDs and you'll need to use binary(36) or char(36).
* I have not tried any of this using TOAD, but I've used both command-line MySQL and Navicat and have the same results for both.
I came across this question when I was evaluating how others "searched" for records where the primary key is a binary type (mainly the binary(16) indicative of GUID's) to compare it to my own. I must admit I was a little taken a back that the answers quickly diverged from the users desire to do manual searches based on the binary field but instead focused on changing the column types themselves to characters which, in this case, is murder on InnoDB storage.
An attempted solution for the lack of results would be to search against the Id column instead of the EntitySecurityKey column. Given JP's original query, the modified, working version would be:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where Id = UNHEX(REPLACE('02a36462-49b7-406a-a3b6-d5accd6695e5', '-', ''));
Again, this assumes the Id field is the binary(16) type. (I wasn't able to determine which column: Id, EntitySecurityKey was the binary(16)). If the EntitySecurityKey is the binary(16) column, then:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = UNHEX(REPLACE('02a36462-49b7-406a-a3b6-d5accd6695e5', '-', ''));
If that failed to yield the desired results, per this question (How to read a .NET Guid into a Java UUID), it would be of interest to try the same query but with the endianness of the first three parts of the GUID reversed:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where Id = UNHEX(REPLACE('6264a302-b749-6a40-a3b6-d5accd6695e5', '-', ''));
The only other thing I can think of is that EntitySecurityKey is a virtual column based on Id, but I don't have enough information on the table set-up to validate this statement.
Cheers.
From your EDIT, which I've a bit modifed:
Please, try
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
WHERE EntitySecurityKey = X'02a3646249b7406aa3b6d5accd6695e5'
Since the strings have 16 hexadecimal byes each, it may works...
Have you checked what the GUID data actually looks like in the table? Since it's the primary key, it quite possibly isn't stored as a GUID string. If that's the case querying the string value clearly won't work.
Possibly it's in binary format? Or possibly it's a string but without the hyphens? Or maybe it's been converted some other way? I don't know without seeing your data. But whatever the case, if a GUID that you know exists isn't being found when you query it, then whatever format it is in, it clearly isn't stored in the format you're querying.
The way I would solve this would be by searching for a different value in a record that you know exists. Or even just query the top few records without a where clause at all.
This will show you what the data actually looks like. With any luck, that will be sufficient for you to work out how it's been formatted.
If that doesn't help, one possible clue might come from this question: Why are binary Guids different from usual representation
Hope that helps.
Not sure how your database is set up, but I would try these two variants:
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = x'02a3646249b7406aa3b6d5accd6695e5'
SELECT Id, EntitySecurityKey, Type
FROM mydb.security_entityreferences
where EntitySecurityKey = x'6264a302b7496a40b6a3d5accd6695e5'