The POCO libraries support MySQL DATE, TIME and DATETIME columns, but not TIMESTAMP. Selecting values from a TIMESTAMP column raises an "unknown field type" exception, since MYSQL_TYPE_TIMESTAMP is not supported in "Poco/Data/MySQL/ResultMetadata.cpp".
In my project I had to change several columns to DATETIME to make it work. This was not a big problem, still I wonder what the reason for this limitation is. If I had to work with an existing database schema that I couldn't alter, I'd be in serious trouble.
Timestamp columns are widely used, hence I don't believe they were simply omitted. Is there an implementation problem as to Timestamp columns? Is there a workaround I could use? Is it planned to add MySQL timestamp support to POCO in the future?
Download the source of POCO libraries then modify the file Data/MySQL/src/ResultMetadata.cpp
std::size_t fieldSize(const MYSQL_FIELD& field)
/// Convert field MySQL-type and field MySQL-length to actual field length
{
...
case MYSQL_TYPE_DATE:
case MYSQL_TYPE_TIME:
case MYSQL_TYPE_DATETIME:
case MYSQL_TYPE_TIMESTAMP: // <-- add this line
return sizeof(MYSQL_TIME);
...
}
Poco::Data::MetaColumn::ColumnDataType fieldType(const MYSQL_FIELD& field)
/// Convert field MySQL-type to Poco-type
{
...
case MYSQL_TYPE_DATETIME:
case MYSQL_TYPE_TIMESTAMP: // <-- add this line
return Poco::Data::MetaColumn::FDT_TIMESTAMP;
...
}
Compile the library and you will have support for TIMESTAMP fields.
Related
I am using Sequelize to map objects to my already existing database, in this database they didnt use the auto-generated timestamps, they did their own datetime work, the current way the database stores the date is in YYYY-MM-DD HH:MM:SS.MS however using sequelize, i am returning a date that looks like YYYY-MM-DD HH:MM:SS.MS +00:00 and i believe this is whats tripping me up. as i am receiving an error Conversion failed when converting date and/or time from character string. I tried even accessing the Date() object and building my own date string, and i got the same error. even though it follows the correct formatting.
for my model i have this at the CreateDateTime (This is the name of the field in the database.
CreateDateTime: {
type: DataTypes.DATE,
defaultValue: `${date.getFullYear()}-${date.getMonth()}-${date.getDay()} ${date.getHours()}:${date.getMinutes()}:${date.getSeconds()}.${date.getMilliseconds()}`,
},
UpdateDateTime: {
type: DataTypes.DATE,
},
I have also, renamed the timestamps that sequelize provides to the appropriate names above. I am not sure much more of how to get around this issue besides maybe using the timestamps provided by sequelize and dropping the ones in the existing.. though id rather not do this. even though eventually ill move this database to a better schema in which the timestamps are autogenerated.
is there a different way around this error? No matter what i seem to do the sql insert query always seems to tack on the +00:00 to the end, is this some kind of default value of using the DataType DATE? is there a different data type i should be using?
So I figured out that the previous team likely wasn't storing an actual datetime timestamp, because once i changed the DataType from DATE to STRING it worked just fine, i was no longer receiving this error, but it still strikes me as strange, because using my DB exporer (i use Razor) when i was having the table described to me, it clearly stated that these fields where of datetime and not varchar, so im not sure why this works, only that it follows the appropriate pattern and it does indeed work.
in my table I have a Generated column of Decimal type,
COLUMN `quantidadeTotal` DECIMAL(18,4) GENERATED ALWAYS AS (((`quantidade` + `quantidadeReservada`) + `quantidadeBloqueada`)) VIRTUAL;
which is the sum of 3 other columns also of Decimal type, the problem occurs when I execute "scaffold", the generated model comes with this column in this format
public decimal? QuantidadeTotal { get; set; }
and in generated context also has reference to the
entity.Property(e => e.QuantidadeTotal)
.HasColumnName("quantidadeTotal")
.HasColumnType("decimal(18,4)");
column
if i try to insert some record in the table i get an error, even if not informing anything in the Generated column, i have to delete the reference of this column in the context and in the model to work.
is there any configuration for scaffold to ignore this type of column or any special treatment?
if not, every time I make a scaffold I will have to adjust all fields manually, and it would be a lot of work.
Pomelo.EntityFrameworkCore.MySql v3.2.4
thanks!
Thanks for reporting this bug on our repository as Database first and Generated column #1256!
We fixed it with:
3.2.x: Add scaffolding support for computed columns #1257
5.0.0+: Add scaffolding support for computed columns, including STORED support #1260
We are starting to use the MySQL json datatype. Is there any recommended best practices when storing default values as NULL or {} for the JSON datatype? What are the PROs and CONs for each?
This would depend upon your business use-case.
null is usually meant to indicate that the value is unknown/doesn't
exists.
{} on the other hand means that the value is known/does
exist and the known value is an empty JSON object {}
Keep in mind that MySQL did not allow non-null default values for JSON columns prior to version 8.0.13 (released on 22/10/2018), according to the documentation:
Prior to MySQL 8.0.13, a JSON column cannot have a non-NULL default value.
If you try it on a previous version you get an error:
BLOB, TEXT, GEOMETRY or JSON column 'column_name' can't have a default value
So if backwards compatibility is a requirement, this is a con for {}.
PhpMyAdmin uses the text null by default, if you add a new JSON column to existing rows.
Note this is different from NULL. NULL means unknown/doesn't exist in the database, null means unknown/doesn't exist after converted to JSON. {} is an empty JSON object.
[42000][1101] BLOB, TEXT, GEOMETRY or JSON column 'xxxx' can't have a default value.
Rails' :timestamp column type lies; it's actually just an alias for :datetime.
I'm using mysql, and I want to use actual unix-timestamp TIMESTAMP columns.
a) Is there a nice way to set this, other than just making the column using SQL?
b) Will ActiveRecord cope with it properly (e.g. converting to Time when necessary, accepting a unix timestamp Integer as input, etc)? What gotchas should I expect to have to handle, and where?
Why:
Speed. This is for an extremely active table that's aggregating outside data sources that already use unix timestamps. Converting to datetime (or even converting first to a db string, which goes through 2 gsubs) uses up the majority of its import time. I could otherwise be doing just a dirt cheap Integer#to_s call.
Timezones. I don't want 'em. I want it stored timezone-agnostically; dealing with timezones is a pain and is completely irrelevant to my needs except at the very final stage before individual user display. The data itself has no need to know what timezone it was recorded in.
Size. It's a large table. TIMESTAMP is half the size of DATETIME.
Yes, I would still be doing updated_at calculations in code, not mysql. That part isn't a bottleneck.
Why your 'why not' is wrong (preëmptively, to show I'm not asking for noobish reasons :-P):
"But TIMESTAMP auto updates": That's only true by default, and can be easily switched off.
I'm actually not using Rails, just ActiveRecord.
Yes, this is based on actual profiling data; I am not early optimizing. ActiveRecord::ConnectionAdapters::AbstractMysqlAdapter#quote (in Quoting#quoted_date [if passing Time] or Mysql2Adapter#quote_string [if preconverting to_s(:db)]) is actually the most CPU-consuming section of my scraper. I want rid of it.
this works (just added whitespace character to type definition, so :timestamp doesn't override it):
t.add_column :sometable, :created_at, 'timestamp '
I'm pretty noobish, but I'll give it a shot. What if you were to add your own custom column, or overwrite the default ones? You can use custom data types with a string like so:
t.add_column :mysql_timestamp, 'timestamp'
and then somewhere else in your logic
def mysql_timestamp
Time.now.strftime("%Y-%m-%d %H:%M:%S")
end
Not sure about b). Only one way to find out!
I'm transferring a MySQL table to MongoDB. There is a primary key id in my MySQL table and I want this id to be converted to the _id in MongoDB.
I use php's MongoCollection::insert( $mysql_array );
However it doesn't work because if I set $mysql_array['_id'] it is seen by mongo as a String instead of a MongoId. I tried $mysql_array['_id'] = new MongoId( $id ) but it doesn't allow me to override the default _id value. I saw that all my MySQL's integer columns are converted to string by MongoCollection::insert(). If I could set MongoCollection::insert() to correctly transfer an integer it would maybe work.
typecast the _id to a integer value like this...
(int) $mysql_array['_id']
You'll find yourself doing this a lot in mongoDB
The ObjectId is a special type in Mongo, but the _id property doesn't have to be of this type. You can't coerce a string or number into an ObjectId, and you shouldn't.
I assume the problem as you perceive it is that your insert worked, but when you looked at the data in the database the _id property didn't look like _id: ObjectId("1234") (if the original ID was 1234). This is as it should be, and it's perfectly fine.
The idea with ObjectId is that it has a predefined structure that makes it guaranteed (mostly) to be unique across a Mongo cluster, but this also means that it has to have this structure, otherwise it is not an ObjectId.
You also mention that all your integer columns are converted to strings. PHP and PHP libraries, are notoriously sloppy when it comes to types, so make sure that it's not the case that the values are already strings when they come from the MySQL result set. Worst case you have to explicitly cast the values before inserting them into Mongo.
You won't be able to convert an arbitrary String value into an Mongo ObjectId due to its specific characteristics (12 bytes -> 24 chars generated from 4 bytes timestamp, 3 bytes client hostname, 2 bytes PID, 3 bytes inc value).
Either you abandon using the MongoId type in your collection's _id-fields and use your MySQL-ID as a string instead (which is not a problem and makes the most sense) or you let Mongo to generate the documents' _id for you, which is also a suitable solution if you want to be able to use the MongoId functions (assuming you're working with PHP):
The MongoId class
Choosing the second solution you still are able to store your MySQL-IDs in another field of the doc, like id or *mysql_id* to reference them later.
Concerning your question about (int) and (string) values: Are you sure they come as a PHP integer from your MySQL DB? If so, they usually should be stored as integers in Mongo. Check it with a var_dump() and in case of incompatibility cast it with an (int). Maybe it would be helpful if you post your select/insert code...
Use MongoCollection::save() and your array should work.