I am attempting to use the MySql Entity Framework provider for a small project. I've always encountered "quirks" when working with the MySql Connector (the one developed by MySQL, not DevArt).
The latest thing I found was that my entities generated from the database are getting long instead of int for their Id fields even though the type in the database is specified as a SIGNED INTEGER, AUTO INCREMENT, PRIMARY KEY.
The only thing I can think of is that LAST_INSERT_ID() always returns long. Since my entities rely on the database to create the identity value, perhaps MySQL Connector makes the Ids long to accommodate the id retrieval after insert?
The easiest workaround for all the issues I've encountered would be to just use SQL Server. However, the target for this app is shared hosting. Even if you pay for a SQL Server add-on, most shared hosts limit the SQL Server instances to 200MB and only allow 1 or 2 databases. That is just too small in my opinion. The same hosts offer several 1GB MySQL databases for free. If you outgrow that, you probably should have a dedicated box anyway.
MySQL will allways return a 64 bit int (or a BIGINT) instead of a 32 bit int. That's why you're getting a long instead of an int.
See this bugreport: http://bugs.mysql.com/bug.php?id=64084, and the MySQL manual on LAST_INSERT_ID(): http://dev.mysql.com/doc/refman/5.5/en/information-functions.html#function_last-insert-id.
Related
I have migrated my existing SQL server database to MySQL server database using MySQL workbench Migration Wizard. Because these are two different database servers, I want to ensure there is no data loss along with stored procedures, triggers, and views, I mean everything is intact. I tried using the MySQL workbench Compare Schema wizard but that only works for two MySQL databases. Please suggest a way to achieve it.
First you should compare database schemas between SQL Server and MySQL to see if there is difference.
I don't think that field are missing, but perharps it needs data type adjustments, index adjestments, etc.
Second, once you fix database schema, you should verify imported datas, including:
total row number for each tables
Row contents
For both, the best is to write a verification script (PHP, node.js, python, etc.) that will list all tables of SQL Server, and for each table check row number, identity counter and then datas itself.
For moving or copying the MS SQL database table's schema into MySQL, you must map data types, find NULL constraint, and determine the field that is set as a PRIMARY KEY.
This procedure does not support conversion of indexes, foreign keys, identity columns, unique or other table constraints, and character set.
MySQL supports all the important MS SQL data types. However, there are some SQL server data types that do not match with MySQL data types. Some of the major data types you’ll need to map MySQL with are as follows:
SQL Server
MySQL
VARCHAR(max)
LONGTEXT
SQL_VARIANT
BLOB
IDENTITY
AUTO_INCREMENT
AUTO_INCREMENT
TEXT CHARACTER SET UTF8
SMALLDATETIME
DATETIME
DATETIMEOFFSET
TIMESTAMP
MONEY
DECIMAL(19,4)
UNIQUEIDENTIFIER
BINARY(16)
SYSNAME
CHAR(256)
Organizations may develop the need to migrate from MS SQL server to MySQL because of its rich feature-set, cross-platform and open source availability, and lower cost.
While migration from one database to another can be performed manually, it can be an extremely time-consuming and error-prone process.
A better alternative is to use specialized database converter software like Stellar Converter for Database, which is specially designed to help DBAs and developers automate the process of converting a database file format to another. The software converts table records and attributes from MS SQL to MySQL database quickly, while preserving database integrity.
I have a database where I store MAC addresses. There is quite some discussion about how to store a MAC address in a MySQL database, I decided to store it as an unsigned BIGINT to save a few bytes per row and achieve faster searches.
This would work well, except that when I retrieve an entity from the table the mac address in the entity is limited to 2147483647, e.i. the PHP MAX_INT value for a 32 bit system. So the mac address will not be correct if it exceeds 2147483647 (which will happen for 99% of addresses).
It seems that somewhere in the process from database to entity the value is parsed as int, which breaks the values.
Is there some way of avoiding this? Would it be possible to force the entity property to be a string instead of int?
Similar question that didn't receive an answer: Cakephp 3 Bigint issue - Not receiving the same contact number
It seems the PHP deployment on my Windows 10 laptop runs in 32 bit. A solution would trying to make my PHP + Apache (XAMPP) run in 64 bit, but I would rather have a general solution.
The conversion happens in the type layer (\Cake\Database\Type), it's triggered by the query object when it fetches rows.
By default big integers are mapped to the \Cake\Database\Type\IntegerType. If you need to run this on a 32-bit system, then you could for example remap the biginteger type to \Cake\Database\Type\StringType. In your config/bootstrap.php:
Type::map('biginteger', \Cake\Database\Type\StringType::class);
See also
Cookbook > Database Access & ORM > Database Basics > Adding Custom Types
I have problem with MS Access 2007 table connected via ODBC to MySQL server (not Microsoft SQL Server).
If unique identifier in MySQL table is BIGINT - all cells content is displayed like this: "#Deleted".
I have found this article:
"#Deleted" errors with linked ODBC tables (at support.microsoft.com)
and it says:
The following are some strategies that you can use to avoid this
behavior:
Avoid entering records that are exactly the same except for the unique index.
Avoid an update that triggers updates of both the unique index and another field.
Do not use a Float field as a unique index or as part of a unique index because of the inherent rounding problems of this data type.
Do all the updates and inserts by using SQL pass-through queries so that you know exactly what is sent to the ODBC data source.
Retrieve records with an SQL pass-through query. An SQL pass-through query is not updateable, and therefore does not cause
"#Delete" errors.
Avoid storing Null values within any field making up the unique index of your linked ODBC table.
but I don't have any of these things "to avoid". My problem is in BIGINT. To make sure if this is it I created 2 tables, one with INT id, one with BIGINT. And this is it.
I can't change BIGINT to INT in my production database.
Is there any way to fix this?
Im using: Access 2007, mysql-connector-odbc-3.51.30-winx64, MySQL server 5.1.73.
You can try basing the form on an Access query, and converting the BIGINT to an INT using CInt() in the query. This happens before the form processing. Depending on your circumstance, you may need to convert to a string (CStr()) in the Query, and then manually handle validating a user has entered a number using IsNumeric. The idea is to trick the form into not trying to interpret the datatype, which seems to be your problem.
Access 2016 now supports BigInt: https://blogs.office.com/2017/03/06/new-in-access-2016-large-number-bigint-support/
It's 2019 and with the latest ODBC driver from Oracle (v 8.0.17) and Access 365 (v 16.0.11904), the problem still occurs.
When the ODBC "Treat BIGINT columns as INT columns" is ticked and in Access support for Bigint is enable in options, the Linked tables with Bigint #id columns (the primary key) shows as deleted. Ruby creates these by default, so we are loathe to fiddle with that.
If we disable the above two option, Access thinks the #id column bigint is a string and shows the data. But then the field type is not bigint or int anymore.
This is quite pathetic, since this problem is almost 10 years old now.
The MySQL driver has an option to convert BIGINT values to INT. Would this solve the issue for you?
I have an application deployed across 2 instances.
Database: MySQL
ORM: Hibernate
However, I need to implement an Oracle sequence like behaviour. Since MySQL doesn't have any, I simply created a table with an AUTO_INCREMENT and a method to return the value from it. It's thread-safe , so its not a problem when I deploy this application on 1 server. However, I dont think this thread-safe behaviour will hold true across multiple JVMs.
What to do in this case?
It is safe to use across multiple JVMs. MySQL issues each ID once. Use getGeneratedKeys() on a ResultSet to retrieve the generated ID.
I have reached the limit of RAM in analyzing large datasets in R. I think my next step is to import these data into a MySQL database and use the RMySQL package. Largely because I don't know database lingo, I haven't been able to figure out how to get beyond installing MySQL with hours of Googling and RSeeking (I am running MySQL and MySQL Workbench on Mac OSX 10.6, but can also run Ubuntu 10.04).
Is there a good reference on how to get started with this usage? At this point I don't want to do any sort of relational databasing. I just want to import .csv files into a local MySQL database and do the subsetting in with RMySQL.
I appreciate any pointers (including "You're way off base!" as I'm new to R and newer to large datasets... this one's around 80 mb)
The documentation for RMySQL is pretty good - but it does assume that you know the basics of SQL. These are:
creating a database
creating a table
getting data into the table
getting data out of the table
Step 1 is easy: in the MySQL console, simply "create database DBNAME". Or from the command line, use mysqladmin, or there are often MySQL admin GUIs.
Step 2 is a little more difficult, since you have to specify the table fields and their type. This will depend on the contents of your CSV (or other delimited) file. A simple example would look something like:
use DBNAME;
create table mydata(
id INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
height FLOAT(3,2)
);
Which says create a table with 2 fields: id, which will be the primary key (so has to be unique) and will autoincrement as new records are added; and height, which here is specified as a float (a numeric type), with 3 digits total and 2 after the decimal point (e.g. 100.27). It's important that you understand data types.
Step 3 - there are various ways to import data to a table. One of the easiest is to use the mysqlimport utility. In the example above, assuming that your data are in a file with the same name as the table (mydata), the first column a tab character and the second the height variable (with no header row), this would work:
mysqlimport -u DBUSERNAME -pDBPASSWORD DBNAME mydata
Step 4 - requires that you know how to run MySQL queries. Again, a simple example:
select * from mydata where height > 50;
Means "fetch all rows (id + height) from the table mydata where height is more than 50".
Once you have mastered those basics, you can move to more complex examples such as creating 2 or more tables and running queries that join data from each.
Then - you can turn to the RMySQL manual. In RMySQL, you set up the database connection, then use SQL query syntax to return rows from the table as a data frame. So it really is important that you get the SQL part - the RMySQL part is easy.
There are heaps of MySQL and SQL tutorials on the web, including the "official" tutorial at the MySQL website. Just Google search "mysql tutorial".
Personally, I don't consider 80 Mb to be a large dataset at all; I'm surprised that this is causing a RAM issue and I'm sure that native R functions can handle it quite easily. But it's good to learn new skill such as SQL, even if you don't need them for this problem.
I have a pretty good suggestion. For 80MB use SQLite. SQLite is a super public domain, lightweight, super fast file-based database that works (almost) just like a SQL database.
http://www.sqlite.org/index.html
You don't have to worry about running any kind of server or permissions, your database handle is just a file.
Also, it stores all data as a string, so you don't even have to worry about storing the data as types (since all you need to do is emulate a single text table anyway).
Someone else mentioned sqldf:
http://code.google.com/p/sqldf/
which does interact with SQLite:
http://code.google.com/p/sqldf/#9._How_do_I_examine_the_layout_that_SQLite_uses_for_a_table?_whi
So your SQL create statement would be like this
create table tablename (
id INT(11) INTEGER PRIMARY KEY,
first_column_name TEXT,
second_column_name TEXT,
third_column_name TEXT
);
Otherwise, neilfws' explanation is a pretty good one.
P.S. I'm also a little surprised that your script is choking on 80mb. It's not possible in R to just seek through the file in chunks without opening it all up in memory?
The sqldf package might give you an easier way to do what you need: http://code.google.com/p/sqldf/. Especially if you are the only person using the database.
Edit: Here is why I think it would be useful in this case (from the website):
With sqldf the user is freed from having to do the following, all of which are automatically done:
database setup
writing the create table statement which defines each table
importing and exporting to and from the database
coercing of the returned columns to the appropriate class in common cases
See also here: Quickly reading very large tables as dataframes in R
I agree with what's been said so far. Though I guess getting started with MySQL (databases) in general is not a bad idea for the long if you are going to deal with data. I mean I checked your profile which says finance PhD student. I don't know if that means quant. finance, but it is likely that you will come across really large datasets in your career. I you can afford some time, I would recommend to learn something about databases. It just helps.
The documentation of MySQL itself is pretty solid and you can a lot of additional (specific) help here at SO.
I run MySQL with MySQL workbench on Mac OS X Snow Leopard too. So here´s what helped me to get it done comparatively easy.
I installed MAMP , which gives my an local Apache webserver with PHP, MySQL and the MySQL tool PHPmyadmin, which can be used as a nice webbased alternative for MySQL workbench (which is not always super stable on a Mac :) . You will have a little widget to start and stop servers and can access some basic configuration settings (such as ports through your browser) . It´s really one-click install here.
Install the Rpackage RMySQL . I will put my connection string here, maybe that helps:
Create your databases with MySQL workbench. INT and VARCHAR (for categorical variables that contain characters) should be the field types you basically need at the beginning.
Try to find the import routine that works best for you. I don't know if you are a shell / terminal guy – if so you'll like what was suggested by neilfws. You could also use LOAD DATA INFILE which is I prefer since it's only one query as opposed to INSERT INTO (line by line)
If you specify the problems that you have more accurately, you'll get some more specific help – so feel free to ask ;)
I assume you have to work a lot with time series data – there is a project (TSMySQL) around that use R and relational databases (such as MySQL, but also available for other DBMS) to store time series data. Besides you can even connect R to FAME (which is popular among financers, but expensive). The last paragraph is certainly nothing basic, but I thought it might help you to consider if it´s worth the hustle to dive into it a little deeper.
Practical Computing for Biologists as a nice (though subject-specific) introduction to SQLite
Chapter 15. Data Organization and Databases