I want to write a dataframe to a mysql database and assign properties like primary key, auto_increment, etc to the created table.
I am using the com.mysql.cj.jdbc.Driver.
Import is:
libraryDependencies += "mysql" % "mysql-connector-java" % "8.0.11"
I checked connection properties and couldn't find anything there.
If the table alredy exists and I created it beforehand with a primary key it is possible to write into the table and keep the schema
df.write.mode("overwrite").option("truncate", "true").jdbc(connectionString, tableName, properties)
Is there a way to write a dataframe to a table that does not exist yet and set the primary key (maybe even with auto_increment property)
I did not find any way yet to accomplish this.
What I get is:
What I want is
Related
Is there are way to make ID fiels in MySql immutable?
It's possible in postgreSQL using PGAdmin for example.
ID field sets as primary key and auto incr but can be changed manually.
I'm creating a mysql db in which to store files of different formats and sizes like pdfs, imgs, zips and whatnot.
So I started looking for examples on the blob data type (which I think is the right data type for storing the above mentioned files) and I stumbled upon this SOquestion. Essentially what the answer suggests is not to store the blob files directly into the "main" table but create two different tables, one for the file description and the other for the blobs themselves (as these can be heavy to get). And connect these tables by a foreign key constraint to tie the file to its description and do a join operation to retrieve the wanted blob if needed.
So I've created the following tables:
create table if not exists file_description(
id int auto_increment primary key,
description_ varchar(200) not null,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) engine=INNODB;
create table if not exists files(
id int auto_increment primary key,
content longblob not null,
format_extension varchar(10) not null,
foreign key (id) references file_description(id)
on update cascade
on delete cascade
)engine=INNODB;
But how can I enforce that after each insertion into the file_description table directly follows an insertion into the files table?
I'm no expert but for what I've seen on triggers they are used in a different way than what I would like to do here. Something like
create trigger whatever
on file_description after insert
...
I don't know, how do I do that?
You cannot enforce through database tools that an insertion into a parent table is followed by an insertion into a child table as the data to be inserted come from outside of the database. You need to design your application in a way that it populates both tables right after each other.
What the application can do is to encapsulate the two insert statements into a single transaction ensuring that either both inserts succeed or both are rolled back leaving your database in a consistent state.
So the table schema looks like:
field - type - index
'id' - INT - PRI
'event_id' - BIGINT(20) - UNI
co1 ... colN (no indexes)
LOAD DATA command:
LOAD DATA LOCAL INFILE 'my-file' REPLACE INTO TABLE mydb.mytable (event_id, col1 .. colN)
and get error:
ERROR 1062 (23000) at line 1: Duplicate entry '2147483647' for key 1
'key 1' refers to the unique key on 'event_id'.
More context:
(1) The table is empty at the time of LOAD DATA.
(2) When I grep for 2147483647 in my source file I get nothing
(3) 2147483647 is the integer max value
(4) I am not actually inserting any value in 'id' -- its just auto incrementing
(5) I am using the 'REPLACE' keyword in the LOAD DATA so even there were dupes, it should know how to deal with them?
This suggest some int overflow issue (ie, I don't think there are any genuine dupes in the source data or in the table), and indeed the values in 'my-file' for 'event_id' are all over the integer max limit. However, the error is odd because 'event_id' column is BIGINT.
As a temporary fix, I dropped the unique index on 'event_id' and the LOAD DATA command worked! The values in 'event_id' were all fine, not truncated or anything. So there is nothing wrong with the table handling the values, but somehow LOAD DATA is checking the uniqueness but as an integer?
Has anyone encountered something similar? Thanks
So what this means is that '2147483647' already exists in that database in a field marked as a key. Just remove the key from that field and you should be fine!
Edit: As stated in your question, event_id is your primary key - you cannot have the same value twice in a primary key.
Best of luck!
the problem is not the data type,
the thing is that you only have 2 fields and one is PK and the other is UNIQUE, so there is no way that you could have repeated a value. When you make an insert or load data it's trying to add twice an entry with this value "2147483647", you have several ways to fixed it.
The first one try to open the file with a text editor and find the repeated value and fixed it, if it's not working try with a mysqldump or phpmyadmin to export the data and edit the file in a text editor.
Make an export again, if the problem persist try export the data by others methods (mysqldum or phpmyadmin) and re import it
Try recreating a table without primary keys, you can use this
create table big_int_test (select * from table1);
and it will make a copy of the table without the PK, INDEXES and FK.
You can delete the index and UNIQUE key constraint, import the data, fix the table (delete the repeated values) and re create the PK and UNIQUE KEY (this is a kind of crazy but It could work)
I recently noticed that there are several bigint type fields in one of my databases that could be replaced with type "int". The problem is that the database is already online and in use and also that there are foreign keys in action, so when I try changing the data type in the offline DB, mysql won't let me and produces the error message: "#1025 - Error on rename of (...)".
So how can I change these fields while keeping the foreign keys happy? (and not wiping out the existing DB!)
You can alter a column from BIGINT to INT as long as there is not any data in the column that would not fit into the smaller INT datatype.
ALTER TABLE mytable
MODIFY COLUMN mycolumn INT(11);
You could try:
Add new columns with proper types.
Copy values from old columns to new columns.
Rename columns.
Update constrains (maybe in two steps -- DROP and CREATE).
Drop old columns.
You can't reduce the size of field when it holds data. You can only expand them.
What you can do is export all the data, disable the keys, empty the tables, change the field sizes & import the data again.
I'm trying to reset the seed for an autoincrement field in SQLITE using AIR/ActionScript/Flex 4.5.
I'm doing what should normally work:
DELETE FROM SQLITE_SEQUENCE WHERE NAME = 'myTable'
I've checked that the database actually contains a SQLITE_SEQUENCE table. I'm able to execute the above statement without errors directly via the Firefox SQLITE plugin (SQLite Manager).
However, when I try to do the same using actionscript, I get an error:
No such table 'SQLITE_SEQUENCE'.
All I could find in searching around was this one guy who posted everywhere he could find - only to be left without an answer:
here
and
here
and
here
Any ideas?
sqlite_sequence table is not created, until you define at least one autoincrement and primary key column in your schema.
You should to insert "Auto increment" to primary key column
at least to one table,
Then SQLite is creating "SQLITE_SEQUENCE" table.
To get all tables have Auto increment:
SELECT * FROM SQLITE_SEQUENCE;