I need to load data into a DB using Sequelize on first application load. The initial excel data was given in the following format:
Car group fields: title | group_code
Car group data:
('Mercedes','M'),
('Volkswagen','VW');
Car Fields: car_code | owner | group_code
Car data:
('11-1135','Fred','M'),
('11-1146','Bob','VW');
--
Ideally what I want to end up with in the DB is the following:
Car group fields: group_id | title | group_code
Car group data:
(1, 'Mercedes','M'),
(2, 'Volkswagen','VW');
Car Fields: car_id | car_code | owner | group_id (refers to the group id created above)
Car data:
(1, '11-1135','Fred', 1),
(2, '11-1146','Bob', 2);
--
What is the best approach to doing this in Sequelize? In SQL I did the following to get around this problem:
1- Converted my Excel file into a bunch of SQL statements
2- Created the following script using those statements (and then i added my own code to fill in the group_id):
CREATE TABLE CarGroup(
group_id INT NOT NULL AUTO_INCREMENT,
title VARCHAR(100) NOT NULL,
group_code VARCHAR(5) NOT NULL,
PRIMARY KEY (`group_id`),
CONSTRAINT UN_car_group_code UNIQUE (group_code)
) ENGINE=InnoDB;
INSERT INTO CarGroup(title,group_code) VALUES ('Mercedes','M');
INSERT INTO CarGroup(title,group_code) VALUES ('Volkswagen','VW');
CREATE TABLE IF NOT EXISTS Car(
car_id INT NOT NULL AUTO_INCREMENT,
car_code VARCHAR(10),
owner VARCHAR(100) NOT NULL,
group_id SMALLINT, -- populated after insert
group_code VARCHAR(10) NOT NULL, -- deleted after insert
PRIMARY KEY (id),
CONSTRAINT `UN_car_code` UNIQUE (`car_code`),
CONSTRAINT `FK_car_group_id` FOREIGN KEY (`group_id`) REFERENCES `CarGroup` (`group_id`)
) ENGINE=InnoDB;
INSERT INTO Car(car_code,owner,group_code) VALUES ('11-1135','Fred','M');
INSERT INTO Car(car_code,owner,group_code) VALUES ('11-1146','Bob','VW');
-- GENERATE GROUP ID'S BASED ON GROUP CODE AND DROP GROUP CODE COLUMN --
update Car INNER JOIN CarGroup ON Car.group_code = CarGroup.group_code
SET Car.group_id = CarGroup.group_id;
alter table Car drop column group_code
I can't see how the above can be achieved by using migrations and seeding as I need to create the model then do seeding and then run the alteration. Is it easier to just run plain SQL statements in Sequelize in this case? Or should I just use the data as it is and link the two tables as a foreign key via the group_code (which is a string - not best performance in comparison to plain INT id).
Any direction on this is muchly appreciated!
Not sure if this is the best approach but since no one answered, but i have decided to do the following:
Create two tables, OriginalCars and Cars. OriginalCars has the original fields that the excel file has (i.e. car_code). The Cars table has the car_id and other fields.
Create the models
Sync the models
Check manually if there is any data in the tables, if not then populate the originalCars table with data. I then do an innerjoin of the OriginalCars with the group table, the resulting data is parsed and added to the Car table with car_id.
Delete the original table as its no longer needed
Feels a tad hacky but it only has to do this on initial load of the App to populate the initial data.
Related
I have a database with multiple tables, and I want to add a column to one table that will be populated with different strings based on the contents of another table.
Below are the tables of interest.
CREATE TABLE Locations(
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
Location VARCHAR(17) NOT NULL,
Is_Property BOOLEAN NOT NULL
);
CREATE TABLE Players(
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
Player_Name VARCHAR(17) NOT NULL,
Token VARCHAR(17) NOT NULL,
FOREIGN KEY (Token) REFERENCES Tokens(Token),
P_Location VARCHAR(17) NOT NULL,
FOREIGN KEY (P_Location) REFERENCES Locations(Location),
Bank_Balance INT NOT NULL DEFAULT 200);
ALTER TABLE Locations ADD INDEX `Location` (`Location`);
CREATE TABLE Properties AS SELECT id,Location FROM Locations
WHERE Is_Property = 1;
ALTER TABLE Properties
ADD CONSTRAINT PK_Properties PRIMARY KEY (id),
ADD COLUMN Colour VARCHAR(6),
ADD COLUMN Cost_And_Rent INT,
ADD COLUMN Owned VARCHAR(3);
CREATE TABLE Properties_Owned(
Player_id INT NOT NULL,
Prop_id INT NOT NULL,
PRIMARY KEY(Player_id, Prop_id),
FOREIGN KEY (Player_id) REFERENCES Players(id),
FOREIGN KEY (Prop_id) REFERENCES Properties(id));
The Properties and Properties_Owned tables are of interest in this case. I want to create a column called Owned in Properties and populate it with "Yes" or "No" based on if the primary key appears under Prop_id in Properties_Owned. Ergo if it does, Properties.Owned will show "Yes", and if not, "No".
I've tried using the CASE function, but I'm unsure of if it can be used without calling a SELECT query. Below is my last attempt to do so, but the syntax is wrong somewhere or just misguided altogether.
CASE
WHEN id IS IN properties_owned.Prop_id THEN Properties.Owned = "Yes"
ELSE "No" ;
It generates the error code:
Error Code: 1064. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'CASE WHEN id IS IN properties_owned.Prop_id THEN Properties.Owned = "Yes" ELSE ' at line 1
Edit: As there was a request for sample data, here it is:
INSERT INTO Locations(Location,Is_Property) VALUES ("GO", 0),
("London", 1),
("Paris", 0),
("China", 1),
("New Zealand", 0),
("Sydney", 1),;
INSERT INTO Players(Player_Name,Token,P_Location,Bank_Balance) VALUES
("Mary","Battleship","London",190),
("Bill","Dog","Paris",500),
("Jane","Car","China",150),
("Norman","Thimble","London",250);
INSERT INTO Properties_Owned(Player_id,Prop_id) VALUES
(1,1),
(2,2),
(3,3),
(4,4),
(4,5);
Thus the Properties_Owned table will look like this:
Player_id | Prop_id
----------|---------
1 |1
2 |2
3 |3
4 |4
4 |5
And so in the Properties table under Owned, if Properties.id appears in Prop_id above, the Owned column should yield "Yes".
The table Properties_Owned reflects an (n:n)-relation (many-to-many). But according to your comment, a property cannot be owned by multiple players at the same time. And that would be an (n:1)-relation (many-to-one). In that case you do not need the Properties_Owned table and can just add a column Player_id (or owned_by_player_id) to the Properties table as foreign key referencing the Players table.
alter table Properties
add column Player_id int default null,
add foreign key (Player_id) references Players(id);
Then the information required for the Owned column will be already in the same table. You just need to "manipulate" it in your SELECT statements. For example with:
select
p.*,
case when Player_id is null then 'No' else 'Yes' end as Owned
from Properties p;
No need to store the same information redundantly. That would "bite" you sooner or later. Avoid redundancy when possible.
If your MySQL version (5.7+ required) supports Generated Columns you can also let the database maintain the redundancy.
alter table Properties
drop column Owned,
add column Owned varchar(3)
as (case when Player_id is null then 'No' else 'Yes' end) virtual;
Now the (genrated) column is dependent on Player_id column and you don't need (and cannot) store anything there but can select it. virtual means that it is not stored but generated (on the fly) when it's needed. Now you can read it in your queries as if it is normal column.
See example on db-fiddle.com
And again: Avoid redundant data when possible. At least use foreign keys to avoid data inconsistency.
You could create a view that extends your table by the column you want. There you can use a CASE statement.
CASE WHEN id IN SELECT Prop_id FROM properties_owned THEN 'Yes' ELSE 'No'
If this is not what you want, you could possibly use triggers on both tables that fill/update the column on on create/on delete
Initially you could fill the column with something like this:
UPDATE Prop_id SET properties_owned = CASE WHEN id IN SELECT Prop_id FROM properties_owned THEN 'Yes' ELSE 'No' WHERE
I recently started to work with SQL in the Visual Studio environment, I have created the following two tables and populated them with values, these are the command for the creation of the tables users and photos:
CREATE TABLE users(
id INTEGER AUTO_INCREMENT PRIMARY KEY,
username VARCHAR(255) UNIQUE NOT NULL,
created_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE photos(
id INTEGER AUTO_INCREMENT PRIMARY KEY,
image_url VARCHAR(255) NOT NULL,
user_id INTEGER NOT NULL,
created_at TIMESTAMP DEFAULT NOW(),
FOREIGN KEY(user_id) REFERENCES users(id)
);
Now these are the statements that I ran to populate the tables
INSERT INTO users(username) Values
('Colton'),('Ruben');
INSERT INTO photos(image_url,user_id) VALUES
('/alskjd76',1),
('/lkajsd98',2);
Now if I run the statement
SELECT *
FROM photos
JOIN users;
I get the tables:
Now if I run the command:
SELECT *
FROM users
JOIN photos;
I get the table
Here are the tables of users and the tables for photos.
Now my question is why is it that "id" column in the second table is changed to 4,4,5,5 when the actual "id" column of the users table only contains the values 1,2? The first instance seems to respect this why doesn't the second?
EDIT: It seems to be displaying the following now when running the commands
SELECT *
FROM photos
JOIN users;
and when I run :
SELECT *
FROM users
JOIN photos;
Edit: this seems to be correct now, is this right, it seems to have been solved with the deletion and recreation of the tables entirely. I think that V.S studio might have mistakenly taken the table to have more present photos with id's 1-3.
I'm very beginner at MySQL and have just started to play around with foreign keys and INNER JOIN operator.
So I've made a few tables like:
CREATE TABLE `models`
(
`id` TINYINT UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR (255) NOT NULL,
`price` MEDIUMINT UNSIGNED NOT NULL,
PRIMARY KEY( `id` )
);
CREATE TABLE `vendors`
(
`id` TINYINT UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR (255) NOT NULL,
`id_model` TINYINT UNSIGNED NOT NULL,
PRIMARY KEY( `id` ),
FOREIGN KEY (`id_model`) REFERENCES models(`id`)
);
CREATE TABLE `cars`
(
`serial_number` MEDIUMINT UNSIGNED NOT NULL,
`id_vendor` TINYINT UNSIGNED NOT NULL,
FOREIGN KEY (`id_vendor`) REFERENCES vendors(`id`),
PRIMARY KEY( `serial_number` )
);
I know how to get output with that. However, the problem is that I don't know how to insert data properly. All I can do is insert data table by table. But how to do it in one query, and if I am inserting Honda Civic and already have Honda Accord, for example, wouldn't it duplicate Honda vendor in the database?
It seems that the structure of the database is not really coherent. Maybe I don't understand what you are trying to do, but ... anyway, here goes.
Assuming that what you want to do is store a list a of cars in a properly normalized relational database, first thing you want to do is think what is happenning in "real life":
Manufacturers (vendors) make/sell cars
Manufacturers (vendors) make different models of cars
Cars have a model (and a serial number in your case)
Models belong to a vendor (manufacturers)
Considering this, your table structure is:
Vendors
- id
- name
Models
- id
- name
- vendor ( foreign key => vendor.id )
Cars
- id
- serial_number
- model ( foreign key => model.id )
You don't need to have a reference to the vendor in the cars table becoause you have a reference to the model, which in turn has a reference to the vendor.
Whe inserting, you do it one by one, making sure that the foreign key entries already exist.
When you insert a car object, you just need to provide the model id.
When you insert a model object you need to provide a vendor id.
So the Honda Civic/Accord situation does not duplicate Honda. The Tables should be something like this:
Vendor
id, name
1, "Honda"
Model
id, name, vendor
1, "Civic", 1
2, "Accord", 1
Cars
id, serial_no, model
1, "A serial", 2 -> a honda accord
2, "Another serial", 1 -> a honda civic
Hope this helps somewhat.
You do need to check if duplicated record exists yourself.
IF EXISTS (SELECT * FROM vendors WHERE Name = 'Honda')
BEGIN
-- Insert into cars with existing vendor id
END
ELSE
BEGIN
IF EXISTIS (SELECT * FROM models WHERE Name = 'your model name')
BEGIN
-- insert into vendors with existing model id
END
ELSE
BEGIN
-- insert into models
-- insert into vendors
-- insert into cars
END
END
You can create stored procedure for it and pass car, vendor and model as parameters.
Or you can list models first, insert them; then all vendors and all cars. Just a silly answer. Welcome more sophisticated solutions.
Note: Apologies if this is a duplicate but I can't find a solution.
I have two databases (one dev and one live) which have exactly the same schema.
To make things easier to explain, assume I have a 'customer' table and a 'quote' table. Both tables have auto increment ids and the quote table has a 'customerid' column that serves as a foreign key to the customer table.
My problem is that I have some rows in my dev database that I want to copy to the live database. When I copy the customer rows I can easily get a new id, but how can i get the new id to be assigned to the 'child' quote table rows?
I know I can manually script out INSERTS to overcome the problem but is there an easier way to do this?
EDIT:
This is a simplified example, I have about 15 tables all of which form a hierarchy using auto-increments and foreign keys. There is considerably more data in the live database so the new ids will be bigger (e.g. dev.customer.id = 4, live.customer.id = 54)
Easiest way without changing any IDs.
Ensure that you are currently in the table where the record you want to copy is in (source db).
Run the following command:
INSERT INTO to_database.to_table
SELECT * FROM from_table WHERE some_id = 123;
No need to specify columns if there is no need to remap anything.
Hope that helps!
I eventually managed to do this (as per my comment) but in order to do so I had to write some code. In the end I created some dummy tables that kept track of the old id against new id so. When copying over records with FK constraints I just looked up the new id based on the old. A bit long winded but it worked.
This post is getting on a bit now so I've marked this as the answer. If anyone out there has better ideas/solutions that work I'll happily 'unmark' it as the accepted answer.
EDIT: As requested here is some pseudo-code that I hope explains how I did it.
I have the two related tables as follows:
CREATE TABLE tblCustomers (
Id int NOT NULL AUTO_INCREMENT,
Name varchar(50) DEFAULT NULL,
Address varchar(255) DEFAULT NULL,
PRIMARY KEY (Id)
)
ENGINE = MYISAM
ROW_FORMAT = fixed;
CREATE TABLE tblQuotes (
Id int NOT NULL AUTO_INCREMENT,
CustomerId int(11) DEFAULT NULL,
QuoteReference varchar(50) DEFAULT NULL,
PRIMARY KEY (Id)
)
ENGINE = MYISAM
ROW_FORMAT = fixed;
I create an extra table that I will use to track old ids against new ids
CREATE TABLE tblLookupId (
Id int NOT NULL AUTO_INCREMENT,
TableName varchar(50) DEFAULT NULL,
OldId int DEFAULT NULL,
NewId int DEFAULT NULL,
PRIMARY KEY (Id)
)
ENGINE = MYISAM
ROW_FORMAT = fixed;
The idea is that I copy the tblCustomer rows one at a time and track the ids as I go, like this:
// copy each customer row from dev to live and track each old and new id
//
foreach (customer in tblCustomers)
{
// track the old id
var oldid = customer.id; // e.g. 1
// insert the new record into the target database
INSERT newdb.tblCustomers (...) VALUES (...);
// get the new id
var newid = SELECT LAST_INSERT_ID() // e.g. 245
// insert the old id and the new id in the id lookup table
INSERT idlookup (TableName, OldId, NewId) VALUES ('tblCustomers', oldid, newid); // this maps 1->245 for tblCustomers
}
When I come to copy the table (tblQuote) with the foreign key I have to first lookup the new id based on the old.
// copy each quote row from dev to live and lookup the foreign key (customer) from the lookup table
//
foreach(quote in tblQuotes)
{
// get the old foreign key value
var oldcustomerid = quote.CustomerId; // e.g 1
// lookup the new value
var newcustomerid = SELECT newid FROM tblIdLookup WHERE TableName='tblCustomers' AND oldid=oldcustomerid; // returns 245
// insert the quote record
INSERT tblQuotes (CustomerId, ...) VALUES (newcustomerid, ...);
}
I've tried to keep this short and to the point (and language agnostic) so the technique can be seen. In my real scenario I had around 15 'cascading' tables so I had to track the new ids of every table not just tblCustomer
Use INSERT ... SELECT:
insert into your_table (c1, c2, ...)
select c1, c2, ...
from your_table
where c1, c2, ... are all the columns except id.
Description:
I am trying to insert user's preferences into a database. If the user hasn't yet placed any, I want a insert, otherwise, I want an update. I know I can insert default values in the creation of the user and than exclusively use update, but that adds another query (I think)
Problem:
I have read up on ON DUPLICATE KEY UPDATE but I don't understand it. This is almost the exact question I have but without the answer. The answer says:
It does sound like it will work for what you want to do as long as you hav the proper column(s) defined as UNIQUE KEY or PRIMARY KEY.
If I do a simple insert like so:
INSERT INTO table (color, size) VALUES ('blue', '18') ...
How will that ever produce at DUPLICATE KEY? As far as mysql knows it's just another insert and the id is auto-incremented. I have the primary key in the table set to unique, but the insert isn't going to check against that, right?
Table:
CREATE TABLE `firm_pref` (
`id` int(9) NOT NULL AUTO_INCREMENT,
`firm_id` int(9) NOT NULL, //user_id in this case
`header_title` varchar(99) NOT NULL,
`statement` varchar(99) NOT NULL,
`footer_content` varchar(99) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1
Well, unless you want your application to be used by a single person only, you would have to specify someone's user_id in that INSERT - when this 'someone' guy or girl updates his/her preferences, right?
This field (user_id) is exactly what would be checked by ON DUPLICATE KEY UPDATE clause.
And if you want to insert a new record, just send NULL instead:
INSERT INTO table (user_id, color, size) VALUES (NULL, 'blue', 18);
... so auto-increment will have a chance to move on and save the day. )
UPDATE: Take note that to understand that some field should be considered a unique identifier, you should mark it as such. Usually it's done naturally, as this field is used as a PRIMARY KEY. But sometimes it's not enough; it means some work for UNIQUE constraint. For example, in your table it can be used like this:
CREATE TABLE `prefs` (
`id` int(9) NOT NULL AUTO_INCREMENT,
`firm_id` int(9) NOT NULL,
...
PRIMARY KEY (`id`),
UNIQUE KEY (`firm_id`)
);
(or you can add this constraint to the existing table with ALTER TABLE prefs ADD UNIQUE (firm_id) command)
Then insert/update query will look like...
INSERT INTO prefs(firm_id, header_title, statement, footer_content)
VALUES(17, 'blue', '18', 'some_footer')
ON DUPLICATE KEY UPDATE
header_title = 'blue',
statement = '18',
footer_content = 'some_footer';
I've built a sort of demo in SQL Fiddle. You can play with it some more to better understand that concept. )
For options, you would normally have an options table that has a list of available options (like color, size etc), and then a table that spans both your options table and users table with the users' values.
For example, your options table:
id | name
=========
1 | color
2 | size
Your users table:
id | name
================
1 | Martin Bean
And an options_users join table:
option_id | user_id | value
===========================
1 | 1 | Blue
2 | 1 | Large
With the correct foreign keys set up in your options_users table, you can have redundant values removed when an option or user is removed from your system. Also, when saving a user's preferences, you can first delete their previous answers and insert the new ones.
DELETE FROM `options_users`
WHERE `user_id` = #user_id;
INSERT INTO `options_users` (`option_id`, `user_id`, `value`)
VALUES (1, #user_id, 'Blue'), (2, #user_id, 'Large');
Hope that helps.