I have a project which is using Sequelize to manage a set of MySQL databases. Thus far I've been able to run simple queries to create new databases, insert parameters into a table, and select data... however, I have a very long .sql file (+1,700 lines) which when executed will set up a database with a specific schema (ie. tables, views, etc.). The problem is that I can not figure out how to execute a script like this using sequelize. I know the script works on a new database because I can execute the sql file from MySQL Workbench, however I do not know how to execute the script from javascript file using sequelize. I've searched forums but can't seem to find any resources either. Can this be done?
You can run raw query by Sequelize using sequelize.query(sql_string)
and you can use fs or fs-extra to read the sql file;
Just mind that you need to set the multiline statement option true in order to run this sql text:
var sql_string = fs.readFileSync('path to file', 'utf8');
const sequelize = new Sequelize('database', 'username', 'password', {
host: 'localhost',
dialect: /* one of 'mysql' | 'mariadb' | 'postgres' | 'mssql' */,
dialectOptions: {
multipleStatements: true
}
});
sequelize.query(sql_string);
Edit 1:
To better understanding of Sequelize class take a look at this
Related
I have a sequelize connection.
Then I create every table I need if they don't already exist. However, before I do that, I want to create the schema itself if it doesn`t exist, so I can run this script anywhere that has mysql installed and not worry. Is there a way to do so with sequelize? Or with any other tool if not.
Following this answer as proposed by Samuel G:
const mysql = require('mysql2/promise');
await mysql.createConnection({
user : user,
password : pwd
}).then((connection: Sequelize) => {
connection.query(`CREATE DATABASE IF NOT EXISTS ${name};`);
});
Problem:
I'm trying to add a new column to a table with tens of millions of rows, without blocking it.
I'm aware there are a few options here, and I've given ALTER TABLE a try with NULL values & also with a default value, but it takes a very long time (not workable) and locks the table.
So I'm trying to write a migration script that will duplicate the structure of the original table to a new table, add my new column to that new table, and then slowly migrate the old table data in to the new table.
My issue is with Knex though on the data copy.
I'm testing locally and the below query runs absolutely fine in MySQL version 5.6.34 in Sequel Pro, but I get a You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ' in Knex.
knex.schema.raw(`
SELECT #prevMaxIdMovedRecord;
SELECT #newMaxIdMovedRecord;
SELECT maxIdMoved FROM migration_status
ORDER BY maxIdMoved DESC
INTO #prevMaxIdMovedRecord;
INSERT IGNORE INTO table_copy (field1, newField1)
SELECT t.field1, 1 FROM table t
WHERE t.id BETWEEN #prevMaxIdMovedRecord AND #prevMaxIdMovedRecord + 50000;
SELECT id FROM table_copy
ORDER BY id DESC
LIMIT 1
INTO #newMaxIdMovedRecord;
INSERT INTO migration_status (maxIdMoved)
VALUES (#newMaxIdMovedRecord);
DELIMITER ;
`)
Any experts here that can assist? I can't see that Knex allows you to specify MySQL version in the Knexfile.js, and I was wondering if maybe there was a mismatch between versions on syntax. All my other commands in terms of new tables etc work fine...
Thanks
The mysql driver won't accept multiple statements in a single query by default. You can demonstrate this with a simple Node program:
const mysql = require('mysql')
connection = mysql.createConnection({
host: 'localhost',
user: 'youruser',
password: 'yourpassword',
database: 'yourdb',
multipleStatements: true
})
connection.connect()
connection.query('SELECT 1+1; SELECT 2+2;', function(err, results) {
if (err) {
throw err
}
console.log(results[0], results[1])
})
connection.end()
If you set multipleStatements to false, you should see an error. While multipleStatements could be passed to the driver via your knexfile.js,
{
client: "mysql",
connection: {
// ... other options ...
multipleStatements: true
}
}
and you might try this, I recommend you use a transaction as executing multiple statements in one query like this on a large table seems fragile at best. See docs for an introduction.
The NestJS documentation about Databases explains how to connect to MySQL using TypeORM and defining Entities.
In my case, I have to connect to an MySQL server that has more than one database and I need to execute SQL queries directly (without using the Entity layer) and fetch their results. Sometimes I also need to run cross-database queries.
How to do that using NestJS?
You can import multiple databases by given the connections different names. You can either directly pass the database configuration in separate TypeOrmModule.forRoot({...}) imports or use a ormconfig.json config file. (However, the ormconfig.json file might still not work with multiple databases, see this thread.)
TypeOrmModule.forRoot({
...defaultOptions,
name: 'personsConnection',
^^^^^^^^^^^^^^^^^^^^^^^^^^
host: 'person_db_host',
entities: [Person],
}),
TypeOrmModule.forRoot({
...defaultOptions,
name: 'albumsConnection',
^^^^^^^^^^^^^^^^^^^^^^^^^
host: 'album_db_host',
// You can also leave the entities empty
entities: [],
})
As stated by Kamil in the comments, you can inject the TypeORM connection object with #InjectConnection('albumsConnection'): Connection and then use the QueryBuilder or the method query to run raw SQL.
const rawData = await connection.query(`SELECT * FROM USERS`);
I am trying to query a comments table from mysql database by language.
Whenever I query by language to fetch chinese comments it displays encoded gibberish characters. but whenever I use python to query, it works.
Cloud Platform: Google Cloud SQL
Database location: Google Cloud SQL
Programming Language: Nodejs
Below is my code
// Require process, so we can mock environment variables
const process = require('process');
const Knex = require('knex');
const express = require('express');
const app = express();
const config = {
user: process.env.SQL_USER,
password: process.env.SQL_PASSWORD,
database: process.env.SQL_DATABASE,
socketPath: `/cloudsql/${process.env.INSTANCE_CONNECTION_NAME}`
};
var knex = Knex({
client: 'mysql',
connection: config
});
app.get('/', (req, res) => {
knex.select('post')
.from('comment')
.where({
'language': 'zh'
}).limit(1).then((rows) => {
res.send(rows);
}).catch((err) => {
res.send(err);
});
});
This is my query result:
"post": "最白痴的部长ï¼æœ€åŸºæœ¬çš„常识和逻辑都没有。真丢人ï¼"
please help.....
The text "最白痴的部长ï¼æœ€åŸºæœ¬çš„常识和逻辑都没有。真丢人ï¼" is what you get if "最白痴的部长基本的常识和逻辑都没有。真丢人" is sent encoded as UTF-8, but is then read and decoded as windows-1252 character set.
There are several different places this mis-decoding could happen:
From the client to the application writing to the database when the data was first added
Between the application and MySQL when adding the data
Across a configuration change in MySQL that wasn't applied correctly.
Between MySQL and the application reading the data.
Between the application and the end client displaying the data to you.
To investigate, I suggest being systematic. Start by accessing the data using other tools, e.g. PHPMyAdmin or the mysql command line in Cloud Shell. If you see the right data, you know the issue is (4) or (5). If the database definitly has the wrong data in it, then it's (1), (2) or (3).
The most common place for this error to happen is (5), so I'll go into that a bit more. This is because often websites set the character set to something wrong, or not at all. To fix this, we must make the character set explicit. You can do this in express.js by adding:
res.set('Content-Type', 'text/plain; charset=utf-8')
i have now new structure of my database, but i need to import the old data in the new format. For that reason i want to use the Laravel seeder, but i need somehow to connect to the old database and make select queries and to tell the seeder how to put the data in the new database.
Is that possible ?
Try:
Examples:
php artisan iseed my_table
php artisan iseed my_table,another_table
Visit: https://github.com/orangehill/iseed
Configure your laravel app to use two mysql connections (How to use multiple database in Laravel), one for the new database, the other for the old one.
I'll fake it like old and new.
In your seeds read from the old database and write into the new.
$old_user = DB::connection('old')->table('users')->get();
foreach ($old_users as $user) {
DB::connection('new')->table('users')->insert([
'name' => $user->name,
'email' => $user->email,
'password' => $user->password,
'old_id' -> $user->id
// ...
]);
}
Make sure to add messages while seeding like $this->command->info('Users table seeded'); or even a progress bar (you can access command line methods) to know at which point of the import you are.
Download package from
Git repo : https://github.com/orangehill/iseed
then update below file src/Orangehill/Iseed/IseedCommand.php
Add below code at line number 75
// update package script
if($this->argument('tables') === null){
$tables = Schema::getConnection()->getDoctrineSchemaManager()->listTableNames();
}
and update getArguments method in same file with below code
array('tables', InputArgument::OPTIONAL, 'comma separated string of table names'),
and then run php artisan iseed so it will get all the tables from your existing db and start creating seeders for all tables