I'm trying to create a sqlalchemy engine using an existing psycopg2 connection.
My first approach is to try and extract a database URI from the psycopg2 connection, but this is failing due to security:
Calling connection.dsn prints all that's necessary to generate your connection string, but the password is obfuscated:
print(connection.dsn)
user=my_user password=xxx dbname=my_db host=localhost port=5432
similarly, calling connection.get_dsn_parameters() returns a dictionary without a password.
print(connection.get_dsn_parameters())
{'user': 'my_user', 'dbname': 'my_db', 'host': 'localhost', 'port': '5432', 'tty': '', 'options': '', 'sslmode': 'prefer', 'sslcompression': '0', 'krbsrvname': '***', 'target_session_attrs': 'any'}
I also see that if I had a connection pool, I could follow the approach from this SO answer, but I only have the connection.
I thought of faking a creator function, but I'm pretty sure this could backfire if sqlalchemy closes my one connection internally:
# connection exists in the scope
def get_conn():
return connection
create_engine('postgresql+psycopg2://', creator=get_conn)
Any ideas?
Related
I'm having issues connecting my Django app in my local machine to MySql Database in Azure? I added my IP in the Rules and am connecting with this:
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': '<servername>.database.windows.net',
'PORT': '3306',
'NAME': '<database_name>',
'USER': '<admin>#<servername>',
'PASSWORD': '<cecret>',
'OPTIONS': {'ssl': {'pem': 'tls.pem'} }
},
I can connect using AzureDataStudio, but not with this configuration in django. I Nmaped my host, found a bunch of open ports but 3306 and 1433 are bound to Sql servers.
Django's runserver shows MySQLdb._exceptions.OperationalError: (2002, "Can't connect to server on '<servername>.database.windows.net' (115)") with this configuration even if I have that server and database within it running.
One example php query string in Azure portal has:
$conn = new PDO("sqlsrv:server = tcp:<server_name>.database.windows.net,1433; Database = <database_name>", "<admin>", "{your_password_here}");
So, I'm assuming I should connect to 1433 but only 3306 works from DataStudio. From python manage.py runserver it shows django.db.utils.OperationalError: (2013, "Lost connection to server at 'handshake: reading initial communication packet', system error: 104") if I try port 1433. I'm at the limit of my knowledge regarding this.
Correction-1: 3306 doesn't seem to work with Azure DataStudio. But using 1433 in Django settings won't even initialize connection.
You need to install the MySQL connector in your local machine to connect the Python with Azure Database for MySQL using below command.
pip install mysql-connector-python
Later, on server's Overview page, make a note of the Server name and Server admin login name.
Add the code example to the file. In the code, replace the <mydemoserver>, <myadmin>, <mypassword>, and <mydatabase> placeholders with the values for your MySQL server and database.
Note: SSL is enabled by default on Azure Database for MySQL servers. You may need to download the DigiCertGlobalRootG2 SSL certificate to connect from your local environment. Replace the ssl_ca value in the code with path to this file on your computer.
import mysql.connector
from mysql.connector import errorcode
# Obtain connection string information from the portal
config = {
'host':'<mydemoserver>.mysql.database.azure.com',
'user':'<myadmin>#<mydemoserver>',
'password':'<mypassword>',
'database':'<mydatabase>',
'client_flags': [mysql.connector.ClientFlag.SSL],
'ssl_ca': '<path-to-SSL-cert>/DigiCertGlobalRootG2.crt.pem'
}
# Construct connection string
try:
conn = mysql.connector.connect(**config)
print("Connection established")
except mysql.connector.Error as err:
if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
print("Something is wrong with the user name or password")
elif err.errno == errorcode.ER_BAD_DB_ERROR:
print("Database does not exist")
else:
print(err)
else:
cursor = conn.cursor()
# Drop previous table of same name if one exists
cursor.execute("DROP TABLE IF EXISTS inventory;")
print("Finished dropping table (if existed).")
# Create table
cursor.execute("CREATE TABLE inventory (id serial PRIMARY KEY, name VARCHAR(50), quantity INTEGER);")
print("Finished creating table.")
# Insert some data into table
cursor.execute("INSERT INTO inventory (name, quantity) VALUES (%s, %s);", ("banana", 150))
print("Inserted",cursor.rowcount,"row(s) of data.")
cursor.execute("INSERT INTO inventory (name, quantity) VALUES (%s, %s);", ("orange", 154))
print("Inserted",cursor.rowcount,"row(s) of data.")
cursor.execute("INSERT INTO inventory (name, quantity) VALUES (%s, %s);", ("apple", 100))
print("Inserted",cursor.rowcount,"row(s) of data.")
# Cleanup
conn.commit()
cursor.close()
conn.close()
print("Done.")
Refer the steps given in this official tutorial to deploy the same.
Apparently I was using connection string and backend that didn't support the "MySql database" I was using. I'm still vague on how it worked but here it goes.
I created a new Azure Database for MySql servers and created a new Database there. Then used connection strings as provided by #UtkarshPal-MT in my original Django's DATABASE={} entry. Didn't have to do anything else. It just connected.
Note: You do compulsorily require to pass that certificate.
Edit1: If you're using this, as I did.
Then you need to use this mssql-django external backend with proper drivers. You can find details on using this backend in that PyPi page.
Okay, so I am a complete newbie and this might be a silly question but I couldn't find any answer related to this. I have created a login page using google authentication using django-allauth and it works perfectly fine but the problem it saves the user data directly into default database but I want it to be stored in my own mysql database.
I changed the default database info in the settings.py file but when I open the site to login I get an error "Social query does not exist". Is there any other changes that I need to do other than changing the default database in settings.py?
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'mysql',
'USER' : '****',
'PASSWORD' : '****',
'HOST' : 'localhost',
'PORT' : 3306
}
}
Maybe you need to run py manage.py migrate for to make the new tables on the data
RDB file of REDIS server that is imported from another server. I have installed the REDIS on my local machine and trying to import it in local REDIS server. mainly i want to store all REDIS data mysql database. Is any direct process to import DUMB.RBD file to mysql database .Is it any other process by which i import the REDIS data into CSV format.any help will be vey useful for me.
Redis is schema free key-value NOSQL database. Here is no any build in solutions for move data from Redis to SQL database or CSV file. But you can write some code to do this themself:
Use SCAN to get all keys in your Redis instance.
Get they values depends on key type (which can be determin with TYPE).
Write to SQL.
Got the Answer with Node js and Redis .
For Import i have used the following command After shut down the redis server
sudo cp /var/dump.rdb /home/etc/redis/dump.rdb
Parses the Redis data with Nodejs and connected it to mysql same time .
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'localhost',
port : '3306',
database : 'db',
user : 'root',
password : 'password',
insecureAuth : true,
multipleStatements: true
});
var post = {name: practice.name, practiceId: practice.id,type: practice.wid};
var query = connection.query('INSERT INTO facilities SET ?', post, function(err, result) { });
console.log(query.sql +';');
How would I connect to my VPS based MySQL database remotely (from a cloud based app) using the Ruby Net::SSH or Net::SSH::Gateway gems and key, not password, authentication?
And then connect to the database with Sequel or DataMapper. I'm assuming that after I manage to get the SSH connection working, I would just setup a Sequel/DM connection to 'sql_user#localhost:3306/database'.
I did locate a couple of similar question here, but they all use password authentication, not keys, and only demonstrate executing raw commands to query the database.
UPDATE: I just cannot seem to get this (Net::SSH with key manager) to work.
UPDATE2: Alright I have managed to get authorization when logging in from a computer that has authorized keys stored in the users local .ssh folder, with the following (port is my custom SQL port on the VPS):
sql_gate = Net::SSH::Gateway.new('192.xxx.xxx.xx','sqluser', port: 26000)
However, I will not be able to create a .ssh folder in the app's VM, so I need to somehow pass the path and filename (I will be creating a public key just for SQL access for specified user) as an option ... but haven't been able to figure out how.
UPDATE: Just need to figure out DataMapper access now. Current code being tested (remote_user_sql is my Ubuntu user, sql_user is the MySQL database user with localhost/127.0.0.1 privileges):
require 'net/ssh/gateway'
require 'data_mapper'
require 'dm-mysql-adapter'
class User
include DataMapp......
.
.
end
ssh_gate = Net::SSH::Gateway.new('192.n.n.n','remote_user_sql', {port: 25000, keys: ["sql_rsa"], keys_only: true})
port = ssh_gate.open('localhost',3306,3307)
child = fork do
DataMapper.setup(:default, {
adapter: 'mysql',
database: 'sql_test',
username: 'sql_user',
password: 'passwd',
host: 'localhost',
port: port})
DataMapper.auto_upgrade!
exit
end
puts "child: #{child}"
Process.wait
ssh_gate.close(port)
My solution, in two parts:
Well I have figured how to make the Net::SSH::Gateway gem using a specified keyfile, and then connect to the VPS through ssh via a port other than 22:
Part 1: Net::SSH::Gateway key authentication
First you must generate the keyfiles you want to use, copy the .pub to the remove server and append it to the ~/.ssh/authorized_keys file (cat sql_rsa.pub >> authorized_keys), and then make sure user_sql (the user I created on the VPS to be used only for this purpose) has been added to AllowUsers list in sshd_config. Make note of port used for ssh (25000 for this example) and use the following code to establish the connection:
ssh_gate = Net::SSH::Gateway.new('192.n.n.n','user_sql', {port: 25000, keys: ["sql_rsa"], keys_only: true})
That will read the keyfile sql_rsa in the same directory as script file, then create a new ssh gateway for 'user_sql'#'192.n.n.n' on port 25000.
I can successfully execute raw shell commands on the remove VPS with:
ssh_gate.exec("ls -la")
To close:
ssh_gate.shutdown!
Unfortunately I am still having problems using DataMapper (do-mysql-adapter) to use the gateway. I will update this answer if I figure that part out, but at least the first half of the problem has been solved.
These are the errors that DataMapper::Logger has reported:
When 127.0.0.1 was used:
Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) (code: 2002, sql state: HY000, query: , uri: )
When localhost was used:
Access denied for user 'user_sql'#'localhost' (using password: YES) (code: 1045, sql state: 28000, query: , uri: )
When the VPS hostname was used:
Unknown MySQL server host 'hostname' (25) (code: 2005, sql state: HY000, query: , uri: )
UPDATE (No success yet): So far the only way I can access the remote MySQL database is by using Net::SSH::Gateway to establish a gateway, and then use the .sshmethod to open a new Net::SSH connection over that gateway, like so:
ssh_gate.ssh('192.n.n.n','user_sql',{port: 25000, keys: ["sql_rsa"], keys_only: true}) do |ssh|
ssh.exec("mysql -u sql_user -p'passwd' -h localhost -P 3306 -e 'SELECT DATABASE();'")
end
In other words, I can only execute SQL commands using the mysql command line. I cannot figure out how to get Sequel or DataMapper to use the gateway to connect.
Part 2: DataMapper/Sequel/mysql2 connection through Net::SSH::Gateway
Make sure your MySQL server is bound to 127.0.0.1 in /etc/mysql/my.cnf, setup your connection - DataMapper example:
DataMapper.setup(:default, {
adapter: 'mysql',
database: 'DATABASE',
username: 'username',
password: 'passwd',
host: '127.0.0.1',
port: 3307}) # local port being forwarded via Net::SSH:Gateway
Followed by any class table definitions and DataMapper.finalize if required. Note that DataMapper doesn't actually connect to the remote MySQL server until either an auto_upgrade!, auto_migrate!, or query is executed, so no need to create the forwarded port yet.
Then create a new Net::SSH::Gateway, and then whenever you need DataMapper/Sequel to access the remote database, just open a port for the process, like so:
port = ssh_gate.open('127.0.0.1',3306,3307)
child = fork do
DataMapper.auto_upgrade! # DM call that accesses MySQL server
exit
end
Process.wait
ssh_gate.close(port)
You may want to put the Net::SSH::Gateway/.open code in a begin..ensure..end block, ensure'ing the port closure and gateway shutdown.
I had to use a fork and Process.wait to establish the connection, without it the method just hangs.
I know there are tones of questions like that but sadly none of them resolved my problems and as I am new to django I don't know what is wrong.
So I installed django and mysql successfully, then the django-mysql package. When try to import the MySQLdb in python (running in the terminal) it works.
But my question is: how do you start mysql and create a database? And how to "link" it to django?
I tried running mysql but it doesn't work. So I tried /usr/local/mysql/bin/mysql which throws an error ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2).
What do I have to do?
Check your system preferences - there should be a MySQL icon on the bottom. There you can start it.
Also I recommend to install MySQLWorkbench to manage your db
I think that #init3 already answered the question regarding how to start mysql. How do you "link" your database to a django project? Well, in your settings.py file you find something like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'my_database_name', # Or path to database file if using sqlite3.
'USER': 'my_mysql_user', # Not used with sqlite3.
'PASSWORD': 'my_mysql_password', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
}
So change NAME, USER and PASSWORD accordingly and Django will use that database to create tables and populate fields.