I am trying to switch between 2 mysql servers at runtime. I do not need to maintain both connections alive all the time.
This is what I am doing
from django.conf import settings
from django.db import connection
from django.contrib.auth.models import User
connection.close()
setattr(settings, 'DATABASE_HOST', 'mysql1.com')
list1 = User.objects.all()
connection.close()
setattr(settings, 'DATABASE_HOST', 'mysql2.com')
list2 = User.objects.all()
I have the following settings.py:
DATABASE_HOST = '' # localhost
DATABASE_NAME = test
...
The database name is the same on all servers and only the content of each tables differ.
I should get list1 != list2 as the users are different on both servers.
The issue is that I always get the list of users from the default database defined in settings.py (which is running on localhost) instead of the one from mysql 1 server and then from mysql 2 server.
Any idea what I am doing wrong here?
Laurent
My guess, from the information, would be a potential error in your set DATABASE_HOST lines (in yor pseudo code above). read: "setattr(settings..."
Other than that, I'm not sure how you've configured your database to switch based on your criteria, as you've not explained this. If you are doing it by model, it may be worth considering how Django knows this, or even using external connections (manually loading the database driver and running commands by hand prior to the render stage), and using the main.
I'd query the whole approach, but mostly because I'm not sure how you're actually differentiating the two databases, or why. Could you provide a bit more information on how you're doing this? I assume the variables you're pulling in dot-points 2 and 5 above are different. I don't need the values, I'm just making sure you've not used the old code duplication and forgotten to edit it (we've all been there).
Note: I'd post this as a comment if I could, but I think the solution may be in how you're pulling the variables. Finally, you could try adding the database name (just the server IP or whatever) to the output, if you're in 'dev'/debug (offline/non-production) mode, to check if it's actually making it to the second server.
For reference, the Django documentation explicitly states you shouldn't do this -- Altering settings at runetime.
There is a lot of talk within the Django community about the ORM supporting multiple connections/databases at once. There's a lot of good reference info out there on it. Check out this blog post: Easy Multi-Database Support for Django and this Django wiki page Multiple Database Support.
In the blog post, Eric Florenzano does something like this in his settings.py file:
DATABASES = dict(
primary = dict(
DATABASE_NAME=DATABASE_NAME,
# ...
),
secondary = dict(
DATABASE_NAME='secondary.db',
# ...
),
)
Related
Python 3.8.8 programm with Flask 2.0.1 and Flask-SQLAlchemy 2.5.1
MySql database, collation of the tables: utf8_general_ci.
I'm using two other sqlserver DB with SQLALCHEMY_BINDS. Everything runs on Windows 10.
Some chars from select queries on the MySql DB comes wrong: "situazione è decisamente migliorata"
should be: "situazione è decisamente migliorata"
This would solve the problem:
mystring.encode('cp1252').decode('utf8')
but I need a solution at program level. I tried:
appending to SQLALCHEMY_DATABASE_URI connection string:
"?charset=utf8" or "?charset=cp1215" and others
setting app.config['MYSQL_CHARSET'] and
app.config['MYSQL_DATABASE_CHARSET'] to 'utf8', 'utf8mb4', 'latin1', 'cp1252'
...
passing a parameter to SQLAlchemy like db = SQLAlchemy(use_native_unicode="utf8"), many variations here too
No attemp worked. Please I need suggestions.
Are you looking for a way to specify per database connection encoding ?
For all connections try to use
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {'encoding': 'cp1252'}
For specific connections to different DBs you can also use engine_options:
engine = create_engine('mysql://user:password#hostname/dbname',
encoding='cp1252')
Got the solution.
The problem was not a problem.
The person who build the original database (that is quite old) coded wrong some characters.
Some of my approaches and the one suggested by olegsv, worked, I checked that debugging deep down into into sqlalchemy data structures, the driver accepted the characters encoding, but the very chars in data were themself worong.
This was unespected.
Maybe I should delete the whole question.
In PyCharm I created a MySQL schema using pymysql on my computer. Now I want to use Peewee to create tables and write the SQL queries. However, I always receive an error message (see below) when trying to connect to the DB.
The user has sufficient rights to create tables in the DB schema as it works flawlessly with pymysql (creating tables as well as the schema works fine).
I looked at similar questions on Stackoverflow and couldn't find a similar problem. Moreover, this problem wasn't experienced in any of the tutorials I looked at, so I'm not entirely sure what could be the culprit causing the error. Below is a minimal working example.
from peewee import*
import peewee
user = 'root'
password = 'root'
db_name = 'peewee_demo'
# The schema with the name 'peewee_demo' exists
db = MySQLDatabase(db_name, user=user, passwd=password)
class Book(peewee.Model):
author = peewee.CharField()
title = peewee.TextField()
class Meta:
database = db
db.connect() # Code fails here
Book.create_table()
book = Book(author="me", title='Peewee is cool')
book.save()
for book in Book.filter(author="me"):
print(book.title)
I would expect the above code to connect to MySQL and then create a new table in the schema "peewee_demo". But instead, the code throws an error message when trying to connect to the DB:
/usr/bin/python3.6: Relink '/lib/x86_64-linux-gnu/libsystemd.so.0' with /lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol clock_gettime'
/usr/bin/python3.6: Relink /lib/x86_64-linux-gnu/libudev.so.1' with /lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
Do you have any ideas how to fix this issue?
Thanks in advance
As #coleifer pointed out in his comment, the error was probably related to a shared library issue in Python. After setting up a new virtual environment and installing all required packages, everything runs perfectly fine.
I just added the answer to be able to close the question. If #coleifer converts his comment into an answer, I'll delete mine and accept his.
I have the exact same database definition for multiple databases ( and database servers ). How do I tell Jooq to use the same database as the "Connection" I created to connect to the DB?
Example ( for MySQL ):
jdbc:mysql://localhost:3306/tsm - my development DB ( tsm ), used to generate code
jdbc:mysql://RemoteAmazonDBHost:3306/customer1 - one of my customers
jdbc:mysql://RemoteAmazonDBHost:3306/customer2 - Another customer
All 3 databases have the same definition, the same tables, indexes, etc. The TSM one is the standard our application uses.
Maybe I should be using DSL.using( Connection, Setting ) instead of DSL.using(Connection)? Is that what the manual is implying here?
If I only have one "Input" schema, do I have to specify it? In other words, can I do something like this:
Settings settings = new Settings()
.withRenderMapping(new RenderMapping()
.withSchemata(
new MappedSchema().withOutput(
databaseInfo.getProperties().getProperty("database.db"))));
Or do I have to do something like this:
Settings settings = new Settings()
.withRenderMapping(new RenderMapping()
.withSchemata(
new MappedSchema().withInput("TSM")
.withOutput(databaseInfo.getProperties().getProperty("database.db"))));
I'm assuming you're using code generation. In that case, it might be the easiest to not generate the schema at all but use <outputSchemaToDefault> in the code generation configuration, e.g.
<configuration>
<generator>
<database>
<inputSchema>your_codegen_input_schema_here</inputSchema>
<outputSchemaToDefault>true</outputSchemaToDefault>
</database>
</generator>
</configuration>
See the manual for details: https://www.jooq.org/doc/latest/manual/code-generation/codegen-advanced/codegen-config-database/codegen-database-catalog-and-schema-mapping/
If you want to keep your generated code with schema qualification and map things at runtime, then your second attempt seems correct. Pass this to your Configuration (i.e. the DSL.using() call):
Settings settings = new Settings()
.withRenderMapping(new RenderMapping()
.withSchemata(new MappedSchema()
.withInput("TSM")
.withOutput(databaseInfo.getProperties().getProperty("database.db"))));
More details can be found here: https://www.jooq.org/doc/latest/manual/sql-building/dsl-context/custom-settings/settings-render-mapping
I am in the process of migrating databases from sqlite to mysql. Now that I've migrated the data to mysql, I'm not able to use my sqlalchemy code (in Python3) to access it in the new mysql db. I was under the impression that sqlalchemy syntax was database agnostic (i.e. the same syntax would work for accessing sqlite and mysql), but this appears not to be the case. So my question is: Is it absolutely required to use a DBAPI in addition to Sqlalchemy to read the data? Do I have to edit all of my sqlalchemy code to now read mysql?
The documentation says: The MySQL dialect uses mysql-python as the default DBAPI. There are many MySQL DBAPIs available, including MySQL-connector-python and OurSQL, which I think means that I DO need a DBAPI.
My old code with sqlite successfully worked like this with sqlite:
engine = create_engine('sqlite:///pmids_info.db')
def connection():
conn = engine.connect()
return conn
def load_tables():
metadata = MetaData(bind=engine) #init metadata. will be empty
metadata.reflect(engine) #retrieve db info for metadata (tables, columns, types)
inputPapers = Table('inputPapers', metadata)
return inputPapers
inputPapers = load_tables()
def db_inputPapers_retrieval(user_input):
result = engine.execute("select title, author, journal, pubdate, url from inputPapers where pmid = :0", [user_input])
for row in result:
title = row['title']
author = row['author']
journal = row['journal']
pubdate = row['pubdate']
url = row['url']
apa = str(author+' ('+pubdate+'). '+title+'. '+journal+'. Retrieved from '+url)
return apa
This worked fine and dandy. So then I tried to update it to work with the mysql db like this:
engine = create_engine('mysql://snarkshark#localhost/pmids_info')
At first when I tried to run my sample code like this, it complained because I didn't have MySqlDB. Some googling around informed me that MySqlDB does NOT work for Python 3. So then I tried pip installing pymysql and changing my engine statement to
engine = create_engine('mysql+pymysql://snarkshark#localhost/pmids_info')
which also ends up giving me various syntax errors when I try to adjust things.
So what I want to know, is if there is any way I can get my current syntax to work with mysql? Since the syntax is from sqlalchemy, I thought it would work perfectly for the exact same data in mysql that was previously in sqlite. Will I have to go through and update ALL of my db functions to use the syntax of the DBAPI?
This will sound like a dumb answer, but you'll need to change all the places where you're using database-specific behavior. SQLAlchemy does not guarantee that anything you do with it is portable across all backends. It leaks some abstractions on purpose to allow you to do things that are only available on certain backends. What you're doing is like using Python because it's cross-platform, then doing a bunch of os.fork()s everywhere, and then being surprised that it doesn't work on Windows.
For your specific case, at a minimum, you need to wrap all your raw SQL in text() so that you're not affected by the supported paramstyle of the DBAPI. However, there are still subtle differences between different dialects of SQL, so you'll need to use the SQLAlchemy SQL expression language instead of raw SQL if you want portability. After all that, you'll still need to be careful not to use backend-specific features in the SQL expression language.
I stumbled upon the following:
def save_formset(self, request, form, formset, change):
instances = formset.save(commit=False)
bargain_id = 0
total_price = Decimal(0)
for instance in instances:
if isinstance(instance, BargainProduct):
total_price += instance.quantity * instance.product.price
bargain_id = instance.id
instance.save()
updateTotal = Bargain.objects.get(id=bargain_id)
updateTotal.total_price = total_price - updateTotal.discount_price
updateTotal.save()
This code is working for me on my local MySQL setup, however, on my live test enviroment running on SQLite3* I get the "Bargain matching query does not exist." error..
I am figuring this is due to a different hierarchy of saving the instances on SQLite.. however it seems they run(and should) act the same..?
*I cannot recompile MySQL with python support on my liveserver atm so thats a no go
Looking at the code, if you have no instances coming out of the formset.save(), bargain_id will be 0 when it gets down to the Bargain.objects.get(id=bargain_id) line, since it will skip over the for loop. If it is 0, I'm guessing it will fail with the error you are seeing.
You might want to check to see if the values are getting stored correctly in the database during your formset.save() and it is returning something back to instances.
This line is giving the error:
updateTotal = Bargain.objects.get(id=bargain_id)
which most probably is because of this line:
instances = formset.save(commit=False)
Did you define a save() method for the formset? Because it doesn't seen to have one built-in. You save it by accessing what formset.cleaned_data returns as the django docs say.
edit: I correct myself, it actually has a save() method based on this page.
I've been looking at this same issue. It is saving the data to the database, and the formset is filled. The problem is that the save on instances = formset.save(commit=False) doesn't return a value. When I look at the built-in save method, it should give back the saved data.
Another weird thing about this, is that it seems to work on my friends MySQL backend, but not on his SQLITE3 backend. Next to that it doesn't work on my MySQL backend.
Local loop returns these print outs (on MySQL).. on sqlite3 it fails with a does not excist on the query
('Formset: ', <django.forms.formsets.BargainProductFormFormSet object at 0x101fe3790>)
('Instances: ', [<BargainProduct: BargainProduct object>])
[18/Apr/2011 14:46:20] "POST /admin/shop/deal/add/ HTTP/1.1" 302 0