The SQLAlchemy ORM allows for:
instance1 = Class1()
instance1.foo = 'bar'
What I want to do, however, is refer the column name 'foo' by passing it as a variable to the instance, something like:
column_name = 'foo'
instance1[column_name] = 'bar' # or
instance1.column(column_name) = 'bar' # or
instance1.Column(column_name) = 'bar'
Or, perhaps to do something like:
instance1=Class1(foo='bar') # or
instance1=Class1('foo':'bar')
or:
dict = {'foo':'bar'}
instance1 = Class1(dict) # or
instance1 = Class1(**dict) # <-- THIS *DOES* WORK
None of the above work, of course. Is there a method of referring to the column name with a variable, please?
FYI, the bigger picture version of what I'm trying to do is to collect data into a dictionary with keys that match a subset of column names in multiple tables. Then I want to dump whatever I've gathered in the dictionary into those tables:
instance1 = Class1()
instance2 = Class2()
for key in data_dictionary:
instance1[key] = data_dictionary[key]
instance2[key] = data_dictionary[key]
or simply:
instance1 = Class1(data_dictionary)
instance2 = Class2(data_dictionary)
Thanks for your help,
Andrew
Thank you, univerio - the following works:
dict = {'foo':'bar'}
instance = Class1(**dict)
As does your other suggestion:
setattr(instance, 'foo', 'bar')
Related
I'm trying to get information from Wikidata. For example, to access to "cobalt-70" I use the API.
API_ENDPOINT = "https://www.wikidata.org/w/api.php"
query = "cobalt-70"
params = {
'action': 'wbsearchentities',
'format': 'json',
'language': 'en',
'search': query
}
r = requests.get(API_ENDPOINT, params = params)
print(r.json())
So there is a "claims" which gives access to the statements. Is there a best way to check if a value exists in the statement? For example, "cobalt-70" have the value 0.5 inside the property P2114. So how can I check if a value exists in the statement of the entity? As this example.
Is there an approach to access it. Thank you!
I'm not sure this is exactly what you are looking for, but if it's close enough, you can probably modify it as necessary:
import requests
import json
url = 'https://www.wikidata.org/wiki/Special:EntityData/Q18844865.json'
req = requests.get(url)
targets = j_dat['entities']['Q18844865']['claims']['P2114']
for target in targets:
values = target['mainsnak']['datavalue']['value'].items()
for value in values:
print(value[0],value[1])
Output:
amount +0.5
unit http://www.wikidata.org/entity/Q11574
upperBound +0.6799999999999999
lowerBound +0.32
amount +108.0
unit http://www.wikidata.org/entity/Q723733
upperBound +115.0
lowerBound +101.0
EDIT:
To find property id by value, try:
targets = j_dat['entities']['Q18844865']['claims'].items()
for target in targets:
line = target[1][0]['mainsnak']['datavalue']['value']
if isinstance(line,dict):
for v in line.values():
if v == "+0.5":
print('property: ',target[0])
Output:
property: P2114
I try a solution which consists to search inside the json object as the solution proposed here : https://stackoverflow.com/a/55549654/8374738. I hope it can help. Let's give you the idea.
import pprint
def search(d, search_pattern, prev_datapoint_path=''):
output = []
current_datapoint = d
current_datapoint_path = prev_datapoint_path
if type(current_datapoint) is dict:
for dkey in current_datapoint:
if search_pattern in str(dkey):
c = current_datapoint_path
c+="['"+dkey+"']"
output.append(c)
c = current_datapoint_path
c+="['"+dkey+"']"
for i in search(current_datapoint[dkey], search_pattern, c):
output.append(i)
elif type(current_datapoint) is list:
for i in range(0, len(current_datapoint)):
if search_pattern in str(i):
c = current_datapoint_path
c += "[" + str(i) + "]"
output.append(i)
c = current_datapoint_path
c+="["+ str(i) +"]"
for i in search(current_datapoint[i], search_pattern, c):
output.append(i)
elif search_pattern in str(current_datapoint):
c = current_datapoint_path
output.append(c)
output = filter(None, output)
return list(output)
And you just need to use:
pprint.pprint(search(res.json(),'0.5','res.json()'))
Output:
["res.json()['claims']['P2114'][0]['mainsnak']['datavalue']['value']['amount']"]
I'm trying to optimize a migration, it's taking too long, about 15 minutes every time you try to run it, because there is a lot of data on this table. It's an old database that have dates like this '14102019' (%d%m%Y) as String, and need to convert them to DateField. I created a DateField for both.
Database is MySQL.
dtobito and dtnasc are the old strings that need to be converted
data_obito and data_nasc are the new DateFields
What works (very slowly):
def date_to_datefield(apps, schema_editor):
Obitos = apps.get_model('core', 'Obitos')
for obito in Obitos.objects.all():
if obito.dtnasc and obito.dtnasc != '':
obito.data_nasc = datetime.strptime(obito.dtnasc, '%d%m%Y')
if obito.dtobito and obito.dtobito != '':
obito.data_obito = datetime.strptime(obito.dtobito, '%d%m%Y')
obito.save()
What doesn't work:
Obitos.objects.update(
data_nasc=datetime.strptime(F('dtnasc'), '%d%m%Y'),
data_obito=datetime.strptime(F('dtobito'), '%d%m%Y')
)
What could work, but I don't know how:
Obitos.objects.raw("""
UPDATE obitos new,
(
SELECT
STR_TO_DATE(dtnasc, '%d%m%Y') AS dtnasc,
STR_TO_DATE(dtobito, '%d%m%Y') AS dtobito,
FROM obitos
) old
SET new.data_nasc = old.dtnasc
SET new.data_obtio = old.dtobito
""")
Try using bulk_update:
def date_to_datefield(apps, schema_editor):
Obitos = apps.get_model('core', 'Obitos')
objs = []
def to_date_object(date_string):
return datetime.strptime(date_string, '%d%m%Y')
for obito in Obitos.objects.all():
updated = False
if obito.dtnasc:
obito.data_nasc = to_date_object(obito.dtnasc)
updated = True
if obito.dtobito:
obito.data_obito = to_date_object(obito.dtobito)
updated = True
if updated:
objs.append(obito)
if objs:
Obitos.objects.bulk_update(objs, ['data_nasc', 'data_obito'])
I have three DynamoDB tables. Two tables have instance IDs that are part of an application and the other is a master table of all instances across all of my accounts and the tag metadata. I have two scans for the two tables to get the instance IDs and then query the master table for the tag metadata. However, when I try writing this to the CSV file, I want to have two separate header sections for each dynamo table's unique output. Once the first iteration is done, the second file write writes to the last row where the first iteration left off instead of starting over at the top in the second header section. Below is my code and an output example to make it clear.
CODE:
import boto3
import csv
import json
from boto3.dynamodb.conditions import Key, Attr
dynamo = boto3.client('dynamodb')
dynamodb = boto3.resource('dynamodb')
s3 = boto3.resource('s3')
# Required resource and client calls
all_instances_table = dynamodb.Table('Master')
missing_response = dynamo.scan(TableName='T1')
installed_response = dynamo.scan(TableName='T2')
# Creates CSV DictWriter object and fieldnames
with open('file.csv', 'w') as csvfile:
fieldnames = ['Agent Not Installed', 'Not Installed Account', 'Not Installed Tags', 'Not Installed Environment', " ", 'Agent Installed', 'Installed Account', 'Installed Tags', 'Installed Environment']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
# Find instances IDs from the missing table in the master table to pull tag metadata
for instances in missing_response['Items']:
instance_missing = instances['missing_instances']['S']
#print("Missing:" + instance_missing)
query_missing = all_instances_table.query(KeyConditionExpression=Key('ID').eq(instance_missing))
for item_missing in query_missing['Items']:
missing_id = item_missing['ID']
missing_account = item_missing['Account']
missing_tags = item_missing['Tags']
missing_env = item_missing['Environment']
# Write the data to the CSV file
writer.writerow({'Agent Not Installed': missing_id, 'Not Installed Account': missing_account, 'Not Installed Tags': missing_tags, 'Not Installed Environment': missing_env})
# Find instances IDs from the installed table in the master table to pull tag metadata
for instances in installed_response['Items']:
instance_installed = instances['installed_instances']['S']
#print("Installed:" + instance_installed)
query_installed = all_instances_table.query(KeyConditionExpression=Key('ID').eq(instance_installed))
for item_installed in query_installed['Items']:
installed_id = item_installed['ID']
print(installed_id)
installed_account = item_installed['Account']
installed_tags = item_installed['Tags']
installed_env = item_installed['Environment']
# Write the data to the CSV file
writer.writerow({'Agent Installed': installed_id, 'Installed Account': installed_account, 'Installed Tags': installed_tags, 'Installed Environment': installed_env})
OUTPUT:
This is what the columns/rows look like in the file.
I need all of the output to be on the same line for each header section.
DATA:
Here is a sample of what both tables look like.
SAMPLE OUTPUT:
Here is what the for loops print out and appends to the lists.
Missing:
i-0xxxxxx 333333333 foo#bar.com int
i-0yyyyyy 333333333 foo1#bar.com int
Installed:
i-0zzzzzz 44444444 foo2#bar.com int
i-0aaaaaa 44444444 foo3#bar.com int
You want to collect related rows together into a single list to write on a single row, something like:
missing = [] # collection for missing_responses
installed = [] # collection for installed_responses
# Find instances IDs from the missing table in the master table to pull tag metadata
for instances in missing_response['Items']:
instance_missing = instances['missing_instances']['S']
#print("Missing:" + instance_missing)
query_missing = all_instances_table.query(KeyConditionExpression=Key('ID').eq(instance_missing))
for item_missing in query_missing['Items']:
missing_id = item_missing['ID']
missing_account = item_missing['Account']
missing_tags = item_missing['Tags']
missing_env = item_missing['Environment']
# Update first half of row with missing list
missing.append(missing_id, missing_account, missing_tags, missing_env)
# Find instances IDs from the installed table in the master table to pull tag metadata
for instances in installed_response['Items']:
instance_installed = instances['installed_instances']['S']
#print("Installed:" + instance_installed)
query_installed = all_instances_table.query(KeyConditionExpression=Key('ID').eq(instance_installed))
for item_installed in query_installed['Items']:
installed_id = item_installed['ID']
print(installed_id)
installed_account = item_installed['Account']
installed_tags = item_installed['Tags']
installed_env = item_installed['Environment']
# update second half of row by updating installed list
installed.append(installed_id, installed_account, installed_tags, installed_env)
# combine your two lists outside a loop
this_row = []
i = 0;
for m in missing:
# iterate through the first half to concatenate with the second half
this_row.append( m + installed[i] )
i = i +1
# adding an empty column after the write operation, manually, is optional
# Write the data to the CSV file
writer.writerow(this_row)
This will work if your installed and missing tables operate on a relatable field - like a timestamp or an account ID, something that you can ensure keeps the rows being concatenated in the same order. A data sample would be useful to really answer the question.
Odoo 9 custom module binary field attachment=True parameter added later after that new record will be stored in filesystem storage.
Binary Fields some old records attachment = True not used, so old record entry not created in ir.attachment table and filesystem not saved.
I would like to know how to migrate old records binary field value store in filesystem storage?. How to create/insert records in ir_attachment row based on old records binary field value? Is any script available?
You have to include the postgre bin path in pg_path in your configuration file. This will restore the file store that contains the binary fields
pg_path = D:\fx\upsynth_Postgres\bin
I'm sure that you no longer need a solution to this as you asked 18 months ago, but I have just had the same issue (many gigabytes of binary data in the database) and this question came up on Google so I thought I would share my solution.
When you set attachment=True the binary column will remain in the database, but the system will look in the filestore instead for the data. This left me unable to access the data from the Odoo API so I needed to retrieve the binary data from the database directly, then re-write the binary data to the record using Odoo and then finally drop the column and vacuum the table.
Here is my script, which is inspired by this solution for migrating attachments, but this solution will work for any field in any model and reads the binary data from the database rather than from the Odoo API.
import xmlrpclib
import psycopg2
username = 'your_odoo_username'
pwd = 'your_odoo_password'
url = 'http://ip-address:8069'
dbname = 'database-name'
model = 'model.name'
field = 'field_name'
dbuser = 'postgres_user'
dbpwd = 'postgres_password'
dbhost = 'postgres_host'
conn = psycopg2.connect(database=dbname, user=dbuser, password=dbpwd, host=dbhost, port='5432')
cr = conn.cursor()
# Get the uid
sock_common = xmlrpclib.ServerProxy ('%s/xmlrpc/common' % url)
uid = sock_common.login(dbname, username, pwd)
sock = xmlrpclib.ServerProxy('%s/xmlrpc/object' % url)
def migrate_attachment(res_id):
# 1. get data
cr.execute("SELECT %s from %s where id=%s" % (field, model.replace('.', '_'), res_id))
data = cr.fetchall()[0][0]
# Re-Write attachment
if data:
data = str(data)
sock.execute(dbname, uid, pwd, model, 'write', [res_id], {field: str(data)})
return True
else:
return False
# SELECT attachments:
records = sock.execute(dbname, uid, pwd, model, 'search', [])
cnt = len(records)
print cnt
i = 0
for res_id in records:
att = sock.execute(dbname, uid, pwd, model, 'read', res_id, [field])
status = migrate_attachment(res_id)
print 'Migrated ID %s (attachment %s of %s) [Contained data: %s]' % (res_id, i, cnt, status)
i += 1
cr.close()
print "done ..."
Afterwards, drop the column and vacuum the table in psql.
I want to get a row from mysql database, then modiy some fields and commit. Before I commit, updating this row shoud be forbidden. so I use with_lockmode('update'). I use the
following RoutingSession and I find it use Slave to do this query. But I want to do this query using Master, what should I do in get_bind???
class RoutingSession(Session):
def get_bind(self, mapper=None, clause=None):
if self._flushing:
return engines['master']
else:
return engines['slave']
row = session.query(SomeTable).filter(SomeTable.id = 1).with_lockmode('update').one
row.somefield = 'newcontent'
session.commit()
I'm assuming you got this recipe from my blog post at Django-style Database Routers in SQLAlchemy. If you look in the section "manual access" you'll see a recipe for when you need to explicitly point the routing session to a certain node:
class RoutingSession(Session):
def get_bind(self, mapper=None, clause=None ):
if self._name:
return engines[self._name]
elif mapper and issubclass(mapper.class_, OtherBase):
return engines['other']
elif self._flushing:
return engines['master']
else:
return engines[
random.choice(['slave1','slave2'])
]
_name = None
def using_bind(self, name):
s = RoutingSession()
vars(s).update(vars(self))
s._name = name
return s
Then you just do your operation with that bind explicitly:
m1 = Session().using_bind("master").query(Model1).first()