Does GradientDescentTrainer update model at the end of each epoch ? or does it update once after a forward pass over a batch?
The GradientDescentTrainer updates the model every batch after the "backwards pass" (assuming you're not using gradient accumulation).
Related
I have a model in Django that holds some data which is irrelevant after a month.
Is there a way to automatically delete it after a certain period of time?
The DB is MySQL if it matters, thing is I can't tell whether this is done in the DB side (perhaps there's a way to configure this via MySQL?), or in my back-end code.
Is there a quick solution, or do I have to write code that does this, and have it run every day, deleting anything that wasn't added a month ago?
Thanks
I'd suggest creating a management command that queries for all the records in your model that are older than one month and delete those records. Then throw that management command into a daily cronjob. This should suit your needs.
you can solve this issue depends on your case,
if this data become with no value and you want to delete it
you can do that by
1- from database & using crontab
DELETE FROM mytable
WHERE date_field < UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 30 DAY));
2- using managment command with crontab
import datetime
samples = Sample.objects.filter(sampledate__gt=datetime.date(2011, 1,
1), sampledate__lt=datetime.date(2011, 1, 31))
3- using celery with periodic task
http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html
You can always let the manager filter for you:
class RecentManager(models.Manager):
def get_queryset(self):
return super(
RecentManager,
self
).get_queryset().filter(
your_timestamp__gt=datetime.datetime.now()-datetime.timedelta(30)
)
class YourModel(models.Model):
#your fields, including your_timestamp
objects = RecentManager()
unrestricted = models.Manager()
#static
def delete_old():
YourModel.unrestricted.filter(
your_timestamp__lt=datetime.datetime.now()-datetime.timedelta(30)
).delete()
Hook up the delete to a management command which you can run in a cronjob or Celery task or whichever other infrastructure you have handy for async execution.
I have a django app that is evolving. The model often changes and I use Django South to apply schema migrations.
Sometimes my changes involve populating new values that are added based on sql logic.
For example, added a new boolean flag for currently paying users. I have added the field, applied the migration but now I want to populate the field based on the data from other table to show who is paying.
I know I can do this with a simple sql statement, but my environment is automated and uses CI. I want to push changes and have the flag populated automatically.
How can I accomplish this? With South? With Django?
There is a thing called data migration, this is a perfect use case for it:
Data migrations are used to change the data stored in your database to
match a new schema, or feature.
from south.v2 import DataMigration
from django.conf import settings
class Migration(DataMigration):
def forwards(self, orm):
# update your user's boolean flag here
See an example of a data migration here.
Or, alternatively, you can open your schema migration .py file and populate your field in forwards() method, like this:
class Migration(SchemaMigration):
def forwards(self, orm):
# Adding field 'User.paying'
db.add_column(u'user', 'paying',
self.gf('django.db.models.fields.BooleanField')(default=True),
keep_default=False)
# update your user's boolean flag here
def backwards(self, orm):
# Deleting field 'User.paying'
db.delete_column(u'user', 'paying')
You can add your code in migration script created by south.
If you have updated a model and done schemamigration with south, it will create a script to apply that migration. It will be in appname/migration/00N_some_name.py.
You can add your code in forwards() method in that script at the end after schema alteration is done.
First, I will try to describe what I am willing to do and then, I will ask my questions.
I need to do the following:
List all rows corresponding to some conditions
Do some tests (e.g: check if it wasn't already inserted), if test passes then insert row into another database
Delete row (whether it passed tests or not)
The following is my implementation
List<MyObject> toAdd = new ArrayList<MyObject>();
for(MyObject obj:list){
if(!notYetInserted(obj){
toAdd.add(obj);
}
}
myObjectDAO.insertList(toAdd);
myObjectDAO.deleteList(list);
The service method is marked transactional.
In my DAO methods for deleteList and insertList are pretty similar so I will just put here method for insert.
public void insertList(final List<MyObject> list){
String sql = "INSERT INTO table_test " +
"(col_id, col2, col3,col4) VALUES (?, ?, ?,?)";
List<Object[]> paramList = new ArrayList<Object[]>();
for (MyObject myObject : list) {
paramList.add(new Object[] {myObject.getColId(),
myObject.getCol2(), myObject .getCol3(), myObject.getCol4()}
);
}
simpleJdbcTemplate.batchUpdate(sql, paramList);
}
I am not sure about the best way to perform such operations, I read here that calling for update inside a loop may slow down the system (especially in my case, I will have about 100K insert/delete at a time). I wonder if these additional loops inside DAO won't slow down my system even more and what would happen if problem happened repeatedly while processing that batch (I thought also about moving test from service to DAO to have only one loop and an additional test, I don't really know if it's a good idea). So, I would like your advices. Thanks a lot.
PS: if you need more details feel free to ask!
This is not necessarily a bad approach, but you are right, it might be really slow. If I were to do a process like this that inserted or deleted this many rows I would probably do it all in a stored procedure. My code would just execute the proc and the proc would handle the list and the enumeration through it (as well as the inserts and deletes).
So I already have a database setup with a few columns and a few rows already inserted in. I'm trying to create a view that you would just input information into a form and press Submit, then a row would be added to the MySQL database with the information you just typed in.
I believe you can do this with admin, but I would like to try without admin and I'm not sure if this is possible? I've been using the MySQL commandline to add rows as of now..
Of coures this is possible this is a building block for data driven websites. You can use a ModelForm as Daniel suggested (they offer built in validation and HTML markup for FREE) to easily map your model to a front end form. It would probably be beneficial to start with django tutorial or documentation first.
At the the very basic, all you have to do is instantiate your model
new_entry = YourModel(name='me', age='222', about='stackoverflow')
then save it
new_entry.save()
This adds it as a new row to your db.
https://docs.djangoproject.com/en/dev/topics/db/models/
Why would it not be possible?
You probably want a modelform (but see the general form introduction first).
Try out this example of Generic Views: http://postneo.com/2005/08/17/django-generic-views-crud (assumes a model named Task)
With Generic Views you get Insert, Update and Delete for free without any real work. give it a try and let me know what you think.
from django.conf.urls.defaults import *
info_dict = {
'app_label': 'tasks',
'module_name': 'tasks',
}
urlpatterns = patterns('',
(r'^tasks/create/?$', 'django.views.generic.create_update.create_object', info_dict ),
(r'^tasks/update/(?P<object_id>\d+)/?$', 'django.views.generic.create_update.update_object', info_dict),
(r'^tasks/delete/(?P<object_id>\d+)/?$', 'django.views.generic.create_update.delete_object', info_dict ),
)
Django Docs: https://docs.djangoproject.com/en/1.2/ref/generic-views/#create-update-delete-generic-views
In LINQ-to-SQL if I update an object in the context but haven't called SubmitChanges, is there a way to "undo" or abandon that update so that the changes won't get submitted when I eventually call SubmitChanges?
For example, if I've updated several objects and then decide I want to abandon the changes to one of them before submitting.
Part 2: same question for Entity Framework, v3.5
Both LINQ to SQL and Entity Framework will use the same call (assuming you still have the active Context):
_dbContext.Refresh(RefreshMode.OverwriteCurrentValues, yourObj);
A more appropriate way would be to treat the Context as a Unit of Work, in which case you would no longer have an active context when refreshing the object. You would simply dispose of the object you're using currently and get a fresh copy from a new context.
I think you can use the .GetOriginalEntityState(yourEntity) to retrieve the original values. Then set your updated entity back to the original
dim db as new yourDataContext
//get entity
dim e1 as yourEntity = (from x in db.table1).take(1)
//update entity
e1.someProperty = 'New Value'
//get original entity
dim originalEntity = db.table1.getOrignalEntityState(e1)
e1 = originalEntity
db.submitChanges()
Very pseudo-code but I think it conveys the right idea. Using this method, you could also just undo one or more property changes without refreshing the entire entity.