How can I achieve looping through a database table's values for a specific column within a model?
Give a model Hub.rb, here is a method I have:
# Methods
def self.update_feed(feed_url)
feed = Feedzirra::Feed.fetch_and_parse(feed_url)
# Updates only the new articles from last run
updated_feed = Feedzirra::Feed.update(feed)
add_entries(updated_feed.new_entries) if updated_feed.updated?
end
I then have a table in my database called "feeds" which has a schema of: id, name, description, url.
What I want to do is loop through each row in the "url" field from the feeds table and do it in the Hub.rb model. It will be an outer wrapper of the function shown above and will pass the feeds.url field from the feeds table into the "feed_url" argument in the Hub.rb method.
Something like this:
# Methods
def self.update_feed
loop do
feed_url = LOOP THROUGH URL COLUMN DATA
feed = Feedzirra::Feed.fetch_and_parse(feed_url)
# Updates only the new articles from last run
updated_feed = Feedzirra::Feed.update(feed)
add_entries(updated_feed.new_entries) if updated_feed.updated?
end
end
This way I can just call Hub.update_feed from the terminal and it will go through all of the URLs I have listed in the feeds table.
How about:
Feed.where("url IS NOT NULL").each do |f|
Hub.update_feed(f.url)
end
Related
I want to fetch data of multiple ids which is provided from the User Interface and these ids are stored in a list. So how can I retrieve the data of these ids using Django ORM?
When I try the following approach It returned nothing
def selectassess(request):
if request.method=='POST':
assess=request.POST.getlist("assess")
quesno=request.POST.getlist("quesno")
subid=request.POST.getlist("subid")
print(quesno,subid)
print(assess)
max_id = Sub_Topics.objects.all().aggregate(max_id=Max("id"))['max_id']
print(max_id)
pk = random.sample(range(1, max_id),3)
subss=Sub_Topics.objects.raw("Select * from Sub_Topics where id=%s",(str(pk),))
context4={
'subss':subss,
}
print(pk)
return render(request,"assessment.html",context)
By applying the below-mentioned approach I can get only one id-data which is specified by typing the index value. But I want to get the data of all ids which are stored in the list so how can I get my required output by using Django ORM
def selectassess(request):
if request.method=='POST':
assess=request.POST.getlist("assess")
quesno=request.POST.getlist("quesno")
subid=request.POST.getlist("subid")
print(quesno,subid)
print("youuuuu")
print(assess)
max_id = Sub_Topics.objects.all().aggregate(max_id=Max("id"))['max_id']
print(max_id)
pk = random.sample(range(1, max_id),3)
sub = Sub_Topics.objects.filter(pk=pk[0]).values('id','subtopic')
context4={
'sub':sub,
}
print(pk)
return render(request,"assessment.html",context4)
You can use filter(pk__in=pk).
Try:
list_of_ids = random.sample(range(1, max_id),3)
sub = Sub_Topics.objects.filter(id__in=list_of_ids).values('id','subtopic')
context: azure - ADF brief process description:
Get a list of the fields defined in the first row of a .csv(blobed) file. This is the first step, detect fields
then 2nd step would be a kind of compare with actual columns of an SQL table
3rd one a stored procedure execution to make the alter table task, finishing with a (customized) table containing all fields needed to successfully load the .csv file into the SQl table.
To begin my ADF pipeline, I set up a lookup activity that "querys" the first line of my blobed file, "First row only" flag = ON.As a second pipeline activity, an "Append Variable" task, there I would like to get all .csv fields(first row) retrieved from the lookup activity, as a list.
Here is where a getting the nightmare.
As far I know, with dynamic content I can get an array with all values (w/ format like {"field1_name":"field1_value_1st_row", "field2_name":"field2_value_1st_row", etc })
with something like #activity('Lookup1').output.firstrow.
Or any array element with #activity('Lookup1').output.firstrow.<element_name>,
but I can't figure out how to get a list of all field names (keys?) of the array.
I will appreciate any advice, many thanks!
I would save the part of LookUp Activity because it seems that you are familiar with it.
You could use Azure Function HttpTrigger to get the key list of firstrow JSON object. For example your json object like this as you mentioned in your question:
{"field1_name":"field1_value_1st_row", "field2_name":"field2_value_1st_row"}
Azure Function code:
module.exports = async function (context, req) {
context.log('JavaScript HTTP trigger function processed a request.');
var array = [];
for(var key in req.body){
array.push(key);
}
context.res = {
body: {"keyValue":array}
};
};
Test Output:
Then use Azure Function Activity to get the output:
#activity('<AzureFunctionActivityName>').keyValue
Use Foreach Activity to loop the keyValue array:
#item()
Still based on the above sample input data,please refer to my sample code:
dct = {"field1_name": "field1_value_1st_row", "field2_name": "field2_value_1st_row"}
list = []
for key in dct.keys():
list.append(key)
print(list)
dicOutput = {"keys": list}
print(dicOutput)
Have you considered doing this in ADF data flow? You would map the incoming fields to a SQL dataset without a target schema. Define a new table name in the dataset definition and then map the incoming fields from your CSV to a new target table schema definition. ADF will write the rows to a new table using that file's schema.
I have written a loop to retrieve data from an API (Limesurvey) based on a list of ids and fill a row of a dataframe with the result of each loop.
I have a list with ids like this:
# list of ids
ids = ['1','427',... ,'847']
My code to query the API based on each item of the list looks as follows:
method = "get_participant_properties"
params = OrderedDict([
("sSessionKey", api.session_key),
("iSurveyID", 12345),
("aTokenQueryProperties", t),
])
# loop through API query with the 'aTokenQueryProperties' stored in the list 'tids'.
attributes = []
for t in ids:
attributes.append(api.query(method=method, params=params))
pd.DataFrame(attributes)
Unfortunately, the result is a dataframe with 158 rows, and each row is the same, i.e. the query result of the last id in my list (847).
You are not passing in the t from the loop. The t in the params definition is unrelated; if I were to run your code now I'd get a NameError exception because t hasn't been set at that point. The t expression in the params mapping is not live, it is not updated each loop iteration.
Set the 'aTokenQueryProperties' key in the loop:
method = "get_participant_properties"
params = OrderedDict([
("sSessionKey", api.session_key),
("iSurveyID", 12345),
("aTokenQueryProperties", None),
])
attributes = []
for t in ids:
params["aTokenQueryProperties"] = t
attributes.append(api.query(method=method, params=params))
Setting "aTokenQueryProperties" to None in the params OrderedDict object at the start is optional; you'd only need to do this if reserving its exact location in the params order is important and even then, because in your example it is the last element in the mapping, you'd end up with the same output anyway.
I'd like to update a table with Django - something like this in raw SQL:
update tbl_name set name = 'foo' where name = 'bar'
My first result is something like this - but that's nasty, isn't it?
list = ModelClass.objects.filter(name = 'bar')
for obj in list:
obj.name = 'foo'
obj.save()
Is there a more elegant way?
Update:
Django 2.2 version now has a bulk_update.
Old answer:
Refer to the following django documentation section
Updating multiple objects at once
In short you should be able to use:
ModelClass.objects.filter(name='bar').update(name="foo")
You can also use F objects to do things like incrementing rows:
from django.db.models import F
Entry.objects.all().update(n_pingbacks=F('n_pingbacks') + 1)
See the documentation.
However, note that:
This won't use ModelClass.save method (so if you have some logic inside it won't be triggered).
No django signals will be emitted.
You can't perform an .update() on a sliced QuerySet, it must be on an original QuerySet so you'll need to lean on the .filter() and .exclude() methods.
Consider using django-bulk-update found here on GitHub.
Install: pip install django-bulk-update
Implement: (code taken directly from projects ReadMe file)
from bulk_update.helper import bulk_update
random_names = ['Walter', 'The Dude', 'Donny', 'Jesus']
people = Person.objects.all()
for person in people:
r = random.randrange(4)
person.name = random_names[r]
bulk_update(people) # updates all columns using the default db
Update: As Marc points out in the comments this is not suitable for updating thousands of rows at once. Though it is suitable for smaller batches 10's to 100's. The size of the batch that is right for you depends on your CPU and query complexity. This tool is more like a wheel barrow than a dump truck.
Django 2.2 version now has a bulk_update method (release notes).
https://docs.djangoproject.com/en/stable/ref/models/querysets/#bulk-update
Example:
# get a pk: record dictionary of existing records
updates = YourModel.objects.filter(...).in_bulk()
....
# do something with the updates dict
....
if hasattr(YourModel.objects, 'bulk_update') and updates:
# Use the new method
YourModel.objects.bulk_update(updates.values(), [list the fields to update], batch_size=100)
else:
# The old & slow way
with transaction.atomic():
for obj in updates.values():
obj.save(update_fields=[list the fields to update])
If you want to set the same value on a collection of rows, you can use the update() method combined with any query term to update all rows in one query:
some_list = ModelClass.objects.filter(some condition).values('id')
ModelClass.objects.filter(pk__in=some_list).update(foo=bar)
If you want to update a collection of rows with different values depending on some condition, you can in best case batch the updates according to values. Let's say you have 1000 rows where you want to set a column to one of X values, then you could prepare the batches beforehand and then only run X update-queries (each essentially having the form of the first example above) + the initial SELECT-query.
If every row requires a unique value there is no way to avoid one query per update. Perhaps look into other architectures like CQRS/Event sourcing if you need performance in this latter case.
Here is a useful content which i found in internet regarding the above question
https://www.sankalpjonna.com/learn-django/running-a-bulk-update-with-django
The inefficient way
model_qs= ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
obj.name = 'foo'
obj.save()
The efficient way
ModelClass.objects.filter(name = 'bar').update(name="foo") # for single value 'foo' or add loop
Using bulk_update
update_list = []
model_qs= ModelClass.objects.filter(name = 'bar')
for model_obj in model_qs:
model_obj.name = "foo" # Or what ever the value is for simplicty im providing foo only
update_list.append(model_obj)
ModelClass.objects.bulk_update(update_list,['name'])
Using an atomic transaction
from django.db import transaction
with transaction.atomic():
model_qs = ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
ModelClass.objects.filter(name = 'bar').update(name="foo")
Any Up Votes ? Thanks in advance : Thank you for keep an attention ;)
To update with same value we can simply use this
ModelClass.objects.filter(name = 'bar').update(name='foo')
To update with different values
ob_list = ModelClass.objects.filter(name = 'bar')
obj_to_be_update = []
for obj in obj_list:
obj.name = "Dear "+obj.name
obj_to_be_update.append(obj)
ModelClass.objects.bulk_update(obj_to_be_update, ['name'], batch_size=1000)
It won't trigger save signal every time instead we keep all the objects to be updated on the list and trigger update signal at once.
IT returns number of objects are updated in table.
update_counts = ModelClass.objects.filter(name='bar').update(name="foo")
You can refer this link to get more information on bulk update and create.
Bulk update and Create
in my controller I have following sequence of commands:
SAVE DATA INTO FIRST TABLE
_get ID of inserted item into table from first step_
SAVE DATA INTO SECOND TABLE WITH ID FROM FIRST COMMAND
if FIRST.save && SECOND.save
do something
And I am wondering, how to get id of item, which is immediately inserted into database... I tried to googling, but I can't find this information...
Thanks in advance for your hints
# SAVE DATA INTO FIRST TABLE
first_instance = FirstModel.new( :foo => :bar )
first_save = first_instance.save
# _get ID of inserted item into table from first step_
first_instance_id = first_instance.id
# SAVE DATA INTO SECOND TABLE WITH ID FROM FIRST COMMAND
second_save = SecondModel.new( :first_model_id => first_instance_id ).save
if first_save && second_save
# do something
end
After saving a model, you can access it's id variable:
#user = User.new
puts #user.id
# => nil
#user.save
puts #user.id
# => 1
Could you just search your database by the updated_at field in your model?
To get the most recent record:
#model1 = Model1.order("updated_at DESC").limit(1)
or better yet, upon saving Model1 in the first place:
#model1 = model1.save
To assign:
#model2.model1_id = #model1.id
Note: if you actually want to save the ID of a specific record, finding the last isn't the best way to go.
This is because another record could be inserted by a different user, right after you inserted Model1 and right before you call Model2.
If you want the two to save together or not at all, you can look into transactions: http://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html
If you're happy with Model1 saving on its own before worrying about Model2, then simply assign the variables as I did above.