How I can perform a mass update for cache_attribute :cached_comment_infos stored on my Posts
It would be something like this:
Post.joins(:comments).where(id: post_ids).update_all(
cached_comment_infos: self.comments.pluck(:author_name, :reference).map { |v| v.join("/") }.join(", ")
)
The output expected for each Post should be something like this : "John Doe/3242, Tom Jed/6264"
I don't think you want update_all. Update_all will update all records to one value.
https://apidock.com/rails/v4.0.2/ActiveRecord/Relation/update_all
Example given in the docs is setting all Customer's wants_email to true
Customer.update_all wants_email: true
You can loop over the posts and update each post in a unique sql statement
Post.joins(:comments).where(id: post_ids).each do |post|
post.cached_comment_infos = post.comments.pluck(:author_name, :reference).map { |v| v.join("/") }.join(", ")
post.save
end
If you really want it to happen in one SQL statement you could look into the gem activerecord-import. You would need to use their 'duplicate key update' feature: https://github.com/zdennis/activerecord-import#duplicate-key-update
Related
I have a table called Coupon.
This table has a column called query which holds a string.
The query string has some logical conditions in it formatted for a where statement. For example:
coupon1.query
=> " '/hats' = :url "
coupon2.query
=> " '/pants' = :url OR '/shoes' = :url "
I want to write a stored procedure that takes as input 2 parameters: a list of Coupon ids and a variable (in this example, the current URL).
I want the procedure to look up the value of the query column from each Coupon. Then it should run that string in a where statement, plugging in my other parameter (current url), then return any Coupon ids that matches.
Here's how I would expect the procedure to behave given the two coupons above.
Example 1:
* Call procedure with ids for coupon1 and coupon2, with #url = '/hats'
* Expect coupon1 to be returned.
Example 2:
* Call procedure with ids for coupon1 and coupon2, with #url = '/pants'
* Expect coupon2 to be returned.
Example 3:
* Call procedure with ids for coupon1 and coupon2, with #url = '/shirts'
* Expect no ids returned. URL does not match '/hats' for coupon1, and doesn't match '/pants or /shoes' for coupon2.
It's easy to test these out in ActiveRecord. Here is just example 1.
#url = '/hats'
#query = coupon1.query
# "'/hats' = :url"
Coupon.where(#query, url: #url).count
=> 2
# count is non-zero number because the query matches the url parameter.
# Coupon1 passes, its id would be returned from the stored procedure.
'/hats' == '/hats'
#query = coupon2.query
# " '/pants' = :url OR '/shoes' = :url "
Coupon.where(#query, url: #url).count
=> 0
# count is 0 because the query does not match the url parameter.
# Coupon2 does not pass, its id would not be returned from the stored procedure.
'/pants' != '/hats', '/shoes' != '/hats'
You could write this as a loop (I'm in ruby on rails with activerecord) but I need something that performs better - I could potentially have lots of coupons so I can't just check each one directly with a loop. The queries contain complex AND/OR logic so I can't just compare against a list of urls either. But here's some code of a loop that is essentially what I'm trying to translate into a stored procedure.
# assume coupon1 has id 1, coupon2 has id 2
#coupons = [coupon1, coupon2]
#url = '/hats'
#coupons.map do |coupon|
if Coupon.where(coupon.query, url: #url).count > 0
coupon.id
else
nil
end
end
=> [1, nil]
Ok, I've been pondering this one.
Big picture:
A. You have a #url you want to search for to find a match among many potential Coupons
B. A coupon has a URL that might match #url
If that's the true extent of the problem, I think you've really over-complicated things.
coupon1.query
=> ["/hats"]
coupon2.query
=> ["/pants", "/shoes"]
#url = '/hats'
Coupon.where('FIND_IN_SET(:url, query) <> 0')
Or something similar, I'm not a mySQL user myself.
However, this is very possible to achieve and may even have a much better ActiveRecord way to do the query.
UPDATE
Ok, I'm missing something. I can't actually reproduce this in console.
#url = '/hats'
#query = coupon1.query
# "'/hats' = :url"
Coupon.where(#query, url: #url).count
> SELECT * FROM 'coupons' WHERE ( '/hats' = '/hats' )
As you can see from the select statement, this will always return all records. It's the same as writing SELECT * FROM 'coupons' WHERE ( true )
How are you actually performing a valid query?
Sorry to post this in my answer, I wanted good formatting.
If I've got something wrong here, maybe we need to move this to a chat room.
I think you have just enough reputation for me to invite you to a room.
UPDATE2
Since you have to compare #query to each record individually, I think you'll have to loop.
But, I don't think you need to use Coupon.where to accomplish this since you are only comparing one record at a time.
#coupons.map do |coupon|
# don't bother putting nil in the array
next unless coupon.query == #url
coupon.id
end
However, your original question was about performance when scaled, and you know you aren't going to solve that with a loop.
Maybe JSONB instead of String so that you could actually do some SQL.
But, even with JSONB, this is still complicated by wanting your conditions to be evaluated properly.
{
"url": {
"AND": ["/hats", "/shoes"],
"OR": ["/pants"]
},
"logged_in": true,
"is_gold_member": false
}
{
"logged_in": false,
"url": "/hats"
}
{
"url": {
"OR": ["/pants", "/shoes"]
}
}
Ultimately, I think what you're doing with query attributes is going to continue to be your stumbling block. It's very clever, but it's not simple.
If it were my app, I think I would go back to considering my use case and try to find a different strategy to map specific coupons to specific parameters in a more on-the-rails way.
I need to update my data iteratively.
But the following way I achieved is the way too time-consuming.
Can I update multiple records with an id-value hash?
SUBST = ''.freeze
re = /<p>|<\/p>/m
(1..1000).each do |id|
choice = QuestionChoice.find id
choice.selections.gsub!(re, SUBST)
choice.save! if choice.changed?
end
Update:
Since I found out my code could be improved by using where
Like the following
QuestionChoice.where(id: (1..1000)).each do |choice|
choice.selections.gsub!(re, SUBST)
choice.save! if choice.changed?
end
But now I still need to call save! for every record which will cost much time.
You are hitting the db 1000 times sequentially to get each record separately, try to use single query to get all records you need to update:
SUBST = ''.freeze
re = /<p>|<\/p>/m
QuestionChoice.where('id <= 1000').map do |q|
q.selections.gsub!(re, SUBST)
q.save! if q.changed?
end
I used to face this problem and I solved it. Try to the following:
MySQL 8.0+:
QuestionChoice.where(id: 1..1000).update_all("selections = REGEXP_REPLACE(selections, '<p>|<\/p>', '')")
Others:
QuestionChoice.where(id: 1..1000).update_all("selections = REPLACE(selections, '</p>', '')")
or
QuestionChoice.where(id: 1..1000).update_all %{
selections =
CASE
WHEN selections RLIKE '<p>|<\/p>'
THEN REPLACE(selections,'<p>|<\/p>', '')
END
WHERE selections RLIKE '<p>|<\/p>'
}
IMPORTANT: Try to put a few backlashes (\) to your regex pattern in the clause if needed.
I'd like to update a table with Django - something like this in raw SQL:
update tbl_name set name = 'foo' where name = 'bar'
My first result is something like this - but that's nasty, isn't it?
list = ModelClass.objects.filter(name = 'bar')
for obj in list:
obj.name = 'foo'
obj.save()
Is there a more elegant way?
Update:
Django 2.2 version now has a bulk_update.
Old answer:
Refer to the following django documentation section
Updating multiple objects at once
In short you should be able to use:
ModelClass.objects.filter(name='bar').update(name="foo")
You can also use F objects to do things like incrementing rows:
from django.db.models import F
Entry.objects.all().update(n_pingbacks=F('n_pingbacks') + 1)
See the documentation.
However, note that:
This won't use ModelClass.save method (so if you have some logic inside it won't be triggered).
No django signals will be emitted.
You can't perform an .update() on a sliced QuerySet, it must be on an original QuerySet so you'll need to lean on the .filter() and .exclude() methods.
Consider using django-bulk-update found here on GitHub.
Install: pip install django-bulk-update
Implement: (code taken directly from projects ReadMe file)
from bulk_update.helper import bulk_update
random_names = ['Walter', 'The Dude', 'Donny', 'Jesus']
people = Person.objects.all()
for person in people:
r = random.randrange(4)
person.name = random_names[r]
bulk_update(people) # updates all columns using the default db
Update: As Marc points out in the comments this is not suitable for updating thousands of rows at once. Though it is suitable for smaller batches 10's to 100's. The size of the batch that is right for you depends on your CPU and query complexity. This tool is more like a wheel barrow than a dump truck.
Django 2.2 version now has a bulk_update method (release notes).
https://docs.djangoproject.com/en/stable/ref/models/querysets/#bulk-update
Example:
# get a pk: record dictionary of existing records
updates = YourModel.objects.filter(...).in_bulk()
....
# do something with the updates dict
....
if hasattr(YourModel.objects, 'bulk_update') and updates:
# Use the new method
YourModel.objects.bulk_update(updates.values(), [list the fields to update], batch_size=100)
else:
# The old & slow way
with transaction.atomic():
for obj in updates.values():
obj.save(update_fields=[list the fields to update])
If you want to set the same value on a collection of rows, you can use the update() method combined with any query term to update all rows in one query:
some_list = ModelClass.objects.filter(some condition).values('id')
ModelClass.objects.filter(pk__in=some_list).update(foo=bar)
If you want to update a collection of rows with different values depending on some condition, you can in best case batch the updates according to values. Let's say you have 1000 rows where you want to set a column to one of X values, then you could prepare the batches beforehand and then only run X update-queries (each essentially having the form of the first example above) + the initial SELECT-query.
If every row requires a unique value there is no way to avoid one query per update. Perhaps look into other architectures like CQRS/Event sourcing if you need performance in this latter case.
Here is a useful content which i found in internet regarding the above question
https://www.sankalpjonna.com/learn-django/running-a-bulk-update-with-django
The inefficient way
model_qs= ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
obj.name = 'foo'
obj.save()
The efficient way
ModelClass.objects.filter(name = 'bar').update(name="foo") # for single value 'foo' or add loop
Using bulk_update
update_list = []
model_qs= ModelClass.objects.filter(name = 'bar')
for model_obj in model_qs:
model_obj.name = "foo" # Or what ever the value is for simplicty im providing foo only
update_list.append(model_obj)
ModelClass.objects.bulk_update(update_list,['name'])
Using an atomic transaction
from django.db import transaction
with transaction.atomic():
model_qs = ModelClass.objects.filter(name = 'bar')
for obj in model_qs:
ModelClass.objects.filter(name = 'bar').update(name="foo")
Any Up Votes ? Thanks in advance : Thank you for keep an attention ;)
To update with same value we can simply use this
ModelClass.objects.filter(name = 'bar').update(name='foo')
To update with different values
ob_list = ModelClass.objects.filter(name = 'bar')
obj_to_be_update = []
for obj in obj_list:
obj.name = "Dear "+obj.name
obj_to_be_update.append(obj)
ModelClass.objects.bulk_update(obj_to_be_update, ['name'], batch_size=1000)
It won't trigger save signal every time instead we keep all the objects to be updated on the list and trigger update signal at once.
IT returns number of objects are updated in table.
update_counts = ModelClass.objects.filter(name='bar').update(name="foo")
You can refer this link to get more information on bulk update and create.
Bulk update and Create
This is somewhat related to this MySQL to update an XML attribute but this time I want to update the node value. I have the following XMLfragment which is in marcxml column:
<leader>00227nz a2200109n 4500</leader>
<controlfield tag="001">1</controlfield>
...
<controlfield tag="005">20091210091717.0</controlfield>
...
I want to update the controlfield value tag 001 such that it becomes a number based on a query. So like this:
<leader>00227nz a2200109n 4500</leader>
<controlfield tag="001">10</controlfield>
...
<controlfield tag="005">20091210091717.0</controlfield>
...
I have initially the following mysql query:
UPDATE auth_header SET marcxml = UpdateXML(marcxml, '//controlfield[#tag="001"]', CONCAT('<controlfield tag="001">', '10', '</controlfield>')) WHERE Extractvalue(marcxml, '//controlfield[#tag="001"]') ='169625';
The table is auth_header and it has authid as primary key (but I guess this does no matter) and it has marcxml column where the xml is stored. The query gives me '0 rows affected.' so it seems it does not work.
Thanks in advance and cheers!
Looking at the discussions here MySQL to update an XML attribute and mysql site https://dev.mysql.com/doc/refman/5.1/en/xml-functions.html#function_updatexml, the query:
UpdateXML(xml_target, xpath_expr, new_xml)
should do the trick.
The xml_target is marcxml in the question's case. The xpath_expr is '//controlfield[#tag="001"]' which is the node that needs editing. The new_xml is to concat , the digit desired, and the closing statement . And lastly, the where expression is also the same with xpath expression above.
Hence:
UPDATE auth_header SET marcxml = UpdateXML(marcxml, '//controlfield[#tag="001"]', CONCAT('<controlfield tag="001">', '10', '</controlfield>' )) WHERE Extractvalue(marcxml, '//controlfield[#tag="001"]') ='169625';
I have a User model that has all the queried fields with the existing data in the database. When I execute the following query-
#user = User.find(4, :select => 'user_fname, user_lname')
Rails throws the following error for the above line
Couldn't find all Users with 'user_id': (4, {:select=>"user_fname, user_lname"}) (found 1 results, but was looking for 2)
What's going wrong?
You can try this. I hope this will help.
#user = User.where("id = ?", 4).select( "user_fname, user_lname")
Rails 4 : use pluck as a shortcut to select one or more attributes without loading a bunch of records just to grab the attributes you want.
Try:
> User.where(id: 4).pluck(:user_fname , :user_lname).first
#=> ["John", "Smith"] # this is just sample of output
You are using #find incorrectly. It takes IDs as arguments, not SQL. It's trying to use that second argument as an ID, which clearly won't work.
http://api.rubyonrails.org/classes/ActiveRecord/FinderMethods.html#method-i-find
find only finds records by id. You can pass in an array of id's but it doesn't take any options. So it thinks the hash you are passing is an id, and it bombs because it can't find a record with that ID. I think what you want is something like:
#user = User.find(4)
fname = #user.fname
lname = #user.lname