I'm building an api that receives webhook data (orders from shopify) and saves them to a relational mysql database. Each order has products in it that belong to collections (categories). I relate the product to the collection, or create a new collection if it doesn't already exist. The problem I run into is that since I can't use findOrCreate() I do a find() first, and then if nothing is found I do a create. However, since nodejs is async, if I get many orders coming in at once, sometimes the same collection isn't found for two different products at the same time, and then one will create it right before the other one tries, so when the other one tries to create it waterline throws an error:
col_id
• A record with that col_id already exists (405145674).
What is the best way to catch this error so I can just do another find() so the function returns the proper collection instead of an undefined? I could just do this:
if (err){ findmethodagain(); }
But then it would try that for any error, not a "record with that id already exists" error.
If you're not seeing this a lot, it wouldn't hurt to just do a second attempt after a delay. You can use Promise.delay from, for example, Bluebird to run a second function after some number of milliseconds on any error. If in most situations one of them creates the collection, then you'll succeed with just one retry.
If you are seeing lots of other errors, well, fix them.
However, race conditions usually indicate that you need to rework the architecture. I'd suggest instead of relying on error attributes or trying to hack around them, you redesign it, at least for the situation where you have to create a new collection. One popular option is to set up a worker queue. In your case, you would just use find in the first pass to see if a collection exists, and if not, kick it off to the queue for a single worker to pick up sequentially. In the worker job, do your find then create. You can use Redis and Bull to set this up quickly and easily.
Related
Good morning,
I am creating a new user, and it's email must be unique. I have declared it as so in the User entity.
I am wondering whether it is better (quicker/best practice) to catch a DataIntegrityViolationException when the user is being created in the DB or if it is better to check if the user exists: select count(*) from User u where u.email=? for example.
I am working in SpringBoot, using MySQL and JPA.
Thank you so much!!
The difference of performance might be very insignificant in that case.
What is the most important here, is to understand what happens by reading the code. You should have some code in your service layer that checks every rule defined is not violated before actually trying to proceeed in databaase. IT will be easier for new comers (and even for you in few weeks) to have a clear view of what is tested and how.
That part of code should then raise an exception with a dedicated code (for example, let's take the ID of the business rule defined by your business analysts) and in a properties, you should have a message corresponding to that rule (the message key could be the id of the rule for example).
Also, while inserting a row in this table, you could get DataIntegrityViolationException for different reasons. So, that is not a durable solution anyway.
I am trying to make a backup table of users, called archived users. It creates the ArchivedUser by taking a hash of the current users attributes (self) and merging in the self.id as the user_id.
When a user is reinstated, their record as an ArchivedUser still remains in the ArchivedUser table. If the user gets deleted a second time, it should update any attributes that have changed.
Currently, it throws a validation error:
Validation failed: User has already been taken, as the self.id already exists in the ArchivedUser table.
What is a better way to handle an object where you update an existing object if possible, or create a new record if it doesn't exist. I am using Rails 4 and have tried find_or_create_by but it throws an error
Mysql2::Error: Unknown column 'device_details.device_app_version'
which is odd, as that column exists in both tables and doesn't get modified.
User Delete Method
# creates ArchivedUser with the exact attributes of the User
# object and merges self.id to fill user_id on ArchivedUser
if ArchivedUser.create!(
self.attributes.merge(user_id: self.id)
)
Thanks for taking a peek!
If your archived_users table is truly acting as a backup for users and not adding any additional functionality, I would ditch the ArchiveUser model and simply add an archived boolean on the User model to tell whether or not the user is archived.
That way you don't have to deal with moving an object to another table and hooking into a delete callback.
However, if your ArchiveUser model does offer some different functionality compared to User, another option would be to use single table inheritence to differentiate the type of user. In this case, you could have User govern all users, and then distinguish between a user being, for example, an ActiveUser or an ArchivedUser.
This takes more setup and can be a bit confusing if you haven't worked with STI, but it can be useful when two similar models need to differ only slightly.
That being said, if you want to keep your current setup, I believe there are a few issues I see with your code:
If you are going to create an object from an existing object, it's good practice to duplicate the object (dup). That way the id won't be automatically set and can be auto-incremented.
If you truly have deleted the User record from the database, there's no reason to store a reference to its id because it's gone. But if you aren't actually deleting the record, you should definitely just use a boolean attribute to determine whether or not the user is active or archived.
I don't have enough context here as to why find_or_create_by isn't working, but if it were the case, then I would keep it as simple as possible. Don't use all the attributes, but just the consistent ones (like id) that you know will return the proper result.
if ArchivedUser.create! # ... is problematic. The bang after create (i.e. create!) will throw an error if the record could not be created, making the if pointless. So, either use if if you don't want errors thrown and want to handle the condition in which the record was not created. Or use create! without if if you do want to throw an error.
I like the idea of Records mainly because we can get away from using someObject.get('someKey')
But Record seems to operate more like some kind of template for records.
If you just instantiate Record each time you need a read only immutable object, simply because you like to access properties as someObject.key, would this be a bad idea?
A call to Record returns a traditional JS type (class) when you call it. Ideally you should store this type reference and create new record instances from it. Otherwise stuff like inheritance-check doesn't work as intended.
You should prefer Record over traditional JS objects or Immutable Maps because of the integrity and immutability it offers while retaining the first class member access. However don't create a Record from a Map just for the sake of syntax convenience, you should use Record in place of the said Map everywhere.
Said that, bear in mind that Record has its own baggage of issues. You can read about it in my other post here... https://stackoverflow.com/a/36357288/2790937
I have a relatively simple Django app, with quite heavy usage that is responsible for quite some concurrency in the db operations.
I have a model Post with a m2m to a Tag model.
A single line in my code, p.add(t) is repeatedly causing mysql exceptions (where p is a Post instance and t is a Tag instance.)
IntegrityError: (1062, "Duplicate entry '329051-1827414' for key 'post_id'")
When this is raised I can manually run this p.add(t) successfully, so it must have to do with some peculiar state that the db/app are in at the time of normal execution. It happens about once every 1000 tag-adding attempts, without any pattern that I can detect (i.e both numbers in the "329051-1827414" pair of the example change)
A CHECK TABLE in mysql on the relevant table shows that they are all seemingly OK.
Any ideas?
Usually you see errors like that when trying to add to an intermediate table if the row being added duplicates the unique-together constraint for the FK's. I'm guessing that in the example you provided "329051" is a Post id and "1827414" is a Tag id.
Normally in Django you can call the add() method repeatedly to add the same instance and Django takes care of everything for you. I'm assuming the model manager maintains some state to help it determine if each add() represents a new or existing row and if the row appears to be new it attempts an insert.
That in itself doesn't explain why you're getting the error. You mention "is responsible for quite some concurrency in the db operations.". Without knowing what that means, I'm guessing that you could be getting a race condition where multiple thread/processes are attempting to add the same new tag around the same time and both are attempting inserts.
I think I'm seeing a similar problem in my app - If I send two identical requests to add a m2m relation (e.g. tag in my case as well), I get that error because the m2m table has a unique constraint on (user, tag). I'm guessing the server is processing the .add functions at the same time.
if not already in database:
# Both invocations reach here because the next line takes some time to process.
create m2m row
I don't know how that can be remedied.
I've been battling this for a while. I'm trying to implement a many to one association. I have a bunch of rows in a table, called readings. These accumulate over time, and every now and then I want to export them. When I export them I want to create a new object called ExportEvent, to track which rows were exported, so they can be re-exported if need be. Therefore Reading has a nullable foreign key relationship with ExportEvent, as I create the readings before I export them.
What I'm finding is that when I then do the export, whether I first create the ExportEvent (evt) and add the readings using
evt.Readings.AddRange(),
or if I use
foreach(reading)
reading.ExportEvent = evt
When I call SubmitChanges I am always getting a new bunch of readings created with the association to evt, and the original records aren't updated.
I pared this back to its simplest though, just to see if I could create the two objects with no association, and I even found when I just retrieved all the readings and updated an int value on them, submitchanges still inserted a bunch of new records. What's going on?
Hmmm. Interesting - just clicked this link in my bookmarks, and found that the question has been resurrected, so will provide the (embarrassing) solution. All of my entities have audit data properties on them - CreatedDate and UpdatedDate. Therefore I've implemented the partial methods for the insert and update of each entity in the datacontext. I had copied and pasted (how often is this the cause of some downfall) some of these insert and update methods for the newly created entities. As a result I'd also copied an error, where the Update[blah] methods were calling ExecuteDynamicInsert, instead of ExecuteDynamicUpdate.
Suffice to say I was very frustrated when for 3 hours I'd been trying frantically to solve this problem, only to find it was due to a (silly) copy/paste error - and only to find the error about 3 mins after I'd posted this question!
Hope this helps someone.
I suspect it is because you are calling AddRange(). This will add the new objects to the data context. Instead, you should try just re attaching the existing objects by called Attach() on your data context.
(Or if you never detached them and still have your original data context, you don't need to do anything, just make the changes to the objects and call SubmitChanges())