I'm working on an application that allows users to edit documents (spreadsheets and other docs) live.
When editing the files, a keyup function is triggered with AJAX which auto-sends a post request to save/update the file.
What I'd like to do, is add a notifications or someway of registering/logging that the user has updated the file. That could then get put into some-sort of a feed.
The problem is, that because there are a so many AJAX requests it would be impractical to log the edit based on AJAX save requests.
What would be a good structure to handle this?
I was thinking of using some sort of time-stamping method, and only log an edits if the previous time-stamp is out of a certain range (like 15mins or something).
Does anyone have any experience with this kind of thing? I'm really not sure what the best solution would be. I'm Trying to come up with few ideas to see what a suitable solution would be to this (in terms of table(s) structure and general direction). Perhaps someone here can help.
If you have so many AJAX request, is going to be quite hard if you have to read on the server every time that you send an AJAX request if there is a log row for your user in the period of time you want, as you have to make a read action over a time field, and then a write action, that is, you'll be all the time reading a table that will have millions of rows. That can hit your performance and the speed of response, and load unnecessarily your database server.
I think that an improvement would be send a var as "log this" signal with your AJAX request. And your table won't have to check if there is or not a previous value, it will only insert your log request if the signal is ON. You only have to keep in the client side a time counter which will set the "log this" signal according to the time, vars, length of edit...
Related
I have an Android frontend.
The Android client makes a request to my NodeJS backend server and waits for a reply.
The NodeJS reads a value in a MySQL database record (without send it back to the client) and waits that its value changes (an other Android client changes it with a different request in less than 20 seconds), then when it happens the NodeJS server replies to client with that new value.
Now, my approach was to create a MySQL trigger and when there is an update in that table it notifies the NodeJS server, but I don't know how to do it.
I thought two easiers ways with busy waiting for give you an idea:
the client sends a request every 100ms and the server replies with the SELECT of that value, then when the client gets a different reply it means that the value changed;
the client sends a request and the server every 100ms makes a SELECT query until it gets a different value, then it replies with value to the client.
Both are bruteforce approach, I would like to don't use them for obvious reasons. Any idea?
Thank you.
Welcome to StackOverflow. Your question is very broad and I don't think I can give you a very detailed answer here. However, I think I can give you some hints and ideas that may help you along the road.
Mysql has no internal way to running external commands as a trigger action. To my knowledge there exists a workaround in form of external plugin (UDF) that allowes mysql to do what you want. See Invoking a PHP script from a MySQL trigger and https://patternbuffer.wordpress.com/2012/09/14/triggering-shell-script-from-mysql/
However, I think going this route is a sign of using the wrong architecture or wrong design patterns for what you want to achieve.
First idea that pops into my mind is this: Would it not be possible to introduce some sort of messaging from the second nodjs request (the one that changes the DB) to the first one (the one that needs an update when the DB value changes)? That way the the first nodejs "process" only need to query the DB upon real changes when it receives a message.
Another question would be, if you actually need to use mysql, or if some other datastore might be better suited. Redis comes to my mind, since with redis you could implement the messaging to the nodejs at the same time...
In general polling is not always the wrong choice. Especially for high load environments where you expect in each poll to collect some data. Polling makes impossible to overload the processing capacity for the data retrieving side, since this process controls the maximum throughput. With pushing you give that control to the pushing side and if there is many such pushing sides, control is hard to achieve.
If I was you I would look into redis and learn how elegantly its publish/subscribe mechanism can be used as messaging system in your context. See https://redis.io/topics/pubsub
Environment:
Windows Server 2003 - IIS 6.x
ASP.NET 3.5 (C#)
IE 7,8,9
FF (whatever the latest 10 versions are)
User Scenario:
User enters search criteria against large data-set. After initiating the request, they are navigated to a results page, where they wait until the data is loaded and can then refine the data.
Technical Scenario:
After user sends search criteria (via ajax call), UI calls back-end service. Back-end service queries transactional system(s) and puts the resulting data into a db "cache" - a denormalized table, set-up for further refining the of the data (i.e. sorting, filtering). UI waits until the data is cached and then upon getting notified that the process is done, navigates to a resulting page. The resulting page then makes a call to get the data from the denormalized table.
Problem:
The search is relatively slow (15-25 seconds) for large queries that end up having to query many systems based on the criteria entered. It is relatively fast for other queries ( <4 seconds).
Technical Constraints:
We can not entirely re-architect this search / results system. There are way to many complexities here between how the UI and the back-end is tied together. The page is required (because of constraints that can not be solved on StackOverflow) to turn after performing the search criteria.
We also can not ask the organization to denormalize the data prior to searching because the data has to be real-time, i.e. if a user makes a change in other systems, the data has to show up correctly if they do a search afterwards.
Process that I want to follow:
I want to cheat a little. I want to issue the "Cache" request via an async HttpHandler in a fire-forget model.
After issuing the query, I want to transition the page to the resulting page.
On the transition page, I want to poll the "Cache" table to see if the data has been inserted into it yet.
The reason I want to do this transition right away, is that the resulting page is expensive on itself (even without getting the data) - still 2 seconds of load time before even getting to calling the service that gets the data from the cache.
Question:
Will the ASP.NET thread that is called via the async handler reliably continue processing even if I navigate away from the page using a javascript redirect?
Technical Boundaries 2:
Yes, I know... This search process does not sound efficient. There is nothing I can do about that right now. I am trying to do whatever I can to get it to perform a little better while we continue researching how we are going to re-architect it.
If your answer is to: "Throw it away and start over", please do not answer. That is not acceptable.
Yes.
There is the property Response.IsClientConnected which is used to know if a long running process is still connected. The reason for this property is a processes will continue running even if the client becomes disconnected and must be manually detected via the property and manually shut down if a premature disconnect occurs. It is not by default to discontinue a running process on client disconnect.
Reference to this property: http://msdn.microsoft.com/en-us/library/system.web.httpresponse.isclientconnected.aspx
update
FYI this is a very bad property to rely on these days with sockets. I strongly encourage you to do an approach which allows you to quickly complete a request that notes in some database or queue of some long running task to complete, probably use RabbitMQ or something like that, that in turns uses socket.io or similar to update the web page or app once completed.
How about don't do the async operation on an ASP.NET thread at all? Let the ASP.NET code call a service to queue the data search, then return to the browser with a token from the service, where it will then redirect to the result page that awaits the completed result? The result page will poll using the token from the service.
That way, you won't have to worry about whether or not ASP.NET will somehow learn that the browser has moved to a different page.
Another option is to use Threading (System.Threading).
When the user sends the search criteria, the server begins processing the page request, creates a new Thread responsible for executing the search, and finishes the response getting back to the browser and redirecting to the results page while the thread continues to execute on the server background.
The results page would keep verifying on the server if the query execution had finished as the started Thread would share the progress information. When it does finish, the results are returned when the next ajax call is done by the results page.
It could also be considered using WebSockets. In a sense that the Webserver itself could tell the browser when it is done processing the query execution as it offers full-duplex communications channels.
I am trying to execute a mySql database query on my website depending on whether a user has "liked" my Facebook page. I have found a few ways to do this, using the PHP and JS SDK's, namely using the API with /USER_ID/likes/PAGE_ID.
When a user has liked my page, I want to add a value to their data in my database, so I thought of adding a function that is called each time the user visits the site, and if they like it, add value to database and also have a boolean value in there so it doesn't keep adding to the value. However, I guessed this would be waste of calls to the server if this happened every time, so I am not sure how to go about setting this up, any ideas?
Unless you are dealing with huge volumes of users I wouldn't worry about it because a check like that on one row of an indexed mysql table should be very quick (200 milliseconds or less for the entire request over a normal internet connection.). And if the data you need is stored on the server, then how could you possibly avoid the trip to the server? Unless you store the data in a cookie.
I'm looking at implementing a live voting system on my website. The website provides a live stream, and I'd like to be able to prompt viewers to select an answer during a vote initiated by the caster. I can understand how to store the data in a mySQL database, and how to process the answers. However:
How would I initially start the vote on the client-side and display it? Should a script be running every few seconds on the page, checking another page to see if a question is available for the user?
Are there any existing examples of a real-time polling system such as what I'm looking at implementing?
You would have to query the server for a new question every few seconds.
The alternative is to hold the connection open until the server sends more data or it times out, which just reduces (but does not eliminate) the server hits. I think it is called "long polling". http://en.wikipedia.org/wiki/Push_technology
You will have to originate the connection from the client-side. The simplest solution is to have the page make an AJAX request every second or so. Web pages don't have to return immediately (they can take 30 seconds or more before responding without the connection timing out). This, opening one connection which doesn't respond until it has something to say, is "long-polling".
You could use setTimeout in JavaScript to make AJAX requests each few seconds to check whether there are new questions.
Yes, long polling might be better, but I'm sure it's a bit more complex. So if you are up to the job, go ahead and use it!
Here's a bit more info on the topic:
http://www.webdevelopmentbits.com/avoiding-long-polling
Whenever I'm to prepare a long form for the client I always want to split it into separate pages, so the visitor doesn't have to fill it all, but does it in steps.
Something like:
Step 1 > Step 2 > Step 3 > Thank You!
I've never done it for one reason: I don't know how to store the data from separate steps efficiently? By efficiently I mean, how to store it, so when a visitor decides not to finish it at Step 3 all the data is deleted.
I've come up with few ways of how this could be resolved, but I'm just not convinced by any of them:
Storing form data in database
I can imagine a table with columns representing each question, with final column representing a bool value whether the form has been completed or not?
But I would have to do a clean-up of the table every now and then (maybe even every time it gets updated with new data?) and delete all entries with complete = 0.
Store form data in session data.
This on the other hand, does not have to store data in database (depending on how sessions are being handled) and all info would be in Cookie. But what if browser doesn't support cookies or user disabled them (rare, but happens), or if form has file attachments, then this is a no-go.
echo'ing form data from previous page as <input type="hidden"> on the next page
Yes, I'm aware this is a rather stupid idea, but it's an alternative. Poor, but it is.
Option 1 seems to be the best, but I feel it's unnecessary to store temporary data in DB. And what if this becomes a very popular form, with a lot of visitors filling it in? The amount of Updates/Deletes could be massive?
I want to know how you deal with it.
Edit
David asked a good question. What technology I'm using?
I personally use PHP+MySQL, but I feel like it's more generic question. Please share your solutions no matter of what server-side technology you use, as I'm sure the concept can be adapted one way or the other to different technologies.
I think the choice between options 1 and 2 comes down to how much data you are storing. I think in most cases the amount of data you are collecting on a form is going to be fairly small (a few kilobytes). In that case, I think storing it in the session data is the way to go. There's not that much overhead in passing that amount of data back and forth. Also, unless your users are on computers where there is a strict security policy in place, the application should work. If you make the page requirements clear users can decide if they want to proceed or not.
If you are storing large amounts of data for the form then a database would be better so you don't need to pass the data back and forth. However, I think this is going to be a fairly rare situation. Even if the application allows the uploading of files you can save those to a temporary location and only write them to the database once the form is completed. The other situation where you might want to use a database is if your form needs to be able to support the user leaving and coming back at a later time to resume the form.
I agree that option 1 is the best, because it has a few benefits over the other 2:
If the data is persisted, users can come back later and continue the process
Your code base will be much cleaner with incremental saves, and it alleviates the need for 1 massive save operation
Your foot print (each page request) will be lighter than option 3
If you're worried about performance, you can queue the data to be saved, since it's not necessary to save it near-real-time.
Edit to clear up a misconception: The data inside PHP Sessions, by default, are NOT stored in Cookies and are capable of storing a lot of data without too much overhead.
I'd go with number 2, but use the cookie only for identifying the user. The session data should actually be stored on your server and the cookie merely provides a lookup key to the session object that contains all the details.
If the site becomes popular and needs to run on more than a single web server, then your session data will need to be persisted in some kind of database anyway. In that case you would need a database that could handle massive amounts of transactions.
Note: I agree that this is a platform independent question. Stack Overflow users prefer to see code in questions and prefer to give code in answers, so that's why I normally ask what language someone is using.
To be brutally honest, just use the database as in option 1 and stop worring about data volumes. Seriously if your site is that successful that it becomes a problem then you ought be able fund a re-vamp to cope.
There's nothing wrong with taking the POST data from the previous step and adding hidden input elements. Just take all the POST data from the previous page that you care about and get them into the current page's form. This way, you don't have to worry about using persistent storage in any form, whether it's on the client side or the server side.
What are the perceived downsides? That there are a lot of extra elements on the page? Not that the user sees. All you have to do is add an element for each input you ask the user to give (on every page, if you want the user to be able to go back). Besides these elements, which don't give any visual clutter, there's nothing extra.
There's also the fact that all the form data will have to be transmitted on every page load. Sure, but this is probably going to be faster than a lookup in a database, and you don't have to worry about getting rid of stale data.