I want to setup a website (intranet in this particular case) that shows realtime updating data. I have the server and the realtime data, it's the software I know less about. I am no stranger to programming, but I am less familiar with web technologies.
Which alternatives do I have? I would prefer open source, and preferably something nimble and transparent as well.
EDIT:
With realtime data I mean a data that refreshes quicker than my monitor does.
I would prefer the data to update 'straight through' and not keep any specific refresh rate on the browser side. The data is to be shown in a regular tabular format, I don't need any fancy graphics. Please note at this stage I am not using any particular scripting framework. That is the purpose of this question, to figure out which one I should use.
I don't know what scripting language and data source your using but this will give you a direction.
Display data updates in real-time with AJAX
On the presumption that the data is retrieved using AJAX, you're after a "polling consumer pattern". In a nutshell, you make a request for your data and it will be blocked by the server until new data is available (or your request times out). When you receive your data, you poll for it again. In the event that you get an error (timeout, server failure etc.) then you might want to implement some back-off policy before trying again.
'hope that this helps.
Related
I want to consume kafka stream into mysql using Python; on top if which I want to build a realtime web based (web app) dashboard that will automatically be refreshed (ajax) on each data insert in the database.
After some searching, found a suggestion that ajax is not good for this purpose.
This post said websockets are better than ajax in terms of Performance.
Because I am not sure on whats the best way to achieve this So your expert advice is needed.
Thanks.
"I want to consume kafka stream into mysql using Python; on top if which I want to build a realtime web based (web app) dashboard that will automatically be refreshed (ajax) on each data insert in the database."
... (wince!) ...
Pretty-please find someone among your peers who can save you from yourself. (And is there any possible way that I can say this to you, such that you can "save face?" I can't think of any.) I'm not-at-all trying to have public fun at your expense. Please – "talk immediately to your manager. (S)He, surely, can help you."
I am certainly not an expert in this field, however my company uses Elastic/Kibana to read in from Kafka topics and display the data on a dashboard. This is just one of many routes you can take, but it works very well for us. You can read a little more into it here:
https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1
I'm struggling to find a solution to what I thought would be a common requirement so I'm hoping someone can help me with some pointers on what to search for/areas to explore.
Background
I'm building an iOS mobile app. I'm storing data locally using realm.io. The app is preinstalled with a snapshot of the content of a Wordpress mySQL database (it uses custom types). The content of the WP database is only written via the Wordpress install, the mobile app cannot write data.
Objective
So, I want to be able to check for changes since a given date (whenever the local database was last updated) and send the changed records to the mobile app (via the wp JSON api?).
I think I can fetch "posts since a date" but I need a full list of all create, update and delete operations since a given date.
Since the app is read-only I thought this type of one-way sync would be pretty straight forward but I can't find a common solution.
Any ideas to point me in the right direction would be great. Obviously, if anyone has any experience of doing this sort if thing with realm.io then that would be amazing :-)
Realm doesn't support yet any sort of synchronization mechanism across different files. We have an issue about that though, but you're likely searching rather for a solution in the immediate future.
Update: Realm launched the Realm Mobile Platform. This offers synchronization functionalities and would greatly simplify the solution for this use case.
You could use e.g. the server-side Node.js binding to pull new data from your MySQL Wordpress installation and push them to a global Realm served by the Realm Object Server. This can be read-only synchronized from the mobile apps, which would automatically receive the deltas and provide updated data to your users.
Whatever mechanism you come up yourself though in the meantime, it would require that you have read-write access from your iOS application to the realm database, so that you can update it with new data.
Pushing changed records as you describe is likely not going to work.
Apple's Push Notification service (APNS), which is the only back communication channel that works when your app is in the background or suspended, allows you to send very small payloads. You would use that to signalize your iOS app, that something changed on the server-side and there is new data to load. You would then initiate a request to a JSON-based API, wait for the response, map the returned JSON to Realm objects and store them in your database.
You want probably read more in the "Downloading Content in the Background" section of background execution chapter in the official App Programming Guides for iOS.
While pre-seeding the database from the app bundle seems like a nice idea, because the user wouldn't need to wait initially after downloading the app, that will enlarge the app itself with data, which might become in the future completely irrelevant.
I'm building a Phonegap application and i'm planning to set it to work in both Offline and Online mode, the idea is to get JSON Data from a PHP server side script and show it in the application, these data contains text, images so my question is is there anyway to set a dynamic cache manifest or any other way in a way that the user can see the already loaded data when he's offline, like caching the json result itself or anything else you can help me with
thanks
What do you mean by "work in both Offline and Online mode"? Are the users able to only see the user created data or can they modify it too? Is it correct to use a Manifest to control caching of user created data? I would have thought you should only really use it for caching the application templates and code. In any case I think you'll probably need something that you can exert more control over, something based perhaps using WebSQL, IndexedDb or LocalStorage.
I have been working on the modify-it-too question for many months and have a solution in the form of javascript(phonegap) <-> server synchronization that works somewhat like a version control system, with version numbers and conflict resolution. There's some pretty great docs and demo on the GitHub page and you can even see a presentation courtesy of SkillsMatter / LondonAJAX.. Currently there is only a JS/Node based server but I will probably do PHP based server because that's my day job If you look at the server code it's pretty easy to implement.
BTW I would use a different method to store the images, why not just download them and store them in a File, I don't know the storage limits, but I bet that is the way that Phonegap will let you get away with most.
Background
I have a (glorified) CRUD application that I'd like to enable HTML5 offline support with. The cache-manifest system looks simple yet powerful, but I'm curious about how I can allow users to access data while offline.
For example, suppose I have these pages for the entity "Case" (i.e. this is CRM case-management software):
http://myapplication.com/Case
http://myapplication.com/Case/{id}
http://myapplication.com/Case/Create
The first URI contains a paged listing of all cases, using the querystring parameters pageIndex and pageSize, e.g. /Case?pageIndex=2&pageSize=20.
The second URI is the template for editing individual cases, e.g. /Case/1 or /Case/56.
Finally, /Case/Create is the form used to create cases.
The Problem
I would like all three to be available offline.
/Case
The simple way would be to add /Case to the cache-manifest, however that would break paging (as the links wouldn't work).
I think I could instead add something like /Case/AllData which is an XML resource, which is cached and if offline then a script on /Case would use this XML data to populate the list and provide for pagination.
If I go for the latter, how can I have this XML data stored in the in-browser SQL database instead of as a cached resource? I think using the SQL database would be more resilient.
/Case/{id}
This is more complicated. There is the simple solution of manually adding /Case/1, /Case/2, /Case/3 etc... to /Case/1234, but there can be hundreds or even thousands of cases so this isn't very practical.
I think the system should provide access to the 30 most recent cases, for example. As above, how can I store this data in the database?
Also, how would this work? If I don't explicitly add /Case/34 to the manifest and the user clicks on to /Case/34 how can I get the browser to load a page that my JavaScript will populate based on the browser's SQL database data and not display the offline message?
/Case/Create
This one is more simple - as it's just an empty page and on the <form>'s submit action my script would detect if it's offline, and if it is offline then it would add it to the browser's SQL database. Does this sound okay?
Thanks!
I think you need to be looking at a LocalStorage database (though it does have some downsides), but there are other alternatives such as WebSQL and IndexedDB.
Also I don't think you should be using numeric Id's if you are allowing people to create as you will get Primary Key conflicts, it is probably best to use something like a GUID.
Another thing you need is the ability to push those new cases onto the server. there could be multiple...
Can they be edited? If they can I think you really need to be thinking about synchronization and conflict resolution hard very hard if that is the case.
Shameless self promotion, I have a project that is designed to handle these very issues, though it's not done, it's close. You can see it (with an ugly but very functional) demo at https://github.com/forbesmyester/SyncIt
I have a editable html5 page and I store new elements in localStorage.
I want to synchronize my page with the server.
I want to know if I can do it without a server side script or if there is some tips to do something like this in a good way.
Thank you :)
You can pull information from the server quite easily using jQuery and then just put it on Local Storage but, if you want to upload local information to the server there is no way around, you have to use some kind of script, tough it's not that difficult, there are many languages (PHP, C#, Python...) and tools you can use.
Keep in mind that when you upload information to the server you have to sanitize it very important security measure.
Basically, the way to go is:
Post the information to the server (using AJAX or a HTML form, either way will do)
Use some server-side script to capture the variables posted.
Sanitize your data (check format, discard non-valid characters, etc)
Store it on database (Do not, ever, concatenate your data with a SQL query ok? that can make you vulnerable to a SQL injection attack), compute something or do stuff.
Return some status to the client (some confirmation maybe?)
You may want to take that confirmation and show a message to the user ("Your info was saved properly" or something like that)
is a javascript timer not sufficient for this manner? or jQuery?
The question really should be more of a problem than a question. If you're updating based on a server's variables then you could use AJAX i believe but if its like increment said variable every X seconds I would focus on using a javascript timer.