I want to consume kafka stream into mysql using Python; on top if which I want to build a realtime web based (web app) dashboard that will automatically be refreshed (ajax) on each data insert in the database.
After some searching, found a suggestion that ajax is not good for this purpose.
This post said websockets are better than ajax in terms of Performance.
Because I am not sure on whats the best way to achieve this So your expert advice is needed.
Thanks.
"I want to consume kafka stream into mysql using Python; on top if which I want to build a realtime web based (web app) dashboard that will automatically be refreshed (ajax) on each data insert in the database."
... (wince!) ...
Pretty-please find someone among your peers who can save you from yourself. (And is there any possible way that I can say this to you, such that you can "save face?" I can't think of any.) I'm not-at-all trying to have public fun at your expense. Please – "talk immediately to your manager. (S)He, surely, can help you."
I am certainly not an expert in this field, however my company uses Elastic/Kibana to read in from Kafka topics and display the data on a dashboard. This is just one of many routes you can take, but it works very well for us. You can read a little more into it here:
https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1
Related
With a colleague of mine, we're building an app written in Dart with Flutter on Android Studio. We've arrived at the point where we need to start integrating a database to collect and send user filled data, and so we chose MongoDB which will be integrated into Docker so that our app is ready to function on multiple devices. Since we will have many users and each of them will be entering their own data, we have a lot of parameters to take into account so we're creating a JSON skeleton to map out the structure of what data goes where. The obstacle is we have no clue what the best way is to approach MongoDB-Docker integration with our Android Studio code, as it is our first time using MongoDB and Docker. Any good tips or resources that could put us on the right track ? Thank you
Hi your real question should be : what will you use in beetween those two ? You should (I guess) create an API to simplify and securise your user-DB interactions.
If you're not familiar with those principles a quick reasearch should help.
If you want to continue without any API, you should put many effort into having a VERY clean code as your app will have a lot of information inside its source code and proceed to create a documentation. Also you could use an ORM.
I am working on a social network app from scratch,
I designed my database (choosing Mysql) to store and retrieve data
The problem starts when I am posting videos and photos and when data become larger and larger when I am waiting to view news feeds (depending on sql query that brings the last posts from friends ans Pages).
The question is :
How can I manage and handle the big data, and how can I make the news feeds service more efficient ?
What do you think of using (Real-time) Databases from Firebase?
If you're thinking about putting up Firebase as your choice for your database, then there's nothing wrong with that. Firebase realtime database is a very secure and great option.
Firebase gives you functionality like analytics, databases, messaging and crash reporting so you can move quickly and focus on your users.
Firebase is built on Google infrastructure and scales automatically, for even the largest apps.
And on top of that, many great apps do use Firebase as their backend.
I'd say to know more, you can just start with a demo project on Firebase to learn how the things happen with Firebase, and after learning, start using it as backend for your social app.
Just go to Firebase Console and start with your own project. To learn how to use Firebase, do refer Firebase docs and if stuck, StackOverflow is always there.
If you want to get some inner details, I have some repositories on GitHub, that might help you in understanding about Firebase and the security rules and other important things.
Demo App
Firebase Security Rules
I know this sounds crazy, but I had a thought and I was willing to try it out. I use GitLab pages for all my online projects, but a lot of them are ASP.NET MVC, which is an issue as I don't think you can run ASP.NET MVC sites on GitLab pages. I then thought, what if I make a site using something like angular or node.js, and have a central API for all my web projects? I thought that was a great idea, until I realized I couldn't use a database either. I guess what I'm asking is, would it be possible to create a REST API that uses JSON files for storage and node.js as the request pages, to create an API without a database?
Of course.
If you think about a database from the perspective of your application code, it is basically just a place to store and retrieve data.
Imagine the database library you are using has two simple methods, store and retrieve. In your application code, you could write db.store('here is the item') and the later on, db.retrieve().
However, those store and retrieve methods could be implemented in many different ways to provide the same effective behavior from the perspective of your application. Some examples:
Send/query the data to/from an external data store, such as PostgreSQL
Write it to a file on disk and read it back later
Store the data in memory
Make HTTP requests to an external system to store the data
Some of these options will be more or less appropriate depending on your exact requirements, however, the general idea is that given a database API, you could implement the exact same method signatures with a completely different approach.
I'm new to webdev and I'm trying to use passport for registration/authentication on a site I'm setting up. I'm also going to write an application in node later on that will be using some of the user data (users will need to provide an API key for an account on another site that I will use to pull data into the application).
At the moment, the main issue I'm having is figuring out what goes where. I've found plenty of resources that explain how to create an app using passport, but nothing shows how it would be incorporated into your website or where the files should be in relation to your website. I'm relatively new to Node.js, and while I've written a few small applications I have never hosted them anywhere.
Bonus question: I'm using MongoDB with passport and I was also planning to use it to store some JSON my application will be receiving from API calls. However, I wanted to use MySQL to store some data as well. More specifically, I'm planning to save the raw JSON then I'll create a relational database out of the data I need from the JSON and then keep the rest in MongoDB for easy access. Is this common/smart, or should I focus on keeping everything in my MongoDB? I'm relatively new to NoSQL.
Thanks in advance for any help.
I would reference this tutorial. I just recently used this to help myself with a new application. Also there is an example of the same thing but in SQL here. So not sure what you mean by " where the files should be in relation to your website". The information related to to authentication should go in your database.
To your "bonus question" you can use two databases. The key here is to ask yourself why and what are the true needs for data, and how is this data accessed and used. From ground up I would like one and stick with it. If at some point later you realize a certain type of data would be better in a different database then you can add it.
Side note: look into an IDE such as webstorm to help you out.
I want to setup a website (intranet in this particular case) that shows realtime updating data. I have the server and the realtime data, it's the software I know less about. I am no stranger to programming, but I am less familiar with web technologies.
Which alternatives do I have? I would prefer open source, and preferably something nimble and transparent as well.
EDIT:
With realtime data I mean a data that refreshes quicker than my monitor does.
I would prefer the data to update 'straight through' and not keep any specific refresh rate on the browser side. The data is to be shown in a regular tabular format, I don't need any fancy graphics. Please note at this stage I am not using any particular scripting framework. That is the purpose of this question, to figure out which one I should use.
I don't know what scripting language and data source your using but this will give you a direction.
Display data updates in real-time with AJAX
On the presumption that the data is retrieved using AJAX, you're after a "polling consumer pattern". In a nutshell, you make a request for your data and it will be blocked by the server until new data is available (or your request times out). When you receive your data, you poll for it again. In the event that you get an error (timeout, server failure etc.) then you might want to implement some back-off policy before trying again.
'hope that this helps.