This article describes how a websocket server for a chat application can look. We are planning to implement something similar; when a message is sent to the server it is sent to the correct recipient based on an authentication token and the message gets saved in a mysql database.
We will eventually host the server on Google App Engine, and I suspect that that will cause some issues with the above described approach, since that depends on all clients being connected to the same server, and that probably won't be the case since multiple instances will be created as needed. Is there a way to connect all instances so that this won't be a problem (Pub/Sub maybe? (That will cause additional costs though)), or should we find a different solution?
One idea I had was to use mysql-events to monitor the binlog from the websocket server for the creation of new rows in the messages table, but I read somewhere that that wasn't recommend. But I can't find where I read that, and maybe that is the best solution.
Since you asked about other solutions, I would recommend looking at Firebase and specifically the Realtime Database. Out of the box it provides all of the functionality that you need for realtime communication between connected clients and Cloud Messaging for clients who aren't.
Here's a tutorial that uses Firestore to create a realtime chat web app, but it can all be applied to the Realtime Database with minor modification. I say that because Firestore has expensive writes, which in my opinion make it unsuitable for a chat backend.
Related
I have a full stack app that uses React, Node.js, Express, and MySQL. I want the react app to respond to database updates similar to Firebase: When data changes, I want a real-time notification sent to my app.
I want to use stock MySQL (no plugins), so that I can use AWS RDB or whatever.
I will use socket.io to push the real-time notifications to the web app.
To avoid off-target responses, I'll summarize various approaches that are not what I am looking for:
The server could poll, or each client could poll. (Not real-time, but included for completeness. When I search, polling is the only solution I find.)
Write a wrapper that handles all MySQL updates, handles subscriptions, and sends the notifications. This is a complicated component that adds complexity. Firebase is popular because it both increases performance and reduces complexity. I like Firebase a lot but want to do the same thing with MySQL.
Use Firebase to handle the real-time notifications. The MySQL wrapper could use Firebase to handle the subscriptions and notifications, but there is still the problem of triggering the notifications in the first place. Also, I don't want to use Firebase. (For example, my application needs to run in an air-gapped environment.)
The question: Using a stock MySQL database, when a table changes, can a notification server discover the change in real-time (no polling), so that it can send notifications?
The approach that works is to listen to the binary logs. This way, any change to the database will be communicated in real-time. The consumer of the binary logs can then publish this information in a number of ways. A common choice is to feed a stream of events to Apache Kafka.
Debezium, Maxwell, and NiFi work this way.
I am doing some research for a mobile app I want to develop, and was wondering whether I could get feedback on the following architecture. Within my future app users should be able to authenticate and register themselves via the mobile app and retrieve and use their settings after a successful authentication.
What I am looking for is an architecture in which user accounts are managed by AWS Cognito, but all application related information is stored in a MySQL database hosted somewhere else.
Why host the database outside of AWS? Because of high costs / vendor lock-in / for the sake of learning about architecture rather than going all-in on AWS or Azure
Why not build the identity management myself? Because in the end I want to focus on the App and don't spent a lot of energy on something that AWS can already provide me with (yeah I know, not quite in line with my last argument above, but otherwise all my time goes into database AND IAM)
One of my assumptions in this design (please correct me if I am wrong) is that it is only possible to retrieve data from a MySQL database with 'fixed credentials'. Therefore, I don't want the app (the user's device) to make these queries (but do this on the server instead) as the credentials to the database would otherwise be stored on the device.
Also, to make it (nearly) impossible for users to run queries on the database with a fake identity, I want the server to retrieve the User ID from AWS Cognito (rather than using the ID token from the device) and use this in the SQL query. This, should protect the service from a fake user ID injection from the device/user.
Are there functionalities I have missed in any of these components that could make my design less complicated or which could improve the flow?
Is that API (the one in the step 3) managed by the AWS API Gateway? If so, your cognito user pool can be set as Authorizer in your AWS API Gateway, then the gateway will take care automatically of the token verification (Authorizers enable you to control access to your APIs using Amazon Cognito User Pools or a Lambda function).
You can also do the token verification in a Lambda if you need to verify something else in the token.
Regarding to the connection between NodeJS (assuming that is an AWS lambda) that will work fine, but keep in mind the security as your customers data will travel outside AWS, and try to use tools like AWS Secret Manager to keep your database passwords safe and rotate them from time to time in your lambda.
I want to be able to allow the user to work offline with a javascript database such as PouchDB or IndexDb to store records not just user data and then sync up to the server when online.
To that end FeathersJS said it could sit in the middle between a legacy api and the Feather's client to handle real-time sync.
Does the real-time sync mean that Feathers is appropriate for use as a two way client to api synchronization with conflict resolution?
My understanding is that the default Feathers realtime sync does not handle offline storage and conflict resolution out of the box.
That said, there are a lot of resources for getting Feathers to accomplish this. I would start here:
https://feathersjs-offline.github.io/docs/
(Edited thanks to Yannick Marcon from a comment)
I am trying to Ajax load from LAN's mysql using chrome app.
I am proposing Ajax because I need chrome app to load up any update in the SQL instantaneously.
Since this app is only used in LAN network, I presume there is no need to maintain a web server (aka running Apache). Can anyone provide some hints as this answer I found on the forum does not help me (an absolute newbie) too much.
https://developer.chrome.com/extensions/xhr
Thank you.
YY
Since this app is only used in LAN network, I presume there is no need to maintain a web server (aka running Apache).
AJAX refers to making a HTTP request to.. something.
Something that can answer HTTP requests is called a web server.
So, you do need some sort of web server. It may be a component of MySQL server, but it's still a web server.
That said, it doesn't look like MySQL has a supported HTTP interface. There is an experimental HTTP Plugin that provides REST API, but it's experimental. Therefore, you would need a separate server application that does what you need.
That said,
I am proposing Ajax because I need chrome app to load up any update in the SQL instantaneously.
AJAX is not a magic bullet. It works well for requesting data, but it is not adapted to receiving updates initiated by the server you're talking to. It's a request-response cycle, and while there are some techniques to use it to push data they are hacks.
WebSockets evolved to cover the bidirectional, persistent communication needs. However, this again would require a web server to sit as a proxy between your DB and your app - this time, WebSockets-capable.
That said, building a Chrome App allows you to connect to a database directly - since Chrome Apps are capable of using chrome.sockets API. You would need a JavaScript library specifically adapted to the task, but those probably exist.
That said, and noting that I'm not an expert on databases, but..
Databases are not designed to notify you about updates. You need to poll them to see if the data has changed. You will not get it instantaneously no matter what interface you use. You'll need to periodically monitor it for changes.
Considering this, depending on what you're trying to ultimately do you may be choosing a wrong instrument.
There's a lot of "buts" here, and it seems like a complex task. You should re-evaluate your readiness as an "absolute newbie" to undertake it.
We have built a LAMP-stack API application via PHP Laravel. This currently uses a local mySQL instance. We have mostly implemented views in AngularJS.
In order to use Firebase, we need to sync data between the authoritative store in mySQL with anything relevant that exists on Firebase, as close to real-time as possible. This means that other parts of the app which are not real-time and don't use Firebase can also serve up fresh content that's very recently been entered into the system.
I know that Firebase is essentially a noSQL database in the cloud. My question is - how do I write a wrapper or a means to sync the canonical version of my Firebase into my database of record - mySQL?
Update to answer - our final decision - ditching Firebase as an option
We have decided against this, as we can easily have a socket.io instance on the same server with an extremely low latency connection to mySQL, so that the two can remain in sync. There's no need to go across the web when resources and endpoints can exist on localhost. It also gives us the option to run our app without any internet connection, which is important if we sell an on-premise appliance to large companies.
A noSQL sync platform like Firebase is really just a temporary store that makes reads/writes faster in semi-real-time. If they attempt to get into the "we also persist everything for you" business - that's a whole different ask with much more commitment required.
The guarantee on eventual consistency between mySQL and Firebase is more important to get right first - to prevent problems down the line. Also, an RDMS is essential to our app - it's the only way to attack a lot of data-heavy problems in our analytics/data mappings - there's very strong reasons most of the world still uses a RDMS like mySQL, etc. You can make those very reliable too - through Amazon RDS and Google Cloud SQL.
There's no specific problem beyond scaling real-time sync that Firebase actually solves for us, which other open source frameworks don't already solve. If their JS lib actually handled offline scenarios (when you START offline) elegantly, I might have considered it, but it doesn't do that yet.
So, YMMV - but in our specific case, we're not considering Firebase for the reasons given above.
The entire topic is incredibly broad, definitely too broad to provide a simple answer to.
I'll stick to the use-case you provided in the comments:
Imagine that you have a checklist stored in mySQL, comprised of some attributes and a set of steps. The steps are stored in another table. When someone updates this checklist on Firebase - how would I sync mySQL as well?
If you insist on combining Firebase and mySQL for this use-case, I would:
Set up your Firebase as a work queue: var ref = new Firebase('https://my.firebaseio.com/workqueue')
have the client push a work item into Firebase: ref.push({ task: 'id-of-state', newState: 'newstate'})
set up a (nodejs) server that:
monitors the work queue (ref.on('child_added')
updates the item in the mySQL database
removes the task from the queue
See this github project for an example of a work queue on top of Firebase: https://github.com/firebase/firebase-work-queue