I'm developing a website with the microservice architecture, and each of the service owns a database. The database stores the data which the microservice needs.
Post, Video services need the user information, so both of the services subscribed to the NEW_USER_EVENT.
The NEW_USER_EVENT will be triggered when there's a new user registered.
Once the services received the NEW_USER_EVENT, they put the incoming user information to each of their own database. So they can do things without asking the User service.
So far so good. But here comes the question:
What if I'm going to create a new service? How do I get the registered user informations and put them in the new service?
Maybe I can get the informations from the existing services. But the events are pushed by the messaging queue (NSQ).
If I'm going to copy the data from one of the microservices, how do I make sure which service has the latest user informations? (Because some services haven't received the latest event)
Read More:
The Hardest Part About Microservices: Your Data
Intro to Microservices, Part 4: Dependencies and Data Sharing
What if I'm going to create a new service? How do I get the registered user informations and put them in the new service?
You have to replay all events to which this new service is subscribed from the beginning of time (you should have an "event store" that keeps all events already happened in your application). Also, you can put a bit smarter logic when replaying events by starting from the most recent ones and going back in time. In this way, you will be able to restore most valuable data first. Just be careful to handle interdependent events correctly.
Data source: The events are pushed by the messaging queue(NSQ), If I'm going to copy the data from one of the microservices, how do I make sure the copy source has the latest user informations?
You are not talking about doing backups, right?
Aside from backups, in the event-driven systems people usually don't copy the data in a classical way, row by row. Instead, they just replaying events from event store from the beginning of time and feeding those events to the event handlers for the new service (or new instance). As a result, new service eventually becomes consistent with the other parts of the system.
Related
I am trying to build a simple project online. I have a MySql Database,Database where I will store different information, such as fake orders made from fake clients. The application will be formed by a frontend made with Javascript and HTML/CSS, while the backend will be a Node/Express API that should handle all the requests and handle the database.
I wanted to know whether there is a way to, each time a new order is made, have a refresh in my own page, and see, for example, a new column in an hypotetical table in my HTML with a minumum latency, avoiding making a request from the client every x seconds. This because it could be quite expensive in terms of bandwith and also prett unefficient.
I thought that each time I connect to the site, I get subscribed to a sort of list in the server, that broadcast a trigger to then update the frontend when tha UPDATE function is triggered in the backend. In other words, every time an update is done on the backend, the server sends a trigger to the clients that he knows are currently connected. Then, the frontend asks for the update directly.
This solutions i really complicated to handle and may be not that performant. I was thinking if there where some functionalities of the frontend or the backend or the database, or any framework that allow me to do this thing.
I would like to have all as real time as possible, using the least bandwith possible. This is because I would like to use the free tier of some online service, and I don't want to consume all the bandwith.
If you have some suggestions of framework or functionalities, or any protocol, you are welcome. Thank you a lot in advice
You can use websockets. When a user creates an order and after there is a success saving to the data base, the backend will push or publish the data to the client who is subscribed to a specific channel. The logic is not complicated at all it is called the pub/sub pattern you should search for it.
Also https://socket.io/ this is library that used on both backend and front end to deal with websockets.
I'm new to Java and just started writing some JavaFX applications.
My current project is to write an application for a consulting company that store a list of customers, add them to a queue and serve them one by one. There are a few staffs and they will running a copy of the application I write on their PC.
What I've done so far:
create Customer.class to handle personal info and store them in a MySQL db
create Staff.class to handle staff info
create Service.class to handle kind of services are available for the customers
create Consultation.class to handle info of a particular consultation such as date of consultation, customer being served, which staff is providing service, the services offered and the outcome
create an ObservableArrayList, store the data in the MySQL db, and display the data on a TableView of each client PC
What I want to do is, after a staff editing the data in the list, the changes will be updated on the TableView of other client PCs automatically.
The possible solutions I can think of includes:
Option 1
Program the application to query the db regularly for an update.
This method is more simple to implement, but I don't want to keep the MySQL server busy by non-stop querys from a number of clients. I do not want any delay between data write and update on other clients. There are more than 10 clients. If each client update once a second, that will mean at least 10 queries per second and the server will never rest. I don't want to put any stress on the server's harddisk.
Option 2
Program the application to broadcast a message every time after they write data to the db and other clients query the database every time they receive a broadcast. I prefer do it this way but I'm not familiar with network programming. That will mean I'll have to spend some time on it before I can continue the project.
Which of the above is a better choice? Is there other way to keep the TableView on the clients keep synchronized?
Which of the above is a better choice? Is there another way to keep the TableView on the clients keep synchronized?
Before choosing - you may consider optimizing them,
Option 1 seems quite expensive as it has to request frequently. But you can optimize it using connection-pool and specifying certain time-interval(minimum 10 sec) to fetch the data.
Option2 is much more convincing as it applies the lazy-loading concept. You may consider looking socket programming to notify all clients to fetch data.
It's quite hard to say which one is the better option - somehow, I prefer to go with the first approach if your application may insert data frequently, otherwise go with the second one.
An alternative solution - listening to the data changes
Here are some QA, these solutions may help you to implement your requirement.
How to implement a DB listener in Java
How to make a database listener with java?
How to listen to new DB records through java
I'm working on a Rails 3.2 application where users create, read and update multiple kinds of reports. We use MySQL and Redis.
I would like to notify users when one of their reports hasn't been updated in the previous X months by showing them notifications on their profile/dashboard page in the browser.
I would prefer to do this asynchronously.
At some point, I would also like to have live, in-app notifications so users can be notified when a report they're watching has been updated or someone likes their report. Conceptually, I was contemplating the best way of going about doing this...
A cron job that runs a SQL query and retrieves all reports that have update_at fields older than 3 months, and creates a notification record linked to that user and report. A new notification record will be created only if user didn't have an existing notification.
A background job, using something like Resque that checks the database periodically throughout the day. Notifications are stored in a queue. This seems like it would scale better into a more robust, in-app notifications feature.
Are these my only two options? Is there a better asynchronous way to listen to the database, and notify a user when one of their records hasn't been updated in 3 months? Is some implementation of websockets necessary here?
I don't believe you necessarily need to use CPU cycles with some always-running background job, but that depends on how powerful you want these notifications to be. If you want them to be real-time, and you plan on using a publisher-subscriber model in the future then I'd consider a redis + node server setup.
The complexity of websockets means more implementation time. Will that extra time spent really benefit you? If it's very important that your notifications be asynchronous and real-time, like exchanging messages in a chat app or getting pop-up notifications on Facebook, then this is the right choice.
However, if all you really want to do is let users know when their reports are past a certain age, when they request their profile page, a cron that runs every couple of hours will be much easier to implement. You could even use Google Analytics to get a feel for when users are putting the heaviest burden on your app, and schedule the job around that.
First, create a rake task that completes the behavior your described above--check for old reports and create notifications. Put that in your app's lib/tasks directory. And then write a cron job like so:
$ 0 6,12,18 * * * cd /path/to/app && bundle exec rake task_name
This task will run 3x a day, at 6am, 12pm and 6pm.
From there, if you want to create an activity feed for your users, you could also implement a synchronous pub-sub solution pretty simply. E.g., User A subscribes to User B's reports, User B publishes a new report, after_save callback to create a notification for User A. This will be slower, it won't be live, but if it's a small app and this feature isn't a core part, then this makes better use of your time than the other options.
I have a problem related to automatic task scheduling.
Currently i am able to find out when my customer has last credited his account, how am i able to find out whether he will pay anything in the next 3 days?
So if no payment has been made in the next three days for any customer, to automatically alert me preferably by a notification directly to my ipad.
I dont want myself to open the app for checks to be done only when i log in, because then if i jump on my application 6 days later, i could have had a customer that hasnt paid in 6 days when the app should have alerted me on the 3rd day so i could ring my customer up to deal with the matter.
I need to work in this matter due to the structure of my application and business.
I am able to monitor everything else but need some insight on how I can go about doing this. the current notification system inside the phone only fires based on time, and I cannot do interval checks where maybe i could run a background task, if that would work then i would have done it like that but thats not the case.
Pavan
If I understand your question correctly, you should compute the interval of the event that you want and post a wake-up timer that is that period of time from "now." If you need it through the notification center, then just handle it silently and clear it from the notifications.
Based on the discussion below:
You will need a little bit of server work. APNS looks complicated, but it really has very few moving parts -- especially if it is a private App. What system component is keeping an eye on Amazon? Do you have an App or web server? For example, if I were to poke a record into your system (purchased services) what workflow is triggered to notify Accounting to process an invoice and collections at a later date? Am I making any sense of your system architecture?
Perfect - you are done. You have all the system components you need and the rest is coding. The server app processes the accounts DB and finds new entries. If found, it publishes a record ID to the APNS server (Apple owns this server). You write code to register to receive the push-notification (subscriber). When you get a push, that will wake up your registered app with the record ID (and some other subscription stuff for bookkeeping -- but you are the only subscriber and only subscribing to one DB table -- so you can largely ignore. Now turn around and query based on that record. Done!
Each day hundreds of thousands of items are inserted, updated and deleted on our service (backend using .Net and a MySql database).
Now we are integrating our service with another service using their RESTful API. Each time an item is inserted, updated or deleted on our service we also need to connect to their web service and use POST, PUT, DELETE.
What is a good implementation of this case?
It seems like not a very good idea to connect to their API each time a user inserts an item on our service as it would be a quite slow experience for the user.
Another idea was to update our database like usual. Then set up another server constant connecting to our database and fetching data that needs to be posted to the RESTful API. Is this the way to go?
How would you solve it? Any guides of implementing stuff like this would be great! Thanks!
It depends if you delay in updating the other service is acceptable or not. If not, than create a event and put this in queue of event processor who can send this to second service.
If delay is acceptable than there can be background batch job that can run periodically and send the data.