I have been working with Angular for some time now. My question is simple, I have a database with multiple tables. There is a clients table and around 7 or 8 other tables that contain information about that client that I need. None of the data from these tables is too terribly large. In order to reduce http calls, it was my thought to load all of the tables and store the data from each into a object stored in a factory.
So once a particular client is called, the http requests are made from each table and each are stored inside of a factory. Then, when a user needs to access that table, its data is stored in memory as the http call has been completed at the outset. When the data is changed, it can make a quick save of the table data and reload it again.
Most of the data is financial containing information about the income and asset categories of the client.
Question is .. is this wise? Am I missing something?
Thanks in advance
Your use of the term factory is inappropriate as a factory is a creational pattern. What you are describing is a facade. It is reasonable for a facade to aggregate data for a client and present it in a unified manner.
So, a remote client requests some data. The server-side facade makes the many requests on behalf of the client and composes the single response.
You have mentioned about caching the data. If you choose to do so, you will need to consider how to manage the cache data for staleness, how much memory you will need, etc.
Related
I have a scenario wherein frontend app makes a call to backend DB (APP -> API Gateway -> SpringBoot -> DB) with a JSON request. Backend returns a very large dataset (>50000 rows) in response sizing ~10 MB.
My frontend app is highly responsive and mission critical, we are seeing performance issues; frontend where app is not responding or timing-out. What can be best design to resolve this issue condering
DB query cant be normalized any further.
SpringBoot code has has cache builtin.
No data can be left behind due to intrinsic nature
No multiple calls can be made as data is needed is first call itself
Can any cache be built in-between frontend and backend?
Thanks.
Sounds like this is a generated report from a search. If this data needs to be associated with each other, I'd assign the search an id and restore the results on the server. Then pull the data for this id as needed on the frontend. You should never have to send 50,000 rows to the client in one go... Paginate the data and pull as needed if you have to. If you don't want to paginate, how much data can they display on a single screen? You can pull more data from the server based on where they scroll on the page. You should only need to return the count of the rows to the frontend, and maybe 100 rows of data. This would allow you to show a scrollbar with the right height. When they scroll to a certain position within the data, you can pull the corresponding offset from the server for that particular search id. Even if you could return all 50,000+ rows in one go, it doesn't sound very friendly to the end user's device to have to load that kind of memory for a functional page.
This is a sign of a flawed frontend that should be redone.
10mb is huge and can be inconsiderate to your users especially if there's a high probability of mobile use.
If possible, it would be best to collect this data on the backend, probably put it onto disk, and then provide only the necessary data to the frontend as it's needed. As the map needs more data, you would make further calls to the backend.
If this isn't possible, you could load this data with the client-side bundle. If the data doesn't update too frequently, you can even cache it on the frontend. This would at least prevent the user from needing to fetch it repeatedly.
Earlier in our database design, we use to create mandate fields for each of the table and few important fields were:
created_by
created_time
created_by_ip
updated_by
updated_time
updated_by_ip
Now, its an era of no-schema design. We prefer mongodb or some other just writing databases.
My question here is:
Is it a good practise to maintain logs in a separate database?
Do we need to create separate log table for each mysql tables considering mongodb or is it okay to have single mongodb audit table for
all mysql tables?
What things need to be considered in querying the results from mongodb?
What should be the structure for mongodb table structure?
Any other alternatives to store logs?
Considering situation where if we want to delete registered user if not authenticated in specified time(max of 48hrs).
If all the time logs are handled in mongodb. How can we query the same from mysql?
You usually want this (audit?) data next to the real data and definitely not in a different DB engine as the number of partial errors to support becomes quite a nightmare (e.g. someone registered, but you fail to insert audit data - is this ok? should the account become orphan? What happens if the app goes down half way?).
Systems that have this separation usually use messaging and 2 different listeners are responsible for storing the data and storing the audit (e.g. one in a relational DB and the other in an event store). In this way you have a higher chance of achieving eventual consistency.
Edit
There are a few options around using messaging and the assumption here is that both sources of data must be in sync (or as close as possible). Please bear in mind that I still think that storing data+audit together is by far the simplest and more sensible approach.
Using messaging, your app can emit a message on certain events (e.g. user created). Then 2 different listeners react to this message. One listener stores the data in one DB engine; Another listener stores the audit data. The problem with this approach is that you might need to ensure ordering on the messages, which makes it really slow.
Another (scary) approach is to use distributed (XA) transactions between MySQL and a messaging system (as mongo doesn't support transactions). Then the data to MySQL and the message would be committed together, and a listener can receive the audit data and store it in mongo.
I need to emphasize that the 2 approaches above are horrible and should never be implemented.
There are more sensible approaches but might require a different tech stack. For example using an EventSourcing+CQRS you can store the events (with the audit data) and store the final read models without the audit data.
Suppose you have a mobile app that needs to ask the server for 20 other Users near your current location. The URL to get this data might look something like this (not escaped):
https://example.com/api/users?lat=40.240239&long=-111.657920&count=20
The server could then respond in one of two ways:
Return all User objects directly, as JSON, in one large array.
Return an array of UUIDs corresponding to the Users who match the request. The client would then have two choices:
a) Send a request for all User objects in one big batch:
https://example.com/api/users?ids=[1,2,3,4...]
b) Send requests for each User independently:
https://example.com/api/users?id=1
https://example.com/api/users?id=2 ...
Currently, my application implements Option #1. In the name of responsiveness, I've eliminated every possible network round-trip by returning as much data as possible in as few requests as possible. However, I'm starting to see problems with this choice due to the fact that my client and server logic are very tightly coupled. It's difficult to maintain, versioning is a nightmare, and caching on the client side is much more difficult than say, Option #2b.
Based on your experience, which option do you recommend (or a different method entirely)? What would you consider the "industry standard" way of serving data to a mobile app?
I'll describe the application I'm trying to build and the technology stack I'm thinking at the moment to know your opinion.
Users should be able to work in a list of task. These tasks are coming from an API with all the information about it: id, image urls, description, etc. The API is only available in one datacenter and in order to avoid the delay, for example in China, the tasks are stored in a queue.
So you'll have different queues depending of your country and once that you finish with your task it will be send to another queue which will write this information later on in the original datacenter
The list of task is quite huge that's why there is an API call to get the tasks(~10k rows), store it in a queue and users can work on them depending on the queue the country they are.
For this system, where you can have around 100 queues, I was thinking on redis to manage the list of tasks request(ex: get me 5k rows for China queue, write 500 rows in the write queue, etc).
The API response are coming as a list of json objects. These 10k rows for example need to be stored somewhere. Due to you need to be able to filter in this queue, MySQL isn't an option at least that I store every field of the json object as a new row. First think is a NoSQL DB but I wasn't too happy with MongoDB in the past and an API response doesn't change too much. Like I need relation tables too for other thing, I was thinking on PostgreSQL. It's a relation database and you have the ability to store json and filter by them.
What do you think? Ask me is something isn't clear
You can use HStore extension from PostgreSQL to store JSON, or dynamic columns from MariaDB (MySQL clone).
If you can move your persistence stack to java, then many interesting options are available: mapdb (but it requires memory and its api is changing rapidly), persistit, or mvstore (the engine behind H2).
All these would allow to store json with decent performances. I suggest you use a full text search engine like lucene to avoid searching json content in a slow way.
I find a interesting problem when working with web service in JSON format.
Assume there's web service. accept several parameters. each parameter has different value set. You can get the response by passing different request parameters.
The request is in JSON format. Because there're so many different combination of request parameters. For performance optimization, I want to cache the request and response pair. and store it into local database. If there's big hash table, I may want to store the request as key, the response as value.
I am thinking the MongoDB maybe a solution. But I am not sure. Is it possible to store request-response as key-value pair in these kind of database? So I can cache the result and response to user immediately.
Thank you.
You won't get any benefit from that level of caching unless your code and database have spectacularly bad performance (in which case you have bigger problems than setting up a cache).
You can use JSON as a key with any key/value store, though it probably makes sense to use a hash as the cache key rather than using the JSON string directly, and a non-persistent in-memory cache with memcached or redis will work a lot better than a complete document database like MongoDB.
Where you will run into big problems with this approach is managing cache expiry - to get real time updates, you need to know exactly which cached objects are affected by a change to a given object. That's easy if the request is a simple get by ID, but next to impossible in the scenario you describe.
The other way to manage cache is expiry is to delete objects from the cache after a given time. This assumes that it is acceptable to show stale data after an update. Caches usually have support for expiry built in. Databases generally don't.