My application generates payments to contractors based on a number of different objects on our platform. For example, we have jobs and line items per job. Each of those have their own details.
After we pay out a contractor, sometimes job details are changed around, for whatever reason, so sometimes we'll have a generated (email) receipt that says one thing but the job details queried from the database say something else.
I've been thinking about saving the current job details as JSON as storing it in a text field for each payment, so that we have a "snapshot" of the object at the time it was paid.
a) Does something like that make sense to do? Or would it be better to lock old jobs so they aren't editable anymore?
b) If it does make sense, would it be worth it/useful to store it in something like MongoDB? Would it be less complicated to just keep it all in mysql?
Related
in vue.js app the main focus is working with prospects. prospects have many things like contacts, listings, and half a dozen other objects/tables.
they also have interactions, which could have 30 or more per prospect, while most things like emails or phones would have 1-3 results. I load 50 prospects at a time in to the front end
I'm trying to decide if loading it all into the front end to work 50 prospects at a time is a good idea, or if i should have a json column with interactions as part of the prospects table that i would update each time an interaction is saved, with minimal info like date, type, subject...
it seems like an extra step (and duplicate data, how important is that?) to update the json column with each interaction, but also seems like it would save looking up and loading data all the time
I'm not a programmer, but have been teaching myself how to do something i need done for my business with tutorials and youtube, any opinions from those who deal with this professionally would be appreciated
also, if anyone wants to tell me how to ask this question in a better formatted way, I'm open ears
thanks
Imagine if you have 1000 data, but you are sending only 50 of them, and your user do a filter by price. Will you display only the filtered data from 50 or 1000 of them?
That depends on whether you want to expose all 1000 data to the front end. It's a choice between that, and calling your server api everytime.
If you are calling the server, consider using a cache like Redis to store your results .
Pseudo code.
Request Received
Check Redis Cache - Redis.get('key')
If key exist - return cache.
Else -
check mysql for latest results.
Redis.set('key', latest results);
CreateRequest Received
- Write to mysql
- Redis.delete('key') // next request to view will create new cache with fresh data.
Your key can be anything like, e.g your url ('/my/url')
https://laravel.com/docs/8.x/redis
Right now I have a table called servers which contains all the servers. Every server has chat rooms whose names and chat history are saved in a JSON column called rooms. The column contains an array of room objects like this:
[
{
name: 'General',
history: []
},
{
name: 'RandomRoomName',
history: []
}
]
Right now, whenever a user sends a message, I just push it into the history of the respective room on the server side, however, I don't actually save it in the database so whenever I restart the server, I lose all the history. Now my question is what is the better way of handling this?
Whenever a user sends a message, I get the rooms of the server, push the message into the correct room's history and UPDATE the database with the updated objects.
or
I rework the database by removing the JSON rooms column, creating a table called rooms and another one called messages, then creating the respective relationships/associations between the tables
Honestly, both implementations feel a bit weird and not really optional.
You really need multiple tables. Jamming stuff into a JSON column is not sustainable and will have lots of race condition issues. As the number of messages grows, the expense of re-writing grows as well, making race conditions more likely. This leads into a death spiral where the whole system will melt down under load.
Option 2 is the only realistic way to go. That's the relational form you're looking for. The cost of insertion will grow relatively slowly over time and is usually easily handled up to the billions of records before index sizes get too big to fit in memory.
Currently we're working on a solution where we want to track (for analysis) the articles a user clicks on/opens and 'likes' from a given list of articles. Subsequently, the user needs to be able to see and re-click/open the article (searching is not needed) in a section on his/her personal user profile. Somewhere around the 100 new articles are posted every day. The increasing(!) amount of daily visitor (users) lies around the 2000 a day. The articles are currently stored and maintained within a MySQL Db.
We could create a new record in the MySQL Db for every article read / 'liked'. 'Worst case' would this create (2500 * 100 = ) 250000 records a day. That won’t hold long of course… So how would you store (process) this within XPages, given the scenario?
My thoughts after reading “the article” :) about MIME/Bean’s: what about keeping 'read articleObjects' in a scope and (periodically) store/save them as MIME on the user profile document? This only creates 100 articleObjects a day (or 36500 a year). In addition, one could come up with a mechanism where articleObjects are shifted from one field to another as time passes by, so the active scope would only contain the 'read articleObjects' from last month or so.
I would say that this exactly what a relational database is for. My first approach would be to have a managed bean (session scope) to read/access user's data in MySQL (JDBC). If you want, you can build an internal cache inside the bean.
For the presented use case, I would not bother with the JDBC datasources in ExtLib. Perhaps even the #Jdbc functions would suffice.
Also, you did not say how you are doing the analysis? If you store the information in Domino, you'll probably have to write an export tool.
I am in the process of creating a simple activity stream for my app.
The current technology layer and logic is as follows:
** All data relating to an activity is stored in MYSQL and an array of all activity id's are kept in Redis for every user.**
User performs action and activity is directly stored in an 'activities' table in MYSQL and a unique 'activity_id' is returned.
An array of this user's 'followers' is retrieved from the database and for each follower I push this new activity_id into their list in Redis.
When a user views their stream I retrieve the array of activity id's from redis based on their userid. I then perform a simple MYSQL WHERE IN($ids) query to get the actual activity data for all these activity id's.
This kind of setup should I believe be quite scaleable as the queries will always be very simple IN queries. However it presents several problems.
Removing a Follower - If a user stops following someone we need to remove all activity_id's that correspond with that user from their Redis list. This requires looping through all ID's in the Redis list and removing the ones that correspond to the removed user. This strikes me as quite unelegant, is there a better way of managing this?
'archiving' - I would like to keep the Redis lists to a length of
say 1000 activity_id's as a maximum as well as frequently prune old data from the MYSQL activities table to prevent it from growing to an unmanageable size. Obviously this can be achieved
by removing old id's from the users stream list when we add a new
one. However, I am unsure how to go about archiving this data so
that users can view very old activity data should they choose to.
What would be the best way to do this? Or am I simply better off
enforcing this limit completely and preventing users from viewing very old activity data?
To summarise: what I would really like to know is if my current setup/logic is a good/bad idea. Do I need a total rethink? If so what are your recommended models? If you feel all is okay, how should I go about addressing the two issues above? I realise this question is quite broad and all answers will be opinion based, but that is exactly what I am looking for. Well formed opinions.
Many thanks in advance.
1 doesn't seem so difficult to perform (no looping):
delete Redis from Redis
join activities on Redis.activity_id = activities.id
and activities.user_id = 2
and Redis.user_id = 1
;
2 I'm not really sure about archiving. You could create archive tables every period and move old activities from the main table to an archive table periodically. Seems like a single properly normalized activity table ought to be able to get pretty big though. (make sure any "large" activity stores the activity data in a separate table, the main activity table should be "narrow" since it's expected to have a lot of entries)
Say I have a site with user profiles that have publicly accessible pages (each profile has several pages each). I'd like to show the users page view statistics (e.g. per page, for a certain time period, etc.). What's a good way to store page views?
Here's what I was thinking:
Table Page Views
================
- Id (PK)
- Profile Id (FK)
- Page Id (FK)
- Timestamp
I'm afraid this solution won't scale. Suggestions?
Your intuition is correct, writing to a database doesn't scale particularly well. You want to avoid a database transaction for each page request.
That noted, is scaling really your concern? If so, and assuming a Internet site (as opposed to intra), skip rolling your own and collect the hit data with Google Analytics or something similar. Then take that data and process it to generate totals per profile.
However, if you're really hellbent on doing it yourself, consider log parsing instead. If you can enumerate the URLs per profile, use that information, and your web server logs, to generate hit totals. Tools such as Microsoft's Log Parser, which can process A LOT of different formats, or *nix command line tools like sed and grep are your friends here.
If enumeration's not possible change code to log the information you need and process that log file.
With logs in place, generate results using a batch process and insert those results into a database using MySQL's LOAD DATA.
Final note on the roll your own approach I've recommended - this will scale a lot better if you have a clustered environment than database transaction per request.
It depends on what kind of reports you want to make available.
If you want to be able to say "this is the list of people that viewed your page between these two dates", then you must store all the data you proposed.
If you only need to be able to say "your page was viewed X times between these two dates", then you only need a table with a page ID, date, and counter. Update the counter column on each page view with a single UPDATE query.
I suppose you can have
tblPerson
personid(pk)
activeProfileID(fk) -- the active profile to use.
timestamp
tblPage
pageid(pk)
data
tblPersonProfile
profileID(pk)
timestamp
tblProfilePages
profilePageID(pk)
profileid(pk)
pageid(pk)
isActive