I'm developing a marketplace-style application that allows users to upload purchasable digital items -> the public purchases these items -> and for my application to pay the users (owners of items) their owed funds via PayPal Payouts API on a daily basis.
I'm struggling with how best to calculate/store the owing balance, and how to map the individual purchase transaction records to the concept of a "payout" (when we send owed funds to the user).
Schema so far:
User
id
name
createdAt
etc.
Purchasable Item
id
user_id (owner)
price
createdAt
etc.
Transaction
id
type ("purchase" or "payout")
status (depending on PayPal response. COMPLETED, FAILED, REFUNDED etc.)
value (integer (lowest demomination of currency). Positive integer for purchase, negative for a payout).
purchasable_id (For "purchase" transactions, reference the ID of the purchasable item that was purchased)
transaction_fee
createdAt
payout_id (?) The ID of the payout (below) this purchase is included in. Not sure about this. This won't be known at the time of the transaction, so it would need to be updated to store it and I'm not sure how to know which transaction will belong in which payout?
Payout
Not sure about this. Feels like a duplicate of a payout transaction entry, but I want a way to store which purchase transactions were paid out in which payouts.
id
status (depending on PayPal response to Payout API webhook. COMPLETED, FAILED, REFUNDED etc.)
createdAt
Logic:
This is where I need the most help.
CRON job. Every 24hrs:
Calculate each users balance by summing the payout_balance_change fields of the Transactions table. i.e balance isn't stored, it's always calculated. Is that a good idea?
Insert a row into "Transactions" of type "payout" with a negative "payout_balance_change". i.e. subtracting the amount we will send in the payout, zeroing their balance in the Transactions table.
Insert a row into "Payouts" table that stores the details of the payout attempt.
Problems:
How will I know which purchase transactions belong to each payout cycle (so I can then store the payout_id in those transaction records). I could use the date of the transaction, and each payout could be for the 24hr period prior to the CRON job? I'm flexible on this and not sure what the most robust logic would be.
Any advice on how best to structure this, or links to similar projects would be greatly appreciated.
Thank you!
and welcome to Stack Overflow.
This question may be a too wide for this format - please do read "how to ask".
Firstly - I'm answering on the assumption this is MySQL. Again - please read "how to ask", tagging with multiple technologies isn't helpful.
Fistly - this on how to store money in MySQL.
Secondly - the common pattern for doing this is to have the transaction table only reflect complete transactions. That way, the current balance is always sum(transaction_value), with the transaction date showing you the balance at a given point in time. You typically store the interim state for each transaction in a dedicated table (e.g. "payout"), and only insert into the transaction table once that payout transaction is complete.
You should remove all the status and transaction_fee references from the transaction table, and store them in the dedicated tables. A transaction_fee can be represented as a transaction in its own right.
If you want to store the relationship between purchase and payout, you might have something like:
Payout
Payout_id
Payout_amount
Payout_status
Payout_date
...
Purchase
Purchase_id
Customer_id
Item_id
Purchase_date
....
Payout_purchase
Purchase_id
Payout_id
Your logic then becomes:
cron job searches all purchases that haven't been paid out (where purchase_id not in (select purchase_id from payout_purchase)
for each vendor:
create new record in payout_purchase
find sum of new payout_purchase records
attempt payout
if (payout succeeded)
insert record into transaction table with vendor ID, payout ID and payout amount
else
handle error case. This could be deleting the record (and logging the failure somewhere else), or by adding a "status" column with the value "failed". The latter option makes it easier to provide your vendors with a statement - "we attempted to pay you, but the payment failed". Either way, you want to have a way of monitoring failures, and monitor them.
end if
next vendor
I've left out the various state and error management logic steps.
Things you want to worry about:
What happens if a purchase occurs while the payout logic is running? You need to make sure you work on defined data sets in each step. For instance, you need to insert data into the "payout_purchase" table, and then work only on those records - new purchases should not be included until the next run.
What happens if a payout fails? You must ensure they are included in the next payment run.
How do you provide a statement to your buyers and sellers? What level of detail do you want?
Transaction management from MySQL may help, but you need to spend time learning the semantics and edge cases.
Related
Basic question of language ability.
I am developing a database to keep track of market trades and provide useful metrics to the user. Most brokers do not supply enough information in the transaction .csv file which would be imported to this database to combine strategies and positions in a useful way, or in a way that I envision can be useful for users. For instance, combining a buy order on AAPL of 1000 shares that was filled in three separate transactions on the same order (1 order for 1000 shares filled initially with 200, then 350, then 450 shares). I could only think of assigning a trade group to each of these so that I can group them.
So, the example above, each transaction would be a separate record, I've created a column in my table with the alias of Trade Group, and each transaction would be assigned 1 under the Trade Group column. The sale of the 1,000 shares, no matter how many transactions it took to fill the order, would also be assigned to trade group 1.
My query combines the shares for both opening and closing transactions by using the trade group and transaction type (buy or sell). If there is a match, 1,000 shares on the buy side, and 1,000 shares on the sell side, then it runs some queries to provide useful data about the trade.
The problem I foresee is that the trade grouping can become cumbersome since it currently has to be manually input. I would like to develop a counter to automatically increment the trade group of each the opening and closing transactions every time the balance of shares = 0.
So if both the buy and sell of the above example belonged to trade group 1, and now I decide to open up a position of 2000 shares of AAPL, and subsequently sell them, those transactions would be automatically assigned trade group 2. And now that the balance of shares is 0 again, the next time I open and close a position on AAPL it will be assigned trade group 3.
That way, I don't need to clutter up my table with something that is manually input which can create mistakes. Instead, the query assigns the trade grouping every time it is run, and the necessary metrics supplied.
Is this something that can be done using SQL alone?
Thanks.
I'm creating an Intranet for my company, and we want to have a stock management in it. We sell and rent alarm systems, and we want to have a good overview of what product is still in our offices, what has been rented or sold, at what time, etc.
At the moment I thought about this database design :
Everytime we create a new contract, this contract is about a location or a sale of an item. So we have an Product table (which is the type of product : alarms, alarm watches, etc.), and an Item table, which is the item itself, with it unique serial number. I thought about doing this, because I'll need to have a trace of where a specific item is, if it's at a client house (rented), if it's sold, etc. Products are related to a specific supplier, to whom we can take orders. But here, I have a problem, shouldn't the order table be related to Product ?
The main concern here is the link between Stock, Item, Movement stock. I wanted to create a design where I'd be able to see when a specific Item is pulled out of our stock, and when it enters the stock with the date. That's why I thought about a Movement_stock table. The Type_Movement is either In / Out.
But I'm a bit lost here, I really don't know how to do it nicely. That's why I'm asking for a bit of help.
I have the same need, and here is how I tackled your stock movement issue (which became my issue too).
In order to modelize stock movement (+/-), I have my supplying and my order tables. Supplying act as my +stock, and my orders my -stock.
If we stop to this, we could compute our actual stock which would be transcribed into this SQL query:
SELECT
id,
name,
sup.length - ord.length AS 'stock'
FROM
product
# Computes the number of items arrived
INNER JOIN (
SELECT
productId,
SUM(quantity) AS 'length'
FROM
supplying
WHERE
arrived IS TRUE
GROUP BY
productId
) AS sup ON sup.productId = product.id
# Computes the number of order
INNER JOIN (
SELECT
productId,
SUM(quantity) AS 'length'
FROM
product_order
GROUP BY
productId
) AS ord ON ord.productId = product.id
Which would give something like:
id name stock
=========================
1 ASUS Vivobook 3
2 HP Spectre 10
3 ASUS Zenbook 0
...
While this could save you one table, you will not be able to scale with it, hence the fact that most of the modelization (imho) use an intermediate stock table, mostly for performance concerns.
One of the downside is the data duplication, because you will need to rerun the query above to update your stock (see the updatedAt column).
The good side is client performance. You will deliver faster responses through your API.
I think another downside could be if you are managing high traffic store. You could imagine creating another table that stores the fact that a stock is being recomputed, and make the user wait until the recomputation is finished (push request or long polling) in order to check if every of his/her items are still available (stock >= user demand). But that is another deal...
Anyway even if the stock recomputation query is using anonymous subqueries, it should actually be quite fast enough in most of the relatively medium stores.
Note
You see in the product_order, I duplicated the price and the vat. This is for reliability reasons: to freeze the price at the moment of the purchase, and to be able to recompute the total with a lot of decimals (without loosing cents in the way).
Hope it helps someone passing by.
Edit
In practice, I use it with Laravel, and I use a console command, which will compute my product stock in batch (I also use an optional parameter to compute only for a certain product id), so my stock is always correct (relative to the query above), and I never manually update the stock table.
This is an interesting discussion and one that also could be augmented with stock availability as of a certain date...
This means storing:
Planned Orders for the Product on a certain date
Confirmed Orders as of a certain date
Orders Delivered
Orders Returned (especially if this is a hire product)
Each one of these product movements could be from and to a location
The user queries would then include:
What is my overall stock on hand
What is due to be delivered on a certain date
What will the stock on hand be as of a date overall
What will the stock on hand be as of a date for a location
The inventory design MUST take into account the queries and use cases of the users to determine design and also breaking normalisation rules to provide adequate performance at the right time.
Lots to consider and it all depends on the software use cases.
Okay, I've searched and can't find an answer to what is probably simple, but eluding me.
I have a table of virtual currency transactions. Some of them are purchasing virtual items and some are purchasing virtual currency (via real-money transactions). I would like to monitor the in-game transactions a player makes after making a real-money transaction.
Simple schema:
userId
timestamp
transactionType (real-money or in-game)
itemId (of the in-game item they bought)
value (positive for real-money transactions that add to the user's balance, negative for in-game purchases)
newBalance (the user's new currency balance)
It's easy enough to tally the purchases made within, say 10 minutes of purchase using a self-join. What I'd like to do is give each user a balance based on the value of the currency they purchased, then tally the purchases they made using that currency until that balance runs out.
I've got to think there's an easy way to do this using user-defined variables, but it's escaping me.
Thanks in advance!
This is not a programming question at all. Let me explain: I am creating a game, for which I need a Database that will store users registration data (e.g. username, email, password). If a user wins, he/she will earn Cash Points. The user will be able to exchange his/her Cash Points by real money, so I consider that the Cash Points is a very, very critical data. My question is: would you store the Cash Points data in the same "users" table? Or would you create a new table named "cash" (for instance) and then store it into? (from a security point of view)
Thanks
It's best if you implement a simple ledger system whereby transactions are recorded against the user's account as credit or debits, and the account itself has a total that can be audited.
You must keep a record of transactions performed if you're involving cash or cash-like currency. If someone complains about missing money you need to be able to verify every transaction that affected their balance and uncover any discrepancies.
This also presumes you're making use of transactions to avoid committing incomplete transactions. The balance adjustment and transaction record should be part of the same transaction.
As always, test this as ruthlessly as you can.
It is considered bad design if you store cash points in the users table. Tables should be normalized. You should store cash points in a separate table and use the userId as the foreign key in that table. You could look into encrypting Cash Points table data as well.
Cashpoints definitely in a separate table but not from security perspective. It's better from design perspective and will allow you to keep a log of CashPoint changes for each user.
Well you should create a database design that resembles a bank balance. That way you can keep track of all changes, this is
create table balance
(id int,
debit numeric (10,2),
credit numeric (10,2),
balance_before numeric(10,2),
balance_after numeric(10,2),
timestamp datetime,
user_id int,
description varchar(32),
...
);
Currently our database is set up so that a payment transactions records a payment type ID, and this links to a payment type (cash, check, credit) table that contains these values. Example:
Payment Transaction:
ID
Amount
Date
Payment Type ID
Payment Type:
ID
Payment Type (Cash, Credit)
My question is whether or not I should just remove the payment type table, and just store the payment type value as text inside the payment transaction.
This is similar to this question. except with payment types it's pretty certain that no new information will ever need to be add data per payment type. 'Cash' doesn't link to anything, there's nothing I need to know about Cash itself, it just is.
As far as I can tell the pros and cons would of replacing the payment type table with a single field would be:
Pros
Removes a mostly unnecessary join whenever the payment type needs to be found.
The payment type for a transaction will always accurately reflect what it was at the time the transaction was recorded. i.e. If I change the 'Cash' record in the payment types table to 'Credit' (for whatever reason), all payment transactions that link to Cash will now be linked to Credit.
Cons
Storing the payment type as a text field will slow down sorting by payment type, and make such a sort somewhat messier than it is now.
The payment type for a transaction will always accurately reflect what it was at the time the transaction was recorded. i.e. If I had a typo and the payment type was stored as 'Kash', I could easily fix that typo and all transactions that link to that payment type will automatically be updated.
I'm leaning towards removing the payment type table and adding the single field to the payment transaction table, what do you recommend would be the best course of action?
I don't agree with either of your pro arguments.
Removes a mostly unnecessary join whenever the payment type needs to
be found.
There's just your assumption that this will be a performance bottleneck. Denormalization is something you should do when you have data that says you must. This isn't one of those times.
The payment type for a transaction will always accurately reflect what
it was at the time the transaction was recorded. i.e. If I change the
'Cash' record in the payment types table to 'Credit' (for whatever
reason), all payment transactions that link to Cash will now be linked
to Credit.
You should not allow someone to modify the payment type this way. Changing the payment type should be another transaction, with its own timestamp.
Any relational database can handle the JOIN and the normalized tables. You're guilty of premature optimization, I fear.
I'd spend less time worrying about this and more time thinking about how you'll deal with history. How long will you keep transactions around before moving them out to a history table? Have you thought about partitioning your database by months according to timestamp? That would be more worthy of your efforts.
If you remove the PaymentType table, you replace a foreign key check with a table CHECK constraint:
PaymentType CHAR(6) NOT NULL CHECK(PaymentType IN('Cash', 'Credit', 'Cheque')
OK — you write 'cheque' as 'check'; just another difference between English and American.
Now, this makes it much more difficult to find out what the possible values are; you have to analyze the system catalog to find out. With the separate table, you can examine the separate table to find out what's allowed. Suppose you start tracking 'Debit' separately from 'Credit'; you add a row to a table, vs a table schema change. Suppose you decide you need to record which codes are allowed in future transactions (so 'Cash' ceases to be an option). You can add a column to the Payment Type table to indicate that this code is no longer valid; it is much harder to do that with a simple CHECK constraint.
So, even though you currently have limited or no extra data in the Payment Type table, I would use the Payment Type table rather than embedding the payment type in the Payment Transaction table.
Were it my design, though, I'd probably use a CHAR(1) or CHAR(2) code as the identifier for the payment type, rather than a numeric column. Granted, all three types start with 'C', so maybe you'd use 'A' for cAsh, 'H' for cHeck, and 'R' for cRedit (and maybe 'D' or 'E' for Debit or dEbit) with a CHAR(1) code; with CHAR(2), you'd use 'CA', 'CH', 'CR' (and maybe 'DE'). The full name can be stored in the payment type table for use in reports. In this case, the benefits are not huge, but saving 4 bytes per record on enough records (a large enough number of small enough records) and it can become a factor in your storage costs. Of course, index overhead also comes into play; if the column in the Payment Transaction table must be indexed, then the smaller field uses less index space.