Deserialize a json object for time series analysis - json

Question

I would like to build a table showing the changes in stock prices over 1M, 3M, 6M, etc. The API call for a specific day in the past
returns the following json:
{
"date": "2018-01-02",
"data": {
"AAPL": {
"open": "170.16",
"close": "172.26",
"high": "172.30",
"low": "169.26",
"volume": "25555934"
},
"MSFT": {
"open": "86.13",
"close": "85.95",
"high": "86.31",
"low": "85.50",
"volume": "22483797"
}
}
}
I have built a for loop in java that goes over the required dates using Calendar and make the API calls for those dates and a list of stock symbols. I am able to deserialize the json using the following code:
JsonParser jsonParser = new JsonParser();
JsonObject jsonObject = (JsonObject) jsonParser.parse(result);
JsonElement jsonElement = jsonObject.get("data");
Set<Map.Entry<String, JsonElement>> entrySet = jsonElement.getAsJsonObject().entrySet();
entrySet.parallelStream().forEach(entry -> {
Stock stk = new Stock();
stk.setSymbol(entry.getKey());
stk.setClose(entry.getValue().getAsJsonObject().get("close").getAsFloat());
stk.setDate(date.getKey());
The problem is that using this code I can only save the data in my database that is unique by date. I wish to save it so that is unique by stock symbol. Presumably I need to collect the dates in a map object as a property to the POJO. Unfortunately, I have not been able to make it work. Any suggestions would be very much appreciated.

Answer 1

Create a flat table in your database, with columns for the date, the symbol (such as AAPL), and the five numbers. So seven columns all together.
This strategy assumes you have no additional data to store per date or per stock. (If you did have additional data per date or stock, you would need multiple tables, with parent tables for the date or the stock.)
Personally, I would slap on another column with a UUID generated as a primary key. I believe in always using surrogate key. But not necessary strictly speaking, if your database supports a multi-column combination as a natural key. You could combine the date and the symbol together as a primary key to uniquely identify each row.
CREATE TABLE daily_high_low_ (
date_of_closing_ DATE ,
symbol_ VARCHAR( 4 ) ,
open_ NUMERIC( 12 , 2 ) ,
close_ NUMERIC( 12 , 2 ) ,
high_ NUMERIC( 12 , 2 ) ,
low_ NUMERIC( 12 , 2 ) ,
volume_ INTEGER ,
PRIMARY KEY ( date_of_closing_ , symbol_ )
)
I am purposely keeping this overly simplistic. In real life you would have to worry about stock ticker symbols not being unique across stock exchanges. And I have hard-coded the numeric precision and scale for US dollars and cents amounts, which may not be appropriate for all markets. And we are ignoring the fact that date varies across time zones around the globe.
As you process your JSON, flatten the data to fit this flat table.
For retrieval, make a class to match this database table.
public class HighLow {
public LocalDate dateOfClosing ;
public String symbol ;
public BigDecimal open, close, high, low ;
public Integer volume ;
}
Load from database into a data structure that meets the needs of your data analysis. Perhaps hard-coded maps for each time period: months1, months3, months6, etc. Each maps the symbol to a list of HighLow objects.
To save memory, you share the same objects across the lists. For example, the list in the months3 map has the same HighLow objects from the list in the months1 map, plus two more months worth of HighLow objects.
Map< String , List< HighLow > > months1 = new TreeMap<>() ;
Map< String , List< HighLow > > months3 = new TreeMap<>() ;
Map< String , List< HighLow > > months6 = new TreeMap<>() ;