Conditional loading of large json files with axios.get - json

A large json file is fetched with the below. It works as expected, however, since the case_data.json is large, the page takes up to two minutes to render.
export default {
name: "cases",
data() {
return {
columns: ['Case number', 'Summary', 'Name', 'Address'],
cases: []
};
},
methods: {
fetchData(){
axios.get('/case_data.json').then(response => {
this.cases = response.data;
})
}
},
created(){
this.fetchData();
}
};
It contains cases from several months, so the idea is to break it down into daily or weekly data portions and only import what is required at the time. The expectation is to improve performance. I would like to pass an input parameter to fetchData() and load a smaller dataset. Does the below approach make sense? Is the fetchData() code correct?
export default {
name: "cases",
props: {
day: {
required: true,
type: String
}
},
data() {
return {
columns: ['Case number', 'Summary', 'Name', 'Address'],
cases: []
};
},
methods: {
fetchData(this.day){
if(this.day == 'day1') axios.get('/case_data_day1.json').then(response => {this.cases = response.data;})
else if(this.day == 'day2') axios.get('/case_data_day2.json').then(response => {this.cases = response.data;})
...
else if(this.day == 'dayN') axios.get('/case_data_dayN.json').then(response => {this.cases = response.data;})
}
},
created(){
this.fetchData();
}
};

use the below code instead, its dynamic, you don't have to write manual condition and it will fetch the data upon this.day and use fetchData() on mounted hook.
methods: {
fetchData(){
let that = this
axios.get(`/case_data_${that.day}.json`).then(response => {that.cases = response.data;})
}
},
mounted(){
this.fetchData();
}

Related

Property does not exist on type 'never' on JSON array

I'm just trying to fetch some JSON data from a url. The JSON data is formatted like so (reduced to two entries for simplicity):
[
{
"id": 1
"name": "Brett",
"gender": "male"
},
{
"id": 2
"name": "Sandra",
"gender": "female"
}
]
I can print profiles using console.log(profiles) and see all the entries in the console, but when i try to access the .name field i get the error
Property 'name' does not exist on type 'never'.
Here is the code for the app:
const URL = 'someurl'
function App() {
const [curId, setId] = useState(0);
//const [curProfile, setCurProfile] = useState(undefined);
const [profiles, setProfiles] = useState([])
useEffect(() => {
fetch(URL)
.then((response) => {
if (response.ok) {
return response.json();
} else {
throw new Error("Something went wrong!");
}
})
.then(
(response) => {
setProfiles(response);
setId(1);
//setCurProfile(profiles[curId - 1]);
})
.catch((error) => {
console.log(error)
})
}, []);
return (
<div className="App">
<p>
{profiles[curId].name}
</p>
</div>
);
}
export default App;
Also as a side question, I'm having some problems storing the current profile in the curProfile variable. Could someone point me in the right direction for that? Thanks!
The initial state of profiles is empty array and curId is 0, so profiles[curId] should be undefined thus profiles[curId].name would be error as initial rendering.
You should always check if profiles is empty or not.
return (
<div className="App">
{profiles.length > 0 &&
<p>
{profiles[curId].name}
</p>
}
</div>
)
You've got to type your state, otherwise Typescript won't know what to expect. You also need to type the response.
Something like:
type Profile = {
id: number,
name: string,
gender: string
}
const [profiles, setProfiles] = useState <Profile[]> ([]);
(...)
setProfiles(response as Profile[]);

Segregate and arrange data in specific format in Angular?

Hi I am developing Angular 5 application. I am trying to arrange data in specific format. I have json data. I want to convert it to specific format.
Below is the specific format.
this.nodes = [
{
name: 'root1',
children: [
{ name: 'child1' }
]
},
{
name: 'root2',
hasChildren: true
},
{
name: 'root3'
}
];
Below is my data.
{
"userid":"e75792f8-cfea-460e-aca2-07a778c92a7c",
"tenantid":"00000000-0000-0000-0000-000000000000",
"username":"karthik",
"emailaddress":"john#krsars.onmicrosoft.com",
"isallowed":false,
"userroles":[
{
"userroleid":"b81e63d1-09da-4aa0-af69-0f086ddb20b4",
"userid":"e75792f8-cfea-460e-aca2-07a778c92a7c",
"roleid":"85d2f668-f523-4b64-b177-b1a78db74234",
"tenantappid":1,
"validfrom":"2018-01-24T00:00:00",
"validto":"2018-01-24T00:00:00",
"isactive":true,
}
]
}
From the above data, I am trying to convert. From the above data each key/value pair I am converting it to format above given, For example, "userid":"e75792f8-cfea-460e-aca2-07a778c92a7c" I want to make it as
{
name: 'userid',
children: [
{ name: 'e75792f8-cfea-460e-aca2-07a778c92a7c' }
]
}
So below I is my code.
for (let key in results) {
if(results[key] instanceof Array){
this.nodes+=
name:key,
hasChildren: true
}+"}"
}
else
{
this.nodes+="{"+name=key,
children: [
{ name: results[key] }
]+"}"
}
}
Finally When i tried to display my data in console.
console.log(this.nodes);
Above my code does not work. Can someone help me to make this work? Any help would be appreciated. Thank you.
Here is a working example. Just to show you which way to go:
doIt() {
let results = JSON.parse('{"userid":"e75792f8-cfea-460e-aca2-07a778c92a7c","tenantid":"00000000-0000-0000-0000-000000000000","username":"karthik","emailaddress":"john#krsars.onmicrosoft.com","isallowed":false,"userroles":[{"userroleid":"b81e63d1-09da-4aa0-af69-0f086ddb20b4","userid":"e75792f8-cfea-460e-aca2-07a778c92a7c","roleid":"85d2f668-f523-4b64-b177-b1a78db74234","tenantappid":1,"validfrom":"2018-01-24T00:00:00","validto":"2018-01-24T00:00:00","isactive":true}]}');
const nodes = [];
for (const key in results) {
if (results[key] instanceof Array) {
const containerTyp2 = {name: '', hasChildren: false};
containerTyp2.name = key;
containerTyp2.hasChildren = true;
nodes.push(containerTyp2);
} else {
const object = {name: ''};
const containerTyp1 = {name: '', children: []};
object.name = key;
containerTyp1.name = key;
containerTyp1.children.push(object);
nodes.push(containerTyp1);
}
}
console.log('nodes: ', nodes);
}

how can I update the number of columns in angular-datatables with server-side rendering. version 6.0.0 or higher

I am having trouble getting my angular-datatable to show a new column list after a rerender. I have followed the example shown in the docs for rerendering and I can get the table to rerender. I am able to manipulate certain features like searching and pageLength, but for some reason I cannot get my columns to change.
I have a very deep data set that would make my table look awful if I rendered all the columns at once, so I would like to give users the ability to select which columns they see.
I would even be open to loading in all the columns at once and just switching visibility off and on, but I cannot effect visibility either.
Has anyone had this issue before?
Rerender function:
rerender(): void {
this.dtElement.dtInstance.then((dtInstance: DataTables.Api) => {
dtInstance.destroy();
// these work
this.dtOptions.searching = true;
this.dtOptions.pageLength = 2;
// these do not
this.dtOptions.columns = newColumnList;
this.dtOptions.columns[some-index].visible = false;
this.dtTrigger.next();
});}
Initial dtOptions:
this.dtOptions = {
searching: false,
pagingType: 'full_numbers',
pageLength: 10,
retrieve: true,
serverSide: true,
processing: true,
language: {
zeroRecords: 'Nothing Found'
},
ajax: (dataTablesParameters: any, callback) => {
const payload = this.passFilterService.processPagination(this.filter, dataTablesParameters);
this.http
.post<any>(
environment.api + '/things/list',
{payload: payload}, {}
).subscribe(resp => {
if (resp.data.data === null) {
resp.data.data = 0;
}
callback({
recordsFiltered: resp.data.totalCount,
data: resp.data.data,
recordsTotal: resp.data.totalCount
});
});
},
columns: this.tableColumns
};
Initial Columns (limited fields):
tableColumns = [
{
title: 'Customer',
data: 'Id',
render: function(data) {
return `Action`;
}
}, {
title: 'Created',
data: 'createdAt',
orderable: true,
visible: true,
}, {
title: 'Updated',
data: 'updatedAt',
orderable: true,
visible: true,
}, {
title: 'Disabled',
data: 'isVoided',
orderable: true,
visible: true,
}
];
Table implementation:
<table datatable [dtOptions]="dtOptions" [dtTrigger]="dtTrigger" class="row-border hover">
</table>
I faced the same issue, spent hours debugging it until I found something that worked for me. I will advice separating the DT config into an independent object that can be loaded separately. Once you update your DT options and any other config, you can use the functions below to reload the entire DT, destroying and reloading it accordingly;
async rerender(newSettings?: DataTables.Settings) {
try {
this.dtElement.dtInstance.then((dtInstance: DataTables.Api) => {
if (newSettings) {
// FIX To ensure that the DT doesn't break when we don't get columns
if (newSettings.columns && newSettings.columns.length > 1) {
dtInstance.destroy();
this.dtOptions = Promise.resolve(newSettings);
this.displayTable(this.dtTableElement);
}
}
});
} catch (error) {
console.log(`DT Rerender Exception: ${error}`);
}
return Promise.resolve(null);
}
This function calls the below one to actually destroy the DT and rerender it.
private displayTable(renderIn: ElementRef): void {
this.dtElement.dtInstance = new Promise((resolve, reject) => {
Promise.resolve(this.dtOptions).then(dtOptions => {
// Using setTimeout as a "hack" to be "part" of NgZone
setTimeout(() => {
$(renderIn.nativeElement).empty();
var dt = $(renderIn.nativeElement).DataTable(dtOptions);
// this.dtTrigger.next();
resolve(dt);
});
}).catch(error => reject(error));
});
}
I removed the dtTrigger execution from the reconstruction function as this was executing twice.
The dtTableElement is defined as #ViewChild('dtTableElement') dtTableElement: ElementRef; where the HTML contains the respective reference on the datatable as:
<table #dtTableElement datatable [dtOptions]="dtOptions" [dtTrigger]="dtTrigger" class="table table-striped row-border hover" width="100%"></table>

Redux and Calendar repeating events

What should be the proper way of storing / handling repeating events in the redux store ?
Problem: Let's say that we have a backend API that generates repeating events trough a complicated business logic.Some of the events might have the same ID. Lets say that generated output looks this way :
[
{
"id": 1,
"title": "Weekly meeting",
"all_day": true,
"starts_at": "2017-09-12",
"ends_at": "2017-09-12"
},
{
"id": 3,
"title": "Daily meeting1",
"all_day": false,
"starts_at": "2017-09-12",
"ends_at": "2017-09-12",
},
{
"id": 3,
"title": "Daily meeting1",
"all_day": false,
"starts_at": "2017-09-13",
"ends_at": "2017-09-13",
},
{
"id": 3,
"title": "Daily meeting1",
"all_day": false,
"starts_at": "2017-09-14",
"ends_at": "2017-09-14",
}
]
Possible solution would be: generate unique ID by having additional property uid composed like this: id + # + starts_at. This way we could identify each occurrence uniquely. (I'm using this right now)
Example:
[
{
"id": 1,
"uid": "1#2017-09-12",
"title": "Weekly meeting",
"all_day": true,
"starts_at": "2017-09-12",
"ends_at": "2017-09-12"
}
]
I'm wondering is there some other way, maybe more elegant than having composed unique id ?
There is a possible pitfall with your current solution. What will happen if id and start_id of two events will be the same? Is it possible scenario in your domain?
Because of that I usually use this nice lib in such cases. It produces really short unique ids, which have some nice properties, like guaranties not to intersect, to be unpredictable and so on.
Also ask yourself if you actually need unique ids in your case. Looks like your back-end have no chance to distinguish events anyways, so why bother? Redux store will happily keep your events event without uid.
Maybe not much of an improvement (if at all) but just using JSON.stringify to check for duplicates could make unique id's obsolete.
const existingEvents = [
{
"id": 3,
"title": "Daily meeting1",
"all_day": false,
"starts_at": "2017-09-14",
"ends_at": "2017-09-14",
}
];
const duplicate = {
"id": 3,
"title": "Daily meeting1",
"all_day": false,
"starts_at": "2017-09-14",
"ends_at": "2017-09-14",
};
const eventIsDuplicate = (existingEvents, newEvent) => {
const duplicate =
existingEvents.find(event => JSON.stringify(event) == JSON.stringify(newEvent));
return typeof duplicate != 'undefined';
};
console.log(eventIsDuplicate(existingEvents, duplicate)); // true
I guess this would only be preferable to your existing solution if, for some reason, you'd want to keep all the uniqueness logic on the client side.
As far as I understand the examples you've given, it seems like the server is sending a particular event whenever the details of the event change.
If that is so, and you want to track the changes to events, your might shape might be an array of objects with all the fields of the event that hold the current data, and a history property which is an array of all previous (or n most recent) event objects and the timestamps at which they were received. This is how your reducers would look, storing only the five most recent event changes for each event. I'm expecting the action to have a payload property which has your standard event property and a timestamp property, which can be easily accomplished in the action creator.
const event = (state = { history: [] }, action) => {
switch (action.type) {
case 'EVENT_FETCHED':
return ({
...action.payload.event,
history: [...state.history, action.payload].slice(-5),
});
default:
return state;
}
};
const events = (state = { byID: {}, IDs: [] }, action) => {
const id = action.payload.event.ID;
switch (action.type) {
case 'EVENT_FETCHED':
return id in state.byID
? {
...state,
byID: { ...state.byID, [id]: event(state.byID[id], action) },
}
: {
byID: { ...state.byID, [id]: event(undefined, action) },
IDs: [id],
};
default:
return state;
}
};
Doing this, you don't need any unique ID. Please let me know if I have misunderstood your problem.
Edit: This is a slight extension of the pattern in the Redux documentation, to store previous events.
At the end this is what I've implemented (for demonstration purpose only - unrelated code is omitted):
eventRoot.js:
import { combineReducers } from 'redux'
import ranges from './events'
import ids from './ids'
import params from './params'
import total from './total'
export default resource =>
combineReducers({
ids: ids(resource),
ranges: ranges(resource),
params: params(resource)
})
events.js:
import { GET_EVENTS_SUCCESS } from '#/state/types/data'
export default resource => (previousState = {}, { type, payload, requestPayload, meta }) => {
if (!meta || meta.resource !== resource) {
return previousState
}
switch (type) {
case GET_EVENTS_SUCCESS:
const newState = Object.assign({}, previousState)
payload.data[resource].forEach(record => {
// ISO 8601 time interval string -
// http://en.wikipedia.org/wiki/ISO_8601#Time_intervals
const range = record.start + '/' + record.end
if (newState[record.id]) {
if (!newState[record.id].includes(range)) {
// Don't mutate previous state, object assign is only a shallow copy
// Create new array with added id
newState[record.id] = [...newState[record.id], range]
}
} else {
newState[record.id] = [range]
}
})
return newState
default:
return previousState
}
}
There is also a data reducer but it's linked in parent reducer due to generic implementation that is re-used for common list responses. Events data are updated and start/end property is removed as it's composed by range (ISO 8601 time interval string). This can be later used by moment.range or split by '/' to get start/end data. I've opted for array of range strings to simplify checking of existing ranges, as they might grow large. I think that primitive string comparison (indexOf or es6 includes) would be faster than looping over complex structure in such cases.
data.js (stripped down version):
import { END } from '#/state/types/fetch'
import { GET_EVENTS } from '#/state/types/data'
const cacheDuration = 10 * 60 * 1000 // ten minutes
const addRecords = (newRecords = [], oldRecords, isEvent) => {
// prepare new records and timestamp them
const newRecordsById = newRecords.reduce((prev, record) => {
if (isEvent) {
const { start, end, ...rest } = record
prev[record.id] = rest
} else {
prev[record.id] = record
}
return prev
}, {})
const now = new Date()
const newRecordsFetchedAt = newRecords.reduce((prev, record) => {
prev[record.id] = now
return prev
}, {})
// remove outdated old records
const latestValidDate = new Date()
latestValidDate.setTime(latestValidDate.getTime() - cacheDuration)
const oldValidRecordIds = oldRecords.fetchedAt
? Object.keys(oldRecords.fetchedAt).filter(id => oldRecords.fetchedAt[id] > latestValidDate)
: []
const oldValidRecords = oldValidRecordIds.reduce((prev, id) => {
prev[id] = oldRecords[id]
return prev
}, {})
const oldValidRecordsFetchedAt = oldValidRecordIds.reduce((prev, id) => {
prev[id] = oldRecords.fetchedAt[id]
return prev
}, {})
// combine old records and new records
const records = {
...oldValidRecords,
...newRecordsById
}
Object.defineProperty(records, 'fetchedAt', {
value: {
...oldValidRecordsFetchedAt,
...newRecordsFetchedAt
}
}) // non enumerable by default
return records
}
const initialState = {}
Object.defineProperty(initialState, 'fetchedAt', { value: {} }) // non enumerable by default
export default resource => (previousState = initialState, { payload, meta }) => {
if (!meta || meta.resource !== resource) {
return previousState
}
if (!meta.fetchResponse || meta.fetchStatus !== END) {
return previousState
}
switch (meta.fetchResponse) {
case GET_EVENTS:
return addRecords(payload.data[resource], previousState, true)
default:
return previousState
}
}
This can be then used by an calendar component with event selector:
const convertDateTimeToDate = (datetime, timeZoneName) => {
const m = moment.tz(datetime, timeZoneName)
return new Date(m.year(), m.month(), m.date(), m.hour(), m.minute(), 0)
}
const compileEvents = (state, filter) => {
const eventsRanges = state.events.list.ranges
const events = []
state.events.list.ids.forEach(id => {
if (eventsRanges[id]) {
eventsRanges[id].forEach(range => {
const [start, end] = range.split('/').map(d => convertDateTimeToDate(d))
// You can add an conditional push, filtered by start/end limits
events.push(
Object.assign({}, state.events.data[id], {
start: start,
end: end
})
)
})
}
})
return events
}
And here is how the data structure looks in the redux dev tools:
Each time the events are fetched, their data is updated (if there is a change) and references are added. Here is an screenshot of redux diff after fetching new events range:
Hope this helps somebody, I'll just add that this still isn't battle tested but more a proof of a concept that's working.
[EDIT] Btw. I'll probably move some of this logic to the backend as then there will be no need to split / join / delete properties.

Create keyed Maps from nested Lists with Immutable.js

I am working with a dataset that cannot be modified on the server side. So I am trying to setup the local data model on the client in a way that I can easily traverse through the model when updating parts of the data.
Therefore I am trying to create a multi-leveled Map from multi-leveled Maps including Lists, that themselves include Maps, etc. (see schematics at the end of this post).
What I am trying to get is a Map containing other Maps, with the key of the included Map being the value of the object (again please see schematics at the end of this post).
I got it to work on the first level:
const firstLevel = data.toMap().mapKeys((key, value) => value.get('value'));
See it in action here: https://jsfiddle.net/9f0djcb0/4/
But there is a maximum of 3 levels of nested data and I can't get my head around how to get the transformation done. Any help appreciated!
The schematic datasets:
// This is what I got
const dataset = [
{
field: 'lorem',
value: 'ipsum',
more: [
{
field: 'lorem_lvl1',
value: 'ispum_lvl1',
more: [
{
field: 'lorem_lvl2',
value: 'ispum_lvl2',
more: [
{
field: 'lorem_lvl3',
value: 'ispum_lvl3',
}
]
}
]
}
]
},
{
field: 'glorem',
value: 'blipsum'
},
{
field: 'halorem',
value: 'halipsum'
}
];
This is where I want to go:
// This is what I want
const dataset_wanted = {
ipsum: {
field: 'lorem',
value: 'ipsum',
more: {
lorem_lvl1: {
field: 'lorem_lvl1',
value: 'ispum_lvl1',
more: {
lorem_lvl2: {
field: 'lorem_lvl2',
value: 'ispum_lvl2',
more: {
lorem_lvl3: {
field: 'lorem_lvl3',
value: 'ispum_lvl3',
}
}
}
}
}
}
},
glorem: {
field: 'glorem',
value: 'blipsum'
},
halorem: {
field: 'halorem',
value: 'halipsum'
}
};
Retrieve nested structures using "getIn" is beter.
const data = Immutable.fromJS(dataset[0]);
const firstLevel = data.getIn(['more']);
const twoLevel = firstLevel.getIn([0,'more']);
const threeLevel = twoLevel.getIn([0,'more']);
console.log(firstLevel.toJS(),twoLevel.toJS(),threeLevel.toJS());
As for a more generative solution, I re-wrote the answer before to a recursive approach:
function mapDeep(firstLevel) {
return firstLevel.map((obj) => {
if (obj.has('more')) {
const sec = obj.get('more').toMap().mapKeys((key, value) => value.get('value'));
const objNext = mapDeep(sec);
obj = obj.set('more', objNext);
}
return obj;
});
}
The first level still needs to be mapped manually before.
const firstLevel = data.toMap().mapKeys((key, value) => value.get('value'));
const secondLevel = mapDeep(firstLevel);
Again, see it in action: https://jsfiddle.net/9f0djcb0/12/
This is good enough for me for now. Still feels like this can be solved smarter (and more performant).. Cheers :)
So after some time passed I came up with a solution that works for me:
let sec, third, objThird;
// 1st level: simple mapping
const firstLevel = data.toMap().mapKeys((key, value) => value.get('value'));
// 2nd level: walk through updated firstLevel's subobjects and do the mapping again:
const secondLevel = firstLevel.map((obj) => {
if (obj.has('more')) {
sec = obj.get('more').toMap().mapKeys((key, value) => value.get('value'));
// 3nd level: walk through updated secondLevel's subobjects and do the mapping again:
objThird = sec.map((o) => {
if (o.has('more')) {
third = o.get('more').toMap().mapKeys((key, value) => value.get('value'));
o = o.set('more', third);
}
return o;
});
obj = obj.set('more', objThird);
}
return obj;
});
See it in action here: https://jsfiddle.net/9f0djcb0/7/
This has been working nicely so far, thur pretty hard-coded. If anyone has a more elegant solution to this, I am happy to learn about it!