I am going to explain the project I am currently working on and please bear with me since English is not my first language.
The project I am working on involves a series of devices connected to the service Watson IOT that periodically (about once every second, not a constant refresh rate though) send data. The data I am talking about comes from many sensors that are very different from each other and are all in the same device. Since they are different, some send data very frequently and some others very rarely. For example I have a temperature sensor that sends data pretty much every second and a 'fault' sensor that sends data only if something goes wrong.
{
"deviceId": "Oven0000",
"data": {
"h": 41.6,
"utctime": 1502222544044
}
}
As you can see from the example above, I just have a "h" parameter and a "utctime" that are saved, not the whole device state.
{
"deviceId": "Oven0000",
"data": {
"utctime": 1502269142953,
"A": 17,
"F": true,
"G": false,
"I": true,
"J": true,
"K": true,
"W": false,
"a": 11,
"b": 4,
"c": 0,
"e": false,
"f": true,
"g": 14,
"h": 60.3,
"q": 1,
"r": 1,
"t": 1,
"x": true,
"z": true
}
}
This example is a full state where all the data is added.
All of this info is then sent from Watson IOT to a Cloudant instance in which it is automatically saved in month indexed databases. Right now I just have a back-end that is responsible for taking all docs from Cloudant, adding to each doc the whole device state, and sending it to a frontend which then visualizes them (see example above).
The backend is written using NodeJS, whilst the frontend is written using Vue.js and some graphical libraries.
Right now, I would like to implement some sort of data analysis on my docs, but not the ones that are stored in the Cloudant DBs since they only have single values. Rather, I'd like to conduct an analysis of the ones that the backend creates, the ones with the full state of the device.
I have thought of having a secondary database, maybe a relational one, where I can send my parsed data from the backend and make some queries.
What do you think of this approach? Is there a better solution? Would you recommend some other types of DBMS?
Have you considered using the Data Management capabilities of the Watson IoT Platform to maintain the state of your device for you? This functionality allows you to define the structure of the state using JSON schema and define mappings that will aggregate the data from different inbound events into a single view of the state.
Related
I'm building API for SinglePageApplication, which handle by Angular in frontend. One thing is not clear to me.
Supose the web applcation has delati journal paige wich display journal,some articles which belongs to this journal and some cool authors which can be not connected to this journal.
Should I build my api urls based on each need page content, for example:
from url /api/journal/<journal_id>
send json:
{
"journal": {
"id": 10,
"name": "new_journal",
"articles": [
{
"name": "cool_article",
"id": 42
},
{
"name": "another_cool_article",
"id": 43
}
]
},
"authors": [
{
"name": "some_name",
"id": 42
},
{
"name": "another_name",
"id": 43
}
]
}
Or I should build my api based on concrete objects and related objects of them.
With urls like this:
/api/journals/<journal_id>
/api/authors/
And frontend side build this page with two GET requests for fetching content.
Sory if my question too broad, I just want to find best bractice for building API to SinglePageApplications.
Does it have any difference of building API enpoints for external web-apps and what I should do if page need to display more objects, which not belong together? Which of the options above is better?
There isn't really a universal right answer for this. It largely depends on the use case for that data you're fetching. I would say to err on the side of splitting this into multiple requests as it grants you flexibility and efficiency in terms of partial updates to the page. That approach also makes exposing an API to the public much easier in terms of being able to just expose what you already have.
If you're dealing with a potentially large (an intentionally relative term) number of concurrent requests though, you may build some composites of related data to mitigate that.
Of course, you could also do a combination of the two as well (first load makes 1 large request, subsequent updates are segmented).
The problem is even if I put condValues to PT10S, when I send request to contextBroker it requests back the reference url rigth away, not after 10 sec, and then it continues to send requests at 10 sec.
My question: is there a way to avoid the first initial request?
Here is a body of the request that I send to server where contextBroker is installed.
{
"entities": [{
"type": "Cycle",
"isPattern": "false",
"id": "someid"
}],
"attributes": [
...
],
"reference": "someurl"
"duration": "P1M",
"notifyConditions": [{
"type": "ONTIMEINTERVAL",
"condValues": [
"PT10S"
]
}]
}
At the present moment (Orion 1.1) initial notification cannot be avoided. However, being able to configure that behaviour would be an interesting feature to develop in the future and, consecuently, a github issue was created time ago about it.
In addition, note that ONTIMEINTERVAL subscriptions are no longer supported so you should avoid to use them:
ONTIMEINTERVAL subscriptions have several problems (introduce state in CB, thus making horizontal scaling configuration much harder, and makes it difficult to introduce pagination/filtering). Actually, they aren't really needed, as any use case based on ONTIMEINTERVAL notification can be converted to an equivalent use case in which the receptor runs queryContext at the same frequency (and taking advantage of the features of queryContext, such as pagination or filtering).
EDIT: the posibility of avoiding initial notification has been finally implemented at Orion. Details are at this section of the documentation. It is now in the master branch (so if you use fiware/orion:latest docker you will get it) and will be include in next Orion version (2.2.0).
I have developed a web application based on RESTFul design where the application takes JSON responce from JAVA based web service and displays in UI and it refreshes the data in every 5 seconds.
The application uses Bootstrap for UI design, Backbone and require.js for implementing an MVC stucture where JSON response is parsed as Backbone collection.
When an admin is using this application the JSON response size is too large(from 800 to 1100 objects).
This is where things get messy. As per my analysis the browser is taking up too much resource.So rest of the application is very slow. For eg if I try to open a modal, system freezes for some time and opens slowly thus giving a very poor user experience.
As per my analysis time is being taken in parsing the data
As a remedy I am removing all comments in code and trying to implement Gzip compression for JSON files/html/css/js.
Sample of the JSON object is pasted below
{
"name": "TEST",
"state": "Lunch",
"time": "00:00:09",
"manager": "TEST",
"site": "C",
"skill": "TEST",
"center": "TEST",
"teamLead": "TEST",
"workGroup": "TEST",
"lanId": "TEST",
"dbID": "TETS",
"loginId": "TEST",
"avgAcwTime": "nn",
"avgHandleTime": "nn",
"avgTalkTime": "nn",
"callsAnswered": "nn",
"dispSkill": "-",
"errCode": null,
"errDesc": null,
"avgAcwTimeth": "medium",
"avgHandleTimeth": "high",
"avgTalkTimeTh": "medium",
"callsAnsweredTh": "medium",
"stateTh": "high"
}
Pagenation can't be done due to some requirements.
Can any one suggest something to improve the perfomance
Also I am fetching data using Backbone.Collection.fetch()
getAgentMetric(){
this.metrices.fetch({
url : (isLocal) ? ('http://localhost:8080/jsons/agent.json') : (prev_this.url + '/agentstat'),
data: JSON.stringify(param),
type: "POST",
dataType: "JSON",
contentType: "application/json",
})
.done(function() {
// passing the datasource from ajax call
prev_this.agentLoacalSource.localdata = prev_this.metrices.toJSON();
});
timeout = setTimeout(_.bind(this.getAgentMetric, this), 5000);
},
Browsers can handle a heck of a lot more than a thousand objects without any strain, so I don't think it's the fact that you are simply requesting a large amount of data from the backend. It's more likely that some of your parsing or rendering code is slow.
A few possibilities without seeing any more of your code:
It really depends on what you're doing here, but I'm going to assume that you aren't using a templating library (hoganjs, handlebarsjs, etc). You should definitely look into using one as they speed things up quite a bit and make generating html a lot easier.
Are you running .append() for each individual model that you render? This will really slow things down. You should generate all of the html that needs to be generated, and then run .append() once.
What kind of event listeners are you adding for each model (if any)? Listing to scroll events without a debounce ends up slowing down your browser, especially if you add a bunch of them.
Unrelated to your slowness issues, there are a few problems that I see with this code:
Your timeout should be called from an .always() function in ajax to prevent concurrent requests from going out if for whatever reason a request is slow.
this.metrices.fetch(...)
.always(function() {
timeout = setTimeout(...);
}.bind(this));
Requests that are simply fetching data should use a GET instead of a POST request type. You can see https://stackoverflow.com/a/3477374/5780021 for more info about this.
I would recommend timing some of your code to see where the slowness is actually happening. This will allow you to actually determine how long things are taking between to points in code.
Firefox console.time
Chrome console.time
IE console.time
Is there a way to display a bunch of JSON objects, WITH relations, as a domain model?
Models could be something like this:
{
"name": "Order",
"status": {
"type": "number",
"null": false,
"default": 1
}
},
"relations": {
"customer": {
"type": "belongsTo",
"model": "Customer",
"foreignKey": "customerId"
}
}
}
Basically, as this image shows, Domain model -to-> code... but the other way around.?
Thanks
PS: If there are no tools that do this out of the box, I assume that there might be frameworks that I could use to create this.. any recommendations?
As I understand your question you are asking if a utility exists that can turn your JSON code into a domain model. If I understand correctly the answer is yes. There is one project on github that is doing something similar: json-discoverer
From the project page you will see the tool was inspired by some research which was published in ICWE (International Conference on Web Engineering) 2013 and 2014. Below is a link to the main article, as it is quite lengthy and detailed I will not attempt to summarize it here.
Discovering Implicit Schemas in JSON Data
Unfortunately, as you mention you can't then edit the domain afterward. But I have still found it to be a fairly useful tool.
I am unaware of any other existing utilities. The only other alternatives are fairly easy to find with a simple search, but only allow for conversion to trees and/or tables.
Am working in jQueryMobile and PhoneGap.
Currently am facing a Big issue. When I click a button and at that time its calls multiple APIs one by one and get results one by one. But in result page its shows only after getting all the Results of all API.
In My Application When I Submit the Button and It return a JSON which contains the list of Different APIs to Call
Then We call each API asynchronously.
We get all the result one by one. And Result is showing only after get all the results.
Then My Question is is there any way to shows the result page when I get First API Result and call all other API at that same time when Data shown in Result page in background?
Currently for shows the result Page it takes more than 30 seconds time. But We need to shows the result page when we get first API result. We get result as JSON Format.
Anybody please can help Me :(
i lost my 1 week for resolve this issue !!
There are a number of HTML template solutions are available jQuery Templates, Underscore, and Mustache etc, better to use Mustache i think that solves your problem. its better to use any templates if you are working on so many DOM manipulation in your projects.
Eg:
HTML Template
<h1>{{header}}</h1>
{{#bug}}
{{/bug}}
{{#items}}
{{#first}}
<li><strong>{{name}}</strong></li>
{{/first}}
{{#link}}
<li>{{name}}</li>
{{/link}}
{{/items}}
{{#empty}}
<p>The list is empty.</p>
{{/empty}}
JSON DATA
{
"header": "Colors",
"items": [
{"name": "red", "first": true, "url": "#Red"},
{"name": "green", "link": true, "url": "#Green"},
{"name": "blue", "link": true, "url": "#Blue"}
],
"empty": false
}
Out put HTML
<h1>Colors</h1>
<li><strong>red</strong></li>
<li>green</li>
<li>blue</li>