Read and write JSON file for mobile app - html

I am working on a mobile app. Since I wanted a solution that would work on multiple platforms, I started with Cordova. As I have much data to handle and many views, I decided to use Ember.
I need to use three different JSON datasets that are updated rather frequently in the database. To enable the mobile app to work offline, I will have to store the JSON data, as well as update them when the database is changed.
Questions
How can I retrieve JSON from another server(CORS blocked)?
How can the retrieved JSON be saved on device? [HTML5 LocalStorage(preferred) or FileAPI]
How to read JSON file and send data as model to templates? (I guess Ember.$.getJSON can do this)
Any help is welcome. Thanks!
Update 1
Since I ran into many issues using Ember-data, I am not using it in my app.
Found this link for cross-domain with ajax
Update 2
CORS can be solved only by JSONP or by setting ACCESS-CONTROL-ALLOW-ORIGIN in the reponse of the server(I tried many other solutions, and failed)
Since the api is in my server, I did the latter.
For 2 and 3, I think I could use this SO question

This is what I found out :
JSON data from a server in different domain
You cannot read JSON data from a server in another domain. This is due to the Same-origin policy implemented in browsers. A browser will retrieve your JSON but will not allow you access to the same. There are two solutions(AFAIK) to this problem :
Using JSONP - I'm not going into the details, but there are many links available for this.
Allow CORS from server - When the server sends JSONified data, you can add additional headers for ACCESS-CONTROL-ALLOW-ORIGIN. After retrieving the JSON from server, the browser checks for this header to either block or allow CORS. I used some decorators for adding crossdomain headers and then my data was successfully read in the browser.
Saving the json data
HTML5 makes everything easier. In your javascript, you just have to use :
localStorage["application.state.data"] = JSON.stringify(json);
or
localStorage.setItem("application.state.data", JSON.stringify(json));
Retrieve works just the same
var data = JSON.parse(localStorage["application.state.data"]);
or
var data = JSON.parse(localStorage.getItem("application.state.data"));

Related

Why am I getting XML results from the Exact Online API where I used to get JSON results?

Just started working on an existing project making use of the Exact Online API.
While I was debugging the project I suddenly only started receiving XML results instead of JSON results from the API. I did not change anything about the endpoints being queried I was just running the existing queries trying to figure some things out.
These are the REST API docs: https://start.exactonline.nl/docs/HlpRestAPIResources.aspx
These are the XML docs: https://support.exactonline.com/community/s/knowledge-base#All-All-DNO-Content-xmlsamplecode
Typical REST API endpoints look like this:
https://start.exactonline.be/api/v1/xxxxxx/salesinvoice/SalesInvoices
Typical XML endpoints look like this:
https://start.exactonline.be/docs/XMLDownload.aspx
I also did not change any settings. I only have access to the tokens and api. I don't have access to the account.
This is an example of an endpoint and query where I previously received JSON but am now receiving XML:
https://start.exactonline.be/api/v1/xxxxxx/salesinvoice/SalesInvoices?$filter=InvoiceID eq guid'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx'&$select=InvoiceID
I tried this manually with Postman and also using the existing code from the project.
Is there some setting I am unaware of? Am i querying the wrong way? Maybe there have been some changes to the API I am unaware of that aren't listed in the release notes?
Please provide the request header Accept in your HTTP request that specifies what content format you prefer to receive: application/json. The default of Exact Online APIs is XML (but seldom to never used).

Caching API response from simple ruby script

I'm working on a simple ruby script with cli that will allow me to browse certain statistics inside the terminal.
I'm using API from the following website: https://worldcup.sfg.io/matches
require 'httparty'
url = "https://worldcup.sfg.io/matches"
response = HTTParty.get(url)
I have to goals in mind. First is to somehow save the JSON response (I'm not using a database) so I can avoid unnecessary requests. Second is to check if the new data is available, and if it is, to override the previously saved response.
What's the best way to go about this?
... with cli ...
So caching in memory is likely not available to you. In this case you can save the response to a file on disk.
Second is to check if the new data is available, and if it is, to override the previously saved response.
The thing is, how can you check if new data is available without doing a request for the data? Not possible (given the information you provided). So you can simply keep fetching data every 5 minutes or so and updating your local file.

Handling large data through restapi

I have a RestFul server that is suppuse to return a large json object more specifically an array of objects to browsers. For example 30,000 points will have a size of 6.5mb.
But I get this content mismatch error in browser when speed is slow. I feel it is because large data throught rest api breaks up something. Even in Postman sometimes it fails to render even though i see data of 6.5 mb received.
My Server is in NodeJS. and return content-type header is application/json.
My Question is
Would it make more sense if I return a .json file. Will the browser be able to handle. If yes, then I will download the file and make front end changes.
Old URL - http://my-rest-server/data
Proposed Url - http://my-rest-server/data.json
What would be content-type in the proposed url?
Your client can't possibly expect to want all of the data at once but still, want their data fast data.
...but you might want to look into sending data in chunks and streams:
https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93

How do I set up my Web Server, to return a JSON String query?

In its most basic form, what I want to do is access data on my web server, from an Android App I'm developing (to be clear, different apps accesses the same data set).
To be more specific, I would like to set up my Web Server so that it returns a JSON String, in response to an HTTP GET request sent from the app. (That's not a requirement by any means, it just seems like how most APIs work).
So far, it seems I need a Web Framework with an MVC, like Ruby on Rails that uses a controller that can decide what to do with an HTTP GET request.
What I don't quite understand yet (not that I understand what I've said so far), is this:
How the data is stored (I know I can use a database, but I'd like to know the best/other options)
How to get query parameters from the URI
How to retrieve the data
How to organize the data into a JSON object/string
How to send the JSON string in response to the GET request
Obviously, I'm not a web development expert. I'm trying to learn, but I don't have the experience yet to even know what to search for. So I appreciate any help and resources you may have.

Preventing access to JSON data in an Angular app

I got a (Flask) backend powering an API that serves JSON to an Angular app.
I love the fact that my backend (algorithms, database) is totally disconnected from my frontend (design, UI) as it could literally run from two distinct servers. However since the view is entirely generated client side everyone can access the JSON data obviously. Say the application is a simple list of things (the things are stored in a JSON file).
In order to prevent direct access to my database through JSON in the browser console I found these options :
Encrypting the data (weak since the decrypting function will be freely visible in the javascript, but not so easy when dealing with minified files)
Instead of $http.get the whole database then filtering with angular, $http.get many times (as the user is scrolling a list for example) so that it is programmatically harder to crawl
I believe my options are still weak. How could I make it harder for a hacker to crawl the whole database ? Any ideas ?
As I understand this question - the user should be permitted to access all of the data via your UI, but you do not want them to access the API directly. As you have figured out, any data accessed by the client cannot be secured but we can make accessing it a little more of PITA.
One common way of doing this is to check the HTTP referer. When you make a call from the UI the server will be given the page the request is coming from. This is typically used to prevent people creating mashups that use your data without permission. As with all the HTTP request headers, you are relying on the caller to be truthful. This will not protect you from console hacking or someone writing a scraper in some other language. #see CSRF
Another idea is to embed a variable token in the html source that bootstraps your app. You can specify this as an angular constant or a global variable and include it in all of your $http requests. The token itself could be unique for each session or be a encrypted expiration date that only the server can process. However, this method is flawed as well as someone could parse the html source, get the code, and then make a request.
So really, you can make it harder for someone, but it is hardly foolproof.
If users should only be able to access some of the data, you can try something like firebase. It allows you to define rules for who can access what.
Security Considerations When designing web applications, consider
security threats from:
JSON vulnerability XSRF Both server and the client must cooperate in
order to eliminate these threats. Angular comes pre-configured with
strategies that address these issues, but for this to work backend
server cooperation is required.
JSON Vulnerability Protection A JSON vulnerability allows third party
website to turn your JSON resource URL into JSONP request under some
conditions. To counter this your server can prefix all JSON requests
with following string ")]}',\n". Angular will automatically strip the
prefix before processing it as JSON.
For example if your server needs to return:
['one','two'] which is vulnerable to attack, your server can return:
)]}', ['one','two'] Angular will strip the prefix, before processing
the JSON.
Cross Site Request Forgery (XSRF) Protection XSRF is a technique by
which an unauthorized site can gain your user's private data. Angular
provides a mechanism to counter XSRF. When performing XHR requests,
the $http service reads a token from a cookie (by default, XSRF-TOKEN)
and sets it as an HTTP header (X-XSRF-TOKEN). Since only JavaScript
that runs on your domain could read the cookie, your server can be
assured that the XHR came from JavaScript running on your domain. The
header will not be set for cross-domain requests.
To take advantage of this, your server needs to set a token in a
JavaScript readable session cookie called XSRF-TOKEN on the first HTTP
GET request. On subsequent XHR requests the server can verify that the
cookie matches X-XSRF-TOKEN HTTP header, and therefore be sure that
only JavaScript running on your domain could have sent the request.
The token must be unique for each user and must be verifiable by the
server (to prevent the JavaScript from making up its own tokens). We
recommend that the token is a digest of your site's authentication
cookie with a salt for added security.
The name of the headers can be specified using the xsrfHeaderName and
xsrfCookieName properties of either $httpProvider.defaults at
config-time, $http.defaults at run-time, or the per-request config
object.
Please Kindly refer the below link,
https://docs.angularjs.org/api/ng/service/$http
From AngularJS DOCs
JSON Vulnerability Protection
A JSON vulnerability allows third party website to turn your JSON resource URL into JSONP request under some conditions. To counter this your server can prefix all JSON requests with following string ")]}',\n". Angular will automatically strip the prefix before processing it as JSON.
There are other techniques like XSRF protection and Transformations which will further add security to your JSON communications. more on this can be found in AngularJS Docs https://docs.angularjs.org/api/ng/service/$http
You might want to consider using JSON Web Tokens for this. I'm not sure how to implement this in Flask but here is a decent example of how it can be done with a Nodejs backend. This example at least shows how you can implement it in Angularjs.
http://www.kdelemme.com/2014/03/09/authentication-with-angularjs-and-a-node-js-rest-api/
Update: JWT for Flask:
https://github.com/mattupstate/flask-jwt