I'm implementing a web service that needs to query a JSON file( size: ~100MB; format: [{},{},...,{}] ) about 70-80 times per second, and the JSON file will be updated every hour. "query a JSON file" means checking if there's a JSON object in the file that has an attribute with a certain value.
Currently I think I will implement the service in Node.js, and import ( mongoimport ) the JSON file into a collection in MongoDB. When a request comes in, it will query the MongoDB collection instead of reading and looking up in the file directly. In the Node.js server, there should be another timer service, which in every hour checks whether the JSON file has been updated, and if it has, it needs to "repopulate" the collection with the data in the new file.
The JSON file is retrieved by sending a request to an external API. The API has two methods: methodA lets me download the entire JSON file; methodB is actually just an HTTP HEAD call, which simply tells whether the file has been updated. I cannot get the incrementally updated data from the API.
My problem is with the hourly update. With the service running, requests are coming in constantly. When the timer detects there is an update for the JSON file, it will download it and when download finishes it will try to re-import the file to the collection, which I think will take at least a few minutes. Is there a way to do this without interrupting the queries to the collection?
Above is my first idea to approach this. Is there anything wrong about the process? Looking up in the file directly just seems too expensive, especially with the requests coming in about 100 times per seconds.
Related
I send JSON's to my app via Postman in a list with a type of mapping(CRUD) to send it to my database.
I want my controller to put all this data, from multiple senders, in a list that will send the information to my DB. The problem is that i don't know how to store in the same list the Json and the Mapping, so when my threads do their work to know if that json must be inserted, updated, deleted and so on.
Do you guys have any ideea ?
PS: It is a spring-boot app that need to be able to send 12000 objects ( made from that jsons ) to the db.
I don't see a reason for putting all data in one list and sharing it later, each HTTP request receives own thread.
On decent server you can handle couple thousands of requests/sec which perform simple CRUD operations.
I'm working on a simple ruby script with cli that will allow me to browse certain statistics inside the terminal.
I'm using API from the following website: https://worldcup.sfg.io/matches
require 'httparty'
url = "https://worldcup.sfg.io/matches"
response = HTTParty.get(url)
I have to goals in mind. First is to somehow save the JSON response (I'm not using a database) so I can avoid unnecessary requests. Second is to check if the new data is available, and if it is, to override the previously saved response.
What's the best way to go about this?
... with cli ...
So caching in memory is likely not available to you. In this case you can save the response to a file on disk.
Second is to check if the new data is available, and if it is, to override the previously saved response.
The thing is, how can you check if new data is available without doing a request for the data? Not possible (given the information you provided). So you can simply keep fetching data every 5 minutes or so and updating your local file.
I have a js frontend, to a backend WAMP router, that has a nodejs app that publishes JSON encoded geo-coded data in real time to a WAMP topic. The frontend subscribes to that WAMP topic and visualizes the moving geo-coded targets on a map. That data is also stored on the backend within MongoDB.
I need to add a function whereby the user can request, play, stop, rewind, and fast-forward through archived data. I have a backend nodejs function that receives the begin/end times for the query from the frontend, makes the MongoDB query and uses the MongoDB db.find.stream() interface to publish the data to a WAMP topic in realtime. The stream.on("data", ...) method pauses/resumes the stream to playback the data in time-step. This works fine for playback and stop. But, I don't know how to handle rewind and fast-forward. Based on what I've seen, you can't rewind or iterate backwards on a stream.
My brute force approach would be to load the entire huge query result into an in memory array and have methods the frontend calls to control the increment/decrement of the array pointer depending on if they are playing or rewinding so that they can see the targets moving on the map and find what they are looking for.
Is there an API or library that would allow me to accomplish rewind without loading the array into memory? Possibly storing the result into a file and moving back and forth through it?
I have an application that downloads data via NSURLConnection in the form of a JSON object; it then displays the data to the user. As new data may be created on the server at any point, what is the best way to 'realise' this and download this data?
At the moment I am planning on having the application download all the data every 30-40 seconds, and then check the data downloaded against the current data: if it is the same do nothing; if it is different, procede with the alterations. However, this seems a bit unnecessary, especially as the data may not change for a while. Is there a more efficient way of updating the application data when new server data is created?
Use ETag if the server supports it.
Wikipedia ETag
"If the resource content at that URL ever changes, a new and different ETag is assigned."
You could send a HTTP HEAD request to the server with the "If-Modified-Since" header set to the time you recieved the last version. If the server handles this correctly, it should return 304 (Not Modified) while the file is unchanged; so as soon as it doesn't return that, you GET the file and procede as usual.
See HTTP/1.1: Header Field Definitions
I am developing an iOS app that uses a single context architecture. I make frequent calls to my API (PHP) and I want to "cache" the output for as long as the session is active. Right now I am saving the output to a variable that is defined in the app.s.
var contacts = {
contactsData: null
};
So I do this to save the output, is it really a good idea? Will it slow things down?
contacts.contactsData = output;
Thankful for all input!
It consist of how big is json file in mb. If device have enough RAM - it is the best way. Also be sure you save decoded json not just request response, so you will not decode it every time.
If json data is too big you must think about some kind of local storage. If Json is always the same (no need to synch every time) save it local.
If you need update it often you can upload extremly needed part with 1 limited request (API config needed) and other data with second background request.