I know this question has been possed before, but the explanation was a little unclear to me, my question is a little more general. I'm trying to conceptualize how one would periodically update data in an iPhone app, using a remote web service. In theory a portion of the data on the phone would be synced periodically (only when updated). While other data would require the user be online, and be requested on the fly.
Conceptually, this seems possible using XML-RPC or JSON and Core data. I wonder if anyone has an opinion on the best way to implement this, I am a novice iPhone developer, but I understand much of the process conceptually.
Thanks
To synchronize a set of entities when you don't have control over the server, here is one approach:
Add a touched BOOL attribute to your entity description.
In a sync attempt, mark all entity instances as untouched (touched = [NSNumber numberWithBool:NO]).
Loop through your server-side (JSON) instances and add or update entities from your Core Data store to your server-side store, or vice versa. The direction of updating will depend on your synchronization policy, and what data is "fresher" on either side. Either way, mark added, updated or sync'ed Core Data entities as touched (touched = [NSNumber numberWithBool:YES])
Depending on your sync policy, delete all entity instances from your Core Data store which are still untouched. Untouched entities were presumably deleted from your server-side store, because no addition, update or sync event took place between the Core Data store and the server for those objects.
Synchronization is a fair amount of work to implement and will depend on what degree of synchronization you need to support. If you're just pulling data, step 3 is considerably simpler because you won't need to push object updates to the server.
Syncing is hard, very hard. Ideally you would want to receive deltas of the changes from the server and then using a unique id for each record in Core Data, update only those records that are new or changed.
Assuming you can do that, then the code is pretty straight forward. If you are syncing in both directions then things get more complicated because you need to track deltas on both sides and handle collisions.
Can you clarify what type of syncing you are wanting to accomplish? Is it bi-directional or pull only?
I have an answer, but it's sucky. I'm currently looking for a more acceptable/reliable solution (i.e. anything Marcus Zarra cooks up).
What I've done needs some work ... seriously, because it doesn't work all the time...
The mobile device has a json catalog of entities, their versions, and a url pointing to a json file with the entity contents.
The server has the same setup, the catalog listing the entities, etc.
Whenever the mobile device starts, it compares the entity versions of it's local catalog with the catalog on the server. If any of those versions on the server are newer, it offers the user an opportunity to download the entity updates.
When the user elects to update, the mobile device now has the url for each of the new/changed entities and downloads it. Once downloaded, the app will blow away all objects for each of the changed entities, and then insert the new objects from JSON. In the event of an error, the deletions/insertions are rolled back to pre-update status.
This works, sort of. I can't catch it in a debug session when it goes awry, so I'm not sure what might cause corruption or inconsistency in the process.
Related
I'm new to Java and just started writing some JavaFX applications.
My current project is to write an application for a consulting company that store a list of customers, add them to a queue and serve them one by one. There are a few staffs and they will running a copy of the application I write on their PC.
What I've done so far:
create Customer.class to handle personal info and store them in a MySQL db
create Staff.class to handle staff info
create Service.class to handle kind of services are available for the customers
create Consultation.class to handle info of a particular consultation such as date of consultation, customer being served, which staff is providing service, the services offered and the outcome
create an ObservableArrayList, store the data in the MySQL db, and display the data on a TableView of each client PC
What I want to do is, after a staff editing the data in the list, the changes will be updated on the TableView of other client PCs automatically.
The possible solutions I can think of includes:
Option 1
Program the application to query the db regularly for an update.
This method is more simple to implement, but I don't want to keep the MySQL server busy by non-stop querys from a number of clients. I do not want any delay between data write and update on other clients. There are more than 10 clients. If each client update once a second, that will mean at least 10 queries per second and the server will never rest. I don't want to put any stress on the server's harddisk.
Option 2
Program the application to broadcast a message every time after they write data to the db and other clients query the database every time they receive a broadcast. I prefer do it this way but I'm not familiar with network programming. That will mean I'll have to spend some time on it before I can continue the project.
Which of the above is a better choice? Is there other way to keep the TableView on the clients keep synchronized?
Which of the above is a better choice? Is there another way to keep the TableView on the clients keep synchronized?
Before choosing - you may consider optimizing them,
Option 1 seems quite expensive as it has to request frequently. But you can optimize it using connection-pool and specifying certain time-interval(minimum 10 sec) to fetch the data.
Option2 is much more convincing as it applies the lazy-loading concept. You may consider looking socket programming to notify all clients to fetch data.
It's quite hard to say which one is the better option - somehow, I prefer to go with the first approach if your application may insert data frequently, otherwise go with the second one.
An alternative solution - listening to the data changes
Here are some QA, these solutions may help you to implement your requirement.
How to implement a DB listener in Java
How to make a database listener with java?
How to listen to new DB records through java
Is it possible to use couchbase syny gateway in the following way:
1) Mobile client queries couchbase for data.
2) No data in couchbase present so this triggers a import of the needed data from for example a mysql database into couchbase.
3) The imported data is then transfered to the mobile client by couchbase synch gateway.
4) The mobile client goes to sleep.
5) After 12 hours of inactivity the data is removed from couchbase.
6) Next day the mobile client still holds the data offline and syncs again which sync gateway
7) the data is again imported to couchbase server and the diffs are synced with the client
Does couchbase provide hooks to implement such an flexable usecase?
If yes could somebody point me to the important api calls?
Many Greetings
The preferred way to do this would run most things through Sync Gateway (the data imports from the external source in particular should go through Sync Gateway, not directly to Couchbase, and removing the data should go through SG also.)
Sync Gateway's sync function runs when SG receives documents. In this sense, there's no way to trigger something based on nothing being there.
One way you might solve this is by having the mobile client push a special purpose document. Your sync function could catch this and react in several ways (fire a webhook request, start a replication, or you could set up something to monitor a changes feed and trigger from that).
Next you have the issue of removing the data on the Server side. Here the question is a little unclear. Typically applications write new revisions to SG, and these get synced to the client (and vice versa). If you remove everything on the Server side, you'll actually end up with what are called tombstone revisions showing the document as deleted. (This is a result of the flexible conflict resolution technique used by Couchbase Mobile. It uses multiversion concurrency control.)
The question is a little unclear. It sounds like you don't want to store the data long term on the Server side. If that's right, I think you could do something like:
Delete the data (through SG)
Have the mobile client push data to SG
Trigger SG again with some special document
Update the data from the external source
Have the client pull updates from SG
That's a very rough outline. This is too complicated to really work out in this format. I suggest you post questions through the Couchbase developer forum to get more details.
So, the short answer, yes, this seems feasible, but a full answer needs more detail on what you're doing and what your constraints are.
I'm looking to implement notifications within my node.js application. I currently use mysql for relational data (users, submissions, comments, etc). I use mongodb for page views only.
To build a notification system, does it make more sense (from a performance standpoint) to use mongodb vs MySQL?
Also, what's the convention for showing new notifications to users? At first, I was thinking that I'd have a notification icon, and they click on it and it does an ajax call to look for all new notifications from the user, but I want to show the user that the icon is actually worth clicking (either with some different color or a bubble with the number of new notifications like Google Plus does).
I could do it when the user logs it, but that would mean the user would only see new notifications when they logged out and back in (because it'd be saved in their session). Should I poll for updates? I'm not sure if that's the recommended method as it seems like overkill to show a single digit (or more depending on the num of notifications).
If you're using node then you can 'push' notifications to a connected user via websockets. The linked document is an example of one well known websocket engine that has good performance and good documentation. That way your application can send notifications to any user, or sets of users, or everyone based on simple queries that you setup.
Data storage is a different question. Generally mysql does have poor perfomance in cases of high scalability, and mongo does generally have a quicker read query response, but it depends on what data structure you wish to use. If your data is in a simple key-value structure with no real need for relational data, then perhaps using a memory store such as Redis would be the most suitable.
This answer has more information on your question too if you want to follow up and investigate more.
I wanted to know, if in general, when integrating 2 or more systems via whatever means (ie. webservice, MQ, etc.), is it a best practice or a standard for your system to capture a snapshot of data that you are sending with another system? I am thinking that this is as an insurance when reconciling is required for scenarios such as prod incidents.
Secondly, I would think this data snapshot is different from audit trail, in that the data being sent itself is saved (ie. xml data, csv file) as a LOB column in a snapshot table. Is this redundant with the audit trail?
For your first question ...
I've done many, many integrations using queues, web services, etc. and I will usually store an audit trail (a high level set of data telling me what happened), but I've never actually stored the payload itself for each call.
A few reasons for that:
The storage of the payloads being sent back and forth can get quite large.
I can usually reconstruct the payload using the audit trail. "Oh entity XYZ with ID 123 was sent yesterday. Let's take a look at what that entity looks like."
If you do the integration really well and have good testing around it, having copies of the payloads becomes unnecessary.
Instead of storing a copy of the payload I would focus on these things for integration:
Good unit tests on both sides and integration testing for the entire process.
Audit logs as you mentioned.
Good retry policies when a message fails (specifically for queues and topics).
Focusing on idempotent messages. So if something fails, you just do it again and everything is ok.
This question already has answers here:
Client-server synchronization pattern / algorithm?
(7 answers)
Closed 9 years ago.
I am starting to setup the core data model for a large scale app, and was hoping for some feedback on proper synchronization methods/techniques when it comes to server database and offline capabilities.
I use PHP and mySQL for my web server / database.
I already know how to connect, receive data, store to core data, etc etc. I am looking more for help with the methodologies and particular instances of tracking data changes to:
A) Ensure the app and server are in sync during online and offline use (i.e. offline activity will get pushed up once back online).
B) Optimize the speed of saving data to the app.
My main questions are:
What is the best way to check what new/updated data in the app still needs to be synchronized (after offline use)?
(i.e. In all my Core Data Entities I put a 'isSynchronized' attribute of BOOL type. Then update to 'YES' once successfully submitted and response is sent back from server). Is this the best way?
What is the best way to optimize speed of saving data from server to core data?
(i.e. How can I only update data in Core Data that is older than what is on server database without iterating through each entity and just updating every single time)? Is it possible without adding a server database column for tracking update timestamps to EVERY table?
Again, I already know how to download data and store it to Core Data, I am just looking for some help with best practices in ensuring synchronization across app and server databases while ensuring optimized processing time.
I store a last modified timestamp in the database on both the core data records on the phone, and the mysql tables on the server.
The phone searches for everything that has changed since the last sync and sends it up to the server along with a timestamp of the last sync, and the server responds with everything that has changed on it's end since the provided sync timestamp.
Performance is an issue when a lot of records have changed. I do the sync on a background NSOpeartion which has it's own managed object context. When the background thread has finished making changes to it's managed object context, there is an API for merging all of the changes into the main thread's managed object context - which can be configured to simply throw away all the changes if there are any conflicts caused by the user changing data while the sync is going on. In that case, I just wait a few seconds and then try doing a sync again.
On older hardware even after many optimisations it was necessary to abort the sync entirely if the user starts doing stuff in the app. It was simply using too many system resources. I think more modern iOS devices are probably fast enough you don't need to do that anymore.
(by the way, when I said "a lot of records have changed" I meant 30,000 or so rows being updated or inserted on the phone)