How can I set Observables to listen to different events from a bucket? So that whenever a new document added to the bucket or existing document updated I can read them and push it to the destination.
Note: Just getting started with couchbase.
Right now the SDK doesn't offer an API to listen to changes in the database. This is something that might be built as a facade on top of DCP, the protocol for cross data center replication, at a later point in time
To add more to what Simon said, you might be able to write your own DCP connector. DCP does provide the stream of changes you are looking for, but you have to have a connector that will ingest that stream. If this is something you might be up for, i'd look at the code for existing connectors like the Kafka connector and see if you could use them to create your own.
As Kirk said, you can definitely use DCP for this, as long as you're ok with writing it in Java, because that's the only SDK that currently has (experimental) support for streaming document changes.
As it happens, David Maier wrote a step-by-step tutorial on how to implement a change notification listener for Couchbase: http://nosqlgeek.blogspot.co.il/2015/05/dcp-magic.html
Related
I have been looking everywhere for a solution to this problem.
At my work, we are trying to integrate Maximo with another system via the other systems REST API (which returns JSON responses). I am able to make this integration work on a small scale, however this API is taking upwards of 5 seconds to respond per request. Currently, I have defined this system as a JSON Resource, and I copy daily "snapshots" of the non-persistent data to a persistent attribute using an automation script. The requests all run in a sequence - which works slowly for 5 assets in testing, and will definitely not scale to 1000's of calls a day.
Assume that the API of the external system cannot be modified in any way... Is there a way to query this API in a non-blocking way? I'd imagine that if I could send a request, and send the next, etc. without needing to wait for a reply to proceed, this would solve the problem.
I looked into Invocation and Publishing Channels, and also Enterprise Services, and it seems like Enterprise Services along with JMS Queues might be what I need, however documentation says that these only support queuing incoming data... and I can't see how this solves my problem.
Any help? I am completely stuck on this.
Thank you!
I had to do something that sounds similar, once. I tried JSON Resources, but they didn't work for me. I ended up using the examples in Maximo 7.6 Scripting Features to do it. The first code sample in that document is a library script for making HTTP/S calls using out-of-the-Maximo-box libraries, and other examples in that document use IBM's JSONObject and JSONArray classes (also available out of the Maximo box) to parse responses.
To get things going concurrently / multithreaded, you could configure a cron task to call your automation script, and configure multiple instances on various schedules to call the same one and use the args or some other mechanism to prevent collisions.
Is there any way to track an external site with socket.io? So whenever the website changes then it would automatically send an event to my server?
For example:
socket.on("newconnectionfromxxxxxxxxxxxxxxx", function (websiteChange) { //websiteChange would be the contents of the updated website.
});
Sorry if this questions is obvious because I couldn't find anything online.
EDIT: The site that I want to be tracking is a JSON file.
socket.io works between two cooperating sites. So, it would only help you here if the site you want to monitor specifically supports an incoming socket.io connection AND it supports sending a notification message over that connection whenever the specific file you are interested in changes.
If the site you want to monitor does not have such specific monitoring features, then the best you could do is to regularly download the file of interest and see for yourself if it has changed.
Since all websites are not really fully event-driven/reactive at this time, you'll have to poll your website target periodically and look at that JSON file for differences. There are many "filewatcher" programs (C#, Java, etc) out there that hook into the native OS filesystem event mechanisms; you could alter one of these programs to trigger an WS or HTTP event to your server. There is a NodeJS wrapper that might be helpful for you: https://github.com/paulmillr/chokidar
Bummer the Unix "watch" utility didn't have an option to trigger events when a diff was detected instead of visually highlighting the change. It would be really cool if it has a "-difftrigger 'some-command'" option...
I want to use Couchbase but I want to implement change tracking in a few areas similar to the way RethinkDB does it.
There appears to be a hand full of ways to have changes pushed to me from a Couchbase server.
DCP
TAP
XDCR
Which one is the correct choice, or is there a better method?
UPDATE
Thanks #Kirk!
Thanks! it looks like DCP does not have a 100% production ready API today (5/19/2015). Your blog ref helped me decide to use XDCR today and migrate to DCP as soon as an official API is ready.
For XDCR this GitHub Repo has been helpful.
Right now the only fully supported way is XDCR as Kirk mentioned already. If you want to save time implementing it, you might want to base your code on this: https://github.com/couchbaselabs/couchbase-capi-server - it implements server side of the XDCR protocol (v1). The ElasticSearch plugin is based on this CAPI server, for example. XDCR is a good choice if your application is a server/service that can wait for incoming connections, so Couchbase (or the administrator) controls how and when Couchbase replicates data to your service.
Depending on what you want to accomplish, DCP might end up being a better choice later, because it's conceptually different from XDCR. Any DCP-based solution would be pull-based (from your code's side), so you have more fine-grained, programmatical, control over how and when to connect to a Couchbase bucket, and how to distribute your connections across different processes if necessary. For a more in-depth example of using DCP, take a look at the Couchbase-Kafka connector here: https://github.com/couchbase/couchbase-kafka-connector
DCP is the proper choice for this if how it works fits your use case and you can write an application to consume the stream as there is no official API...yet. Here is a blog post about doing this in java by one of the Couchbase Solutions Engineers, http://nosqlgeek.blogspot.de/2015/05/dcp-magic.html
TAP is basically deprecated at this point. It is still in the product, but DCP is far superior to it in most every fashion.
XDCR could be used, as it uses DCP, but you'd have to write a plug-in for XDCR. So you'd just be better off to write one directly to consume the DCP stream.
I'm trying to build my first Sproutcore App and I struggle to connect it to a MySQL-Database or any datasource other than fixture. I can't seem to find ANY tutorial except this one from 2009 which is marked as deprecated: http://wiki.sproutcore.com/w/page/12413058/Todos%2007-Hooking%20Up%20to%20the%20Backend .
Do people usually not connect SC-Apps to a Database? If they do so, how do they find out how to? Or does the above mentioned tutorial still work? A lot of gem-commands in the introduction seems to already differ from the official Sproutcore getting-started-guide.
SproutCore apps, as client-side "in-browser" apps, cannot connect directly to a MySQL or any other non-browser database. The application itself runs only within the user's browser (it's just HTML, CSS & JavaScript once built and deployed) and typically accesses any external data via XHR requests to an API or APIs. Therefore, you will need to create a service wrapper around your MySQL database in order for your client-side app to be able to load and update data.
There are two things worth mentioning. The first is that since the SproutCore app contains all of your user interface and a great deal of business logic, your API can be quite simple and should only return raw data (such as JSON). The second is that, I should mention that the client-server design, while more tedious to implement, is absolutely necessary in practice, because you can never trust the client side code, which is in the hands of a possibly nefarious user. Therefore, your API should also act as the final gatekeeper to validate all requests from the client.
This tutorial I found helped me a lot. Its very brief and demonstrates how to implement a very simple login-app, how to send post-requests (triggered by the login-button-action) to the backend-server and how to asynchronously process the response inside the Sproutcore-App:
http://hawkins.io/2011/04/sproutcore_login_tutorial/
I'm planning to build an application that runs on WEB(AngularJS), Android(Native), IOS(Native).
I have experience with MongoDB, but I found CouchBase which sounds really good for me.
I read documentation and I found out I need to use sync_gatway to sync my mobile databases with main database server and reverse, until now everything is fine.
I also need to use "channels" to share records with multiple users.
The problem comes when I need to implement this for web application.
In their documentation on "Working with web applications" they explain how "bucket shadowing" is working, but they also say:
Bucket shadowing is meant to enable sync for existing Couchbase Server
apps. If you are creating a new app with both mobile and web clients,
we recommend starting with the Sync Gateway REST APIs, and connecting
backend services using the Changes Worker Pattern.
After reading Sync Gateway REST API I found out I'm limited to facebook and persona authentication. So I can't use my own authentication mechanism?
Also, there is nothing specified in REST API about channels?
Is there any example project or more documentation about this? I couldn't find anything :(
If someone has experience with this, please explain how this works.
Thanks
There is also Custom (Indirect) Authentication available on Sync Gateway, which you can use for any type of auth you need.
But you have to hide Sync Gateway's Admin API under your backend layer.
As for the channels: it is responsibility of Sync Function to route different documents to necessary channels based on Document data.
Here is a good video that describes how to build production architecture around Couchbase Lite.
I'm probably late for the party - but as of today I'd recommend taking a look in the PouchDB project for the WEB AngularJS side - they match pretty well and will sync with Couchbase.
Regarding authentication, I just released an article on that topic, find it here. Hope this helps somebody