I want to use Couchbase but I want to implement change tracking in a few areas similar to the way RethinkDB does it.
There appears to be a hand full of ways to have changes pushed to me from a Couchbase server.
DCP
TAP
XDCR
Which one is the correct choice, or is there a better method?
UPDATE
Thanks #Kirk!
Thanks! it looks like DCP does not have a 100% production ready API today (5/19/2015). Your blog ref helped me decide to use XDCR today and migrate to DCP as soon as an official API is ready.
For XDCR this GitHub Repo has been helpful.
Right now the only fully supported way is XDCR as Kirk mentioned already. If you want to save time implementing it, you might want to base your code on this: https://github.com/couchbaselabs/couchbase-capi-server - it implements server side of the XDCR protocol (v1). The ElasticSearch plugin is based on this CAPI server, for example. XDCR is a good choice if your application is a server/service that can wait for incoming connections, so Couchbase (or the administrator) controls how and when Couchbase replicates data to your service.
Depending on what you want to accomplish, DCP might end up being a better choice later, because it's conceptually different from XDCR. Any DCP-based solution would be pull-based (from your code's side), so you have more fine-grained, programmatical, control over how and when to connect to a Couchbase bucket, and how to distribute your connections across different processes if necessary. For a more in-depth example of using DCP, take a look at the Couchbase-Kafka connector here: https://github.com/couchbase/couchbase-kafka-connector
DCP is the proper choice for this if how it works fits your use case and you can write an application to consume the stream as there is no official API...yet. Here is a blog post about doing this in java by one of the Couchbase Solutions Engineers, http://nosqlgeek.blogspot.de/2015/05/dcp-magic.html
TAP is basically deprecated at this point. It is still in the product, but DCP is far superior to it in most every fashion.
XDCR could be used, as it uses DCP, but you'd have to write a plug-in for XDCR. So you'd just be better off to write one directly to consume the DCP stream.
Related
How can I set Observables to listen to different events from a bucket? So that whenever a new document added to the bucket or existing document updated I can read them and push it to the destination.
Note: Just getting started with couchbase.
Right now the SDK doesn't offer an API to listen to changes in the database. This is something that might be built as a facade on top of DCP, the protocol for cross data center replication, at a later point in time
To add more to what Simon said, you might be able to write your own DCP connector. DCP does provide the stream of changes you are looking for, but you have to have a connector that will ingest that stream. If this is something you might be up for, i'd look at the code for existing connectors like the Kafka connector and see if you could use them to create your own.
As Kirk said, you can definitely use DCP for this, as long as you're ok with writing it in Java, because that's the only SDK that currently has (experimental) support for streaming document changes.
As it happens, David Maier wrote a step-by-step tutorial on how to implement a change notification listener for Couchbase: http://nosqlgeek.blogspot.co.il/2015/05/dcp-magic.html
I´ve been checking the Rest API and I cannot find an API to delete a particular document, or all documents of one particular view. Just the delete of the bucket or view.
Anyone know if is that possible?
generally speaking the (official, supported) REST API will deal with the cluster and buckets, whereas key/value operations are handled through the memcached binary protocol, and the SDKs are here to give you a facade over all this (including k/v) in your language of choice, idiomatically.
tl:dr no at least not in a supported stable way, you should use one of the SDKs
I can't find it either. Of course you could easily roll your own front-end wrapping a Couchbase client. Here's an example in ruby this.
I'm planning to build an application that runs on WEB(AngularJS), Android(Native), IOS(Native).
I have experience with MongoDB, but I found CouchBase which sounds really good for me.
I read documentation and I found out I need to use sync_gatway to sync my mobile databases with main database server and reverse, until now everything is fine.
I also need to use "channels" to share records with multiple users.
The problem comes when I need to implement this for web application.
In their documentation on "Working with web applications" they explain how "bucket shadowing" is working, but they also say:
Bucket shadowing is meant to enable sync for existing Couchbase Server
apps. If you are creating a new app with both mobile and web clients,
we recommend starting with the Sync Gateway REST APIs, and connecting
backend services using the Changes Worker Pattern.
After reading Sync Gateway REST API I found out I'm limited to facebook and persona authentication. So I can't use my own authentication mechanism?
Also, there is nothing specified in REST API about channels?
Is there any example project or more documentation about this? I couldn't find anything :(
If someone has experience with this, please explain how this works.
Thanks
There is also Custom (Indirect) Authentication available on Sync Gateway, which you can use for any type of auth you need.
But you have to hide Sync Gateway's Admin API under your backend layer.
As for the channels: it is responsibility of Sync Function to route different documents to necessary channels based on Document data.
Here is a good video that describes how to build production architecture around Couchbase Lite.
I'm probably late for the party - but as of today I'd recommend taking a look in the PouchDB project for the WEB AngularJS side - they match pretty well and will sync with Couchbase.
Regarding authentication, I just released an article on that topic, find it here. Hope this helps somebody
As our systems grow, there are more and more servers and services (different types and multiple instances of the same type that require minor config changes). We are looking for a "cetralized configuration" solution, preferably existing and nothing we need to develop from scrtach.
The idea is something like, service goes up, it knows a single piece of data (its type+location+version+serviceID or something like that) and contacts some central service that will give it its proper config (file, object or whatever).
If the service that goes online can't find the config service it will either use a cached config or refuse to initialize (behavior should probably be specified in the startup parameters it's getting from whom or whatever is bringing it online)
The config service should be highly avaiable i.e. a cluster of servers (ZooKeeper keeps sounding like a perfect candidate)
The service should preferably support the concept of inheritence, allowing a global configuration file for the type of service and then specific overrides or extensions for each instance of the service by its ID. Also, it should support something like config versioning, allowing to keep different configurations of the same service type for different versions since we want to rely more on more on side by side rollout of services.
The other side of the equation is that there is a config admin tool that connects to the same centralized config service, and can review and update all the configurations based on the requirements above.
I know that if I modify the core requirement from serivce pulling config data to having the data pushed to it I can use something like puppet or chef to manage everything. I have to be honest, I have little experience with these two systems (our IT team has more), but from my investigations I can say it seemed they are NOT the right tools for this job.
Are there any systems similar to the one I describe above that anyone has integrated with?
I've only had experience with home grown solutions so my answer may not solve your issue but may help someone else. We've utilized web servers and SVN robots quite successfully for configuration management. This solution would not mean that you would have to "develop from scratch" but is not a turn-key solution either.
We had multiple web-servers each refreshing its configurations from a SVN repository at a synchronized minute basis. The clients would make requests of the servers with the /type=...&location=...&version=... type of HTTP arguments. Those values could then be used in the views when necessary to customize the configurations. We did this both with Spring XML files that were being reloaded live and standard field=value property files.
Our system was pull only although we could trigger a pull via JMX If necessary.
Hope this helps somewhat.
Config4* (of which I am the maintainer) can provide you with most of the capabilities you are looking for out-of-the-box, and I suspect you could easily build the remaining capabilities on top of it.
Read Chapters 2 and 3 of the "Getting Started" manual to get a feel for Config4*'s capabilities (don't worry, they are very short chapters). Doing that should help you decide how well Config4* meets your needs.
You can find links to PDF and HTML versions of the manuals near the end of the main page of the Config4* website.
I'm trying to use CouchDB with HTML/standalone REST architecture. That is, no other app server other than CouchDB and ajax style javascript calling CouchDB.
It looks like cross scripting is a problem. I was using Cloudkit/Tokyo Cabinet before and it seems like the needed callback function was screwing it up in the URL.
Now I'm trying CouchDB and getting the same problem.
Here are my questions:
1) Are these problems because the REST/JSON store like CouchDB or CloudKit is running on a different port from my web page? They're both run locally and called from "localhost".
2) Should I let CouchDB host my page and serve the HTML?
3) How do I do this? The documentation didnt seem so clear...
Thanks,
Alex
There is a simple answer: store static HTML as attachments to CouchDB documents. That way you can serve the HTML directly from the CouchDB.
There is a command-line tool to help you do this, called CouchApp
The book Mikeal linked to also has a chapter (Managing Design Documents) on how to use CouchApp to do this.
3) you can use CouchDB shows to generate HTML (or any content type)
There are huge advantages to having CouchDB serve/generate your HTML.
For one thing, the pages (which are HTTP resources) are tied to the data or to the queries on the data and CouchDB knows when to update the etag when the page has changed. This means that if you stick nginx in front of CouchDB and say "cache stuff" you get all the free caching you would normally need to build yourself.
I would push for nginx > apache in front of CouchDB because Apache isn't all that great at handling concurrent connections and nginx + erlang (CouchDB) are great at it.
Also, you can write these views in JavaScript which are documented well in the CouchDB book http://books.couchdb.org/relax/ or in Python using my view server http://github.com/mikeal/couchdb-pythonviews which isn't really documented at all yet but I'll be getting to it soon :)
I hope that view servers in other languages start implementing the new features in the view server protocol as well so that anyone can write standalone apps in CouchDB.
I think one way is thorugh mod_proxy in Apache. It forward the request from Apache to Couchdb so may solve the cross scripting issue.
# Configuration file for proxy
ProxyVia ON
ProxyPass /couchdb http://<<couchdb host>>:5984/sampleDB
ProxyPassReverse /couchdb http://<<couchdb host>>:5984/sampleDB
I can't help thinking you need some layer between the presentation layer (HTML) and the model (CouchDB).
That way you can mediate requests, provide additional facilities and functionality. At the moment you seem to be rendering persisted objects direct to the presentation layer, and you'll have no facility to change or extend the behaviour of your system going forward.
Adopting a model-view-controller architecture will insulate your model from the presentation layer and give you some flexibility going forwards.
(I confess I can't advise on your cross-site-scripting issues)