Environment: C# Winforms, SDK 1.2
I am new to Couchbase so I may have missed something. What I want to achieve is a push notification to a client(s) when a document updates after being replicated by XDCR. My use case is as follows:
Two clusters (CL1, CL2) with a single node in each
Two clients (K1, K2) such that K1 connected to CL1, K2 connected to CL2
K1 makes a change to DocA
DocA is replicated via XDCR from CL1 --> CL2
Once DocA is in CL2 client K2 is notified
Is this something that the Observe() method can do or is there another approach? Using TAP seems to be risky as that is an internal API that is subject to change.
--D
Answer posted to the Couchbase forums as well, but copying here... http://www.couchbase.com/forums/thread/notify-client-change#comment-1008518
The client is not aware of XDCR replication and it does not have any hooks into XDCR events. So unfortunately, there's no way for a client instance to know that a key was replicated to its cluster.
Assuming that K2 needs to operate on data that's been replicated to C2, an (admittedly untested) alternative would be to have C2 replicate to an ASP.NET endpoint. In this case, you could use the Nancy endpoint I recently wrote - http://blog.couchbase.com/xdcr-aspnet-and-nancy.
K1 writes a key to C1
C1 (via XDCR) sends the key to C2
C2 (via XDCR) sends the key to the Nancy endpoint
K2 is used inside of an IReplicationHandler subclass (see blog for details)
So basically, after C1 XDCR replicates to C2, you push from C2 to your app, which would have XDCR endpoints defined, either via Nancy or by copying the code from my Nancy app into your existing web app. The endpoint host isn't important, but the URIs (e.g., /pools) and JSON responses are. Again, the idea here is that you're treating the K2 app as a third XDCR endpoint, but one that receives its keys only after C2 gets them from C1.
Observe is used to specify durability requirements when storing or removing keys. For example, you could ask the client to consider a store operation successful if and only if a key was persisted to its master node and replicated (in memory or disk) to two replica nodes. http://www.couchbase.com/docs/couchbase-sdk-net-1.2/couchbase-sdk-net-st...
Related
I'm trying add Custom Authentication. I use 2 instances of Sync-Gateway behind an Nginx.
So how can we manage Session with 2 instances of Sync-Gateway?
The custom authentication is achieved by calling the "//_session" endpoint. This then returns a cookie that logs me in for the first Sync-Gateway. If I now want to synchronize the second Sync-Gateway too, I make a call to "//_session" which overrides the first cookie.
Is there any way to create a global Session?
Please see this answer, from bbrks, here: https://forums.couchbase.com/t/custom-authentication-with-2-sync-gateway/29762 :
If you’re using 2 separate CB buckets/SG databases, they’re completely
independent systems and user-information is not shared between them.
I'm finalizing a Data Studio connector and noticing some odd behavior with the number of API calls.
Where I'm expecting to see a single API call, I'm seeing multiple calls.
In my apps script I'm keeping a simple tally which increments by 1 every url fetch and that is giving me the correct number I expect to see with getData().
However, in my API monitoring logs (using Runscope) I'm seeing multiple API requests for the same endpoint, and varying numbers for different endpoints in a single getData() call (they should all be the same). E.g.
I can't post the code here (client project) but it's substantially the same framework as the Data Connector code on Google's docs. I have caching and backoff implemented.
Looking for any ideas or if anyone has experienced something similar?
Thanks
Per the this reference, GDS will also perform semantic type detection if you aren't explicitly defining this property for your fields. If the query is semantic type detection, the request will feature sampleExtraction: true
When Data Studio executes the getData function of a community connector for the purpose of semantic detection, the incoming request will contain a sampleExtraction property which will be set to true.
If the GDS report includes multiple widgets with different dimensions/metrics configuration then GDS might fire multiple getData calls for each of them.
Kind of a late answer but this might help others who are facing the same problem.
The widgets / search filters attached to a graph issue getData calls of their own. If your custom adapter is built to retrieve data via API calls from third party services, data which is agnostic to the request.fields property sent forward by GDS => then these API calls are multiplied by N+1 (where N = the amout of widgets / search filters your report is implementing).
I could not find an official solution for this either, so I invented a workaround using cache.
The graph's request for getData (typically requesting more fields than the Search Filters) will be the only one allowed to query the API Endpoint. Before starting to do so it will store a key in the cache "cache_{hashOfReportParameters}_building" => true.
if (enableCache) {
cache.putString("cache_{hashOfReportParameters}_building", 'true');
Logger.log("Cache is being built...");
}
It will retrieve API responses, paginating in a look, and buffer the results.
Once it finished it will delete the cache key "cache_{hashOfReportParameters}building", and will cache the final merged results it buffered so far inside "cache{hashOfReportParameters}_final".
When it comes to filters, they also invoke: getData but typically with only up to 3 requested fields. First thing we want to do is make sure they cannot start executing prior to the primary getData call... so we add a little bit of a delay for things that might be the search filters / widgets that are after the same data set:
if (enableCache) {
var countRequestedFields = requestedFields.asArray().length;
Logger.log("Total Requested fields: " + countRequestedFields);
if (countRequestedFields <= 3) {
Logger.log('This seams to be a search filters.');
Utilities.sleep(1000);
}
}
After that we compute a hash on all of the moving parts of the report (date range, plus all of the other parameters you have set up that could influence the data retrieved form your API endpoints):
Now the best part, as long as the main graph is still building the cache, we make these getData calls wait:
while (cache.getString('cache_{hashOfReportParameters}_building') === 'true') {
Logger.log('A similar request is already executing, please wait...');
Utilities.sleep(2000);
}
After this loop we attempt to retrieve the contents of "cache_{hashOfReportParameters}_final" -- and in case we fail, its always a good idea to have a backup plan - which would be to allow it to traverse the API again. We have encountered ~ 2% error rate retrieving data we cached...
With the cached result (or buffered API responses), you just transform your response as per the schema GDS needs (which differs between graphs and filters).
As you start implementing this, you`ll notice yet another problem... Google Cache is limited to max 100KB per key. There is however no limit on the amount of keys you can cache... and fortunately others have encountered similar needs in the past and have come up with a smart solution of splitting up one big chunk you need cached into multiple cache keys, and gluing them back together into one object when retrieving is necessary.
See: https://github.com/lwbuck01/GASs/blob/b5885e34335d531e00f8d45be4205980d91d976a/EnhancedCacheService/EnhancedCache.gs
I cannot share the final solution we have implemented with you as it is too specific to a client - but I hope that this will at least give you a good idea on how to approach the problem.
Caching the full API result is a good idea in general to avoid round trips and server load for no good reason if near-realtime is good enough for your needs.
I'm building an application that stores files into the FIWARE Object Storage. I don't quite understand what is the correct way of storing files into the storage.
The code python code snippet below taken from the Object Storage - User and Programmers Guide shows 2 ways of doing it:
def store_text(token, auth, container_name, object_name, object_text):
headers = {"X-Auth-Token": token}
# 1. version
#body = '{"mimetype":"text/plain", "metadata":{}, "value" : "' + object_text + '"}'
# 2. version
body = object_text
url = auth + "/" + container_name + "/" + object_name
return swift_request('PUT', url, headers, body)
The 1. version confuses me, because when I first looked at the only Node.js module (repo: fiware-object-storage) that works with Object Storage, it seemed to use 1. version. As the module was making calls to the old (v.1.1) API version instead of the presumably newest (v.2.0), referencing to the python example, not sure if that is an outdated version of doing it or not.
As I played more with the module, realised it didn't work and the code for it was a total mess. So I forked the project and quickly understood that I will need rewrite it form the ground up, taking the above mention python example from the usage guide as an reference. Link to my repo.
As of writing this the only methods that aren't implement is the object storage (PUT) and object fetching (GET).
Had some addition questions about the Object Storage which I sent to fiware-lab-help#lists.fiware.org, but haven't heard anything back so asking them here.
Haven't got much experience with writing API libraries. Should I need to worry about auth token expiring? I presume it is not needed to make a new authentication, every time we interact with storage. The authentication should happen once when server is starting-up (we create a instance) and it internally keeps it. Should I implement some kind of mechanism that refreshes the token?
Does the tenant id change? From the quote below is presume that getting a tenant I just a one time deal, then later you can use it in the config to make less authentication calls.
A valid token is required to access an object store. This section
describes how to get a valid token assuming an identity management
system compatible with OpenStack Keystone is being used. If the
username, password and tenant details are known, only step 3 is
required. source
During the authentication when fetching tenants how should I select the "right" one? For now i'm just taking the first one similar as the example code does.
Is it true that a object storage container belongs to only a single region?
Use only what you call version 2. Ignore your version 1. It is commented out in the example. It should be removed from the documentation.
(1) The token will be valid for some period of time. This could be an hour or a day, depending on the setup. This period of time should be specified in the token that is returned by the authentication service. The token needs to be periodically refreshed.
(2) The tenant id does not change.
(3) Typically only one tenant id is returned. It is possible, however, that you were assigned more than one id, in which case you have to pick which one you are currently using. Containers typically belong to a single tenant and are not shared between tenants.
(4) Containers are typically limited to a single region. This may change in the future when multi-region support for a container is added to Swift.
Solved my troubles and created the NPM module that works with the FIWARE Object Storage: https://github.com/renarsvilnis/fiware-object-storage-ge
Is there an equivalent of PostgresQL's notify and listen in MySQL? Basically, I need to listen to triggers in my Java application server.
Ok, so what I found is that you can make UDF functions in mysql that can do anything but need to be written in C/C++. They can be then called from triggers on updates in database and notify your application when update happened. I saw that there are some security concerns. I did not use it myself but from what I can see it looks like something that could accomplish what you want to do and more.
http://dev.mysql.com/doc/refman/5.6/en/adding-udf.html
The github project mysql-notification provides a MySQL user defined function MySQLNotification() as a plugin to MySQL that will send notification events via a socket interface. This project includes a sample NodeJS test server that receives the notification events that could be adapted for Java or any other socket service.
Example use:
$ DELIMITER ##
$ CREATE TRIGGER <triggerName> AFTER INSERT ON <table>
FOR EACH ROW
BEGIN
SELECT MySQLNotification(NEW.id, 2) INTO #x;
END##
Project includes full source code and installation instructions for OSX and Linux. License is GNU v3.
No, there aren't any built-in functions like these yet.
You need to "ping" (every 1-5 seconds) database with selecting with premade flag like "read" 0/1. After
SELECT * FROM mytable WHERE read = 0
update it with read = 1
I needed to do this, so I designed my application to send the update notices itself.
E.g.
--Scenario--
User A is looking at record 1
User B saves an update to record 1 while User A has it open.
Process:
I wrote my own socket server as a Windows Service. I designed a que like system which is basically,
EntityType EntityID NoticeType
Where the EntityType is the type of Poco in my data layer that needs to send out notices, EntityID is the primary key value of the row that changed in sql (the values of the poco), and NoticeType is 1 Updated, 2 Inserted, and 3 Deleted.
The socket server accepts connections from the server side application code on a secure connection "meaning client side code cannot make requests designed to be sent by the server side application code"
The socket server accepts a message like
900 1 1023 1
Which would mean the server needs to notify concerned client connections that Entity Type 1 "Person" with ID 1023 was Updated.
The server knows what users need to be notified because when User's look at a record, they are registered in the socket server as having an interest in the record and the record's ID which is done by the web socket code in the client side javascript.
Record 1 is a POCO in my app code that has an IsNew and IsDirty field. "Using EntityFrameWork6 and MySql" If UserB's save caused an actual change (and not just saving existing data) IsDirty will be true on the postback on UserB's POCO.
The application code see's the record is dirty then notifies the socket server with a server side sent socket "which will be allowed" that says Entity 1 with ID 1023 was Updated.
The socket server sees it, and puts it in the que.
Being .Net, I have a class for concerned users that uses the same pocos from the data layer running in the Socket Server window service. I use linq to select users who are working with an entity matching the entity type and primary key id of the entity in the que.
It then loops through those users and sends them a socket like
901 1 1023 1 letting them know the entity was updated.
The javascript in the client side receives it causing users B's page to do an ajax postback on Record 1, But what happens with UserA's is different.
If user A was in the process of making a change, they will get a pop up to show them what changed, and what their new value will be if they click save and asks them which change they want to keep. If UserA doesn't have a change it does an ajax postback with a notification bar at the top that says "Record Change: Refreshed Automatically" that expires after a few seconds.
The Cons to this,
1. It's very complex
2. It won't catch insert/update/delete operations from outside of the application.
In my case, 2 won't happen and if 2 does happen it's by myself or another dev who knows how to manually create the notify que requests "building an admin page for that".
You can use https://maxwells-daemon.io to do so.
It is based on mysql bin logs, when changes in database is occurred it will send json message with updates to kafka, rabbitmq or other streaming platforms
I have a rather complex and large database with about 3000+ objects (tables/triggers/sps combined). I inherited this DB and restructuring it is probably 3-4 years away.
meanwhile, I need to implement a pub sub feature for any insert/update/delete on these tables. Given number of tables and existing queries probably query notification (and SQL Dependency) will not work. What I am looking for is a way to push the changes (what changed in table - like records PK and table name) on the service broker so I can use external activator to then retrieve change,and then use my custom pub sub from that point onwards.
I have pretty much all the ducks lined up except for the way to push change notification on service broker.
Any help/pointers are appreciated.
Thanks.
N M
PS. I did look around for similar postings and did come across a few however, MSDN articles they referred to seem to have all removed - not sure what's going on on MSDN site.
For external activator look at Microsoft SQL Server 2008 Feature Pack- "Microsoft SQL Server 2008 R2 Service Broker External Activator".
For console application (that processes messages) great idea is to drop an eye in codeplex. There is good examples.
To put event notification (notifications, that will be used by external activator service) code looks something like this:
Create Queue ExternalActivatorQueue;
Create Service ExternalActivatorService On Queue ExternalActivatorQueue
([http://schemas.microsoft.com/SQL/Notifications/PostEventNotification])
Create Event Notification NotifyExternalActivator
On Queue dbo.ProcessQueue
For QUEUE_ACTIVATION
To Service 'ExternalActivatorService', 'current database'
To send message in the queue:
Declare #h UniqueIdentifier;
Declare #x xml = '<tag/>';
Begin Dialog Conversation #h
From Service MyTableService
To Service 'ProcessService'
With Encryption = OFF;
Send On Conversation #h(#x)
All steps i done to make it work is here, but just in Latvian :). There actually is almost what you need (trigger that sends messages when data are inserted in table..).