Multiple inserts in Couchbase failing - couchbase

Couchbase newbie question:
I am trying to insert 1 million records in couchbase, but I see that nearly about 0.5 million records get inserted (Admin console shows 517365 as Item Count).
Also, from the admin GUI, I can only see 1000 records (10 pages of 100 records each)
Am wondering where rest of the records are vanishing !
1)Can somebody help me with this ?
2)Which log file I should be looking at to find insertion failure errors ?
I suspect Couchbase has a internal queue. Once it gets full, further requests are dropped. If yes, then how to configure the queue size ?
PS: I tried looking into the logs C:\Program Files\Couchbase\Server\var\lib\couchbase\logs, but couldn't figure out anything.
public class Test {
public static void main(String[] args) {
ArrayList<URI> nodes = new ArrayList<URI>();
String cbUrl = "http://127.0.0.1:8091/pools";
String dbName = "deafult";
CouchbaseClient client = null;
try {
nodes.add(URI.create(cbUrl));
client = new CouchbaseClient(nodes, dbName, "");
insertRecords(client);
System.out.println("Test Over");
} catch (Exception e) {
e.printStackTrace();
} finally {
// client.shutdown();
}
}
public static void insertRecords(CouchbaseClient client) throws Exception {
int num = 1000000;
for (int n = 1; n <= num; n++) {
System.out.println("Adding: " + n);
client.set(n + "", 0, n + "");
}
}
}

The set operation in the Couchbase Java SDK is asynchronous. This means once the call returns there is no guarantee that you've even sent the operation to Couchbase since it may not have even been written to the network buffer yet. In order to make sure the operation has completed you need to call the get() function on the object (which is a Future) returned by the set() API.
In other words replace this line:
client.set(n + "", 0, n + "");
with this one:
client.set(n + "", 0, n + "").get();

To expand on #mikewied's answer, to check that all 1,000,000 set operations have completed without having to call .get() explicitly on each one (and hence converting the calls from async to sync), you need to add a listener to each set which tracks how many of your operations have completed.
There's a nice example of how to do this in the blog post announcing Couchbase Java SDK 1.2 :-
final CountDownLatch latch = new CountDownLatch(100);
for (int i = 0; i < 100; i++) {
OperationFuture<Boolean> future = client.set("key-" + i, "value");
future.addListener(new OperationCompletionListener() {
#Override
public void onComplete(OperationFuture<?> future) throws Exception {
latch.countDown();
}
});
}
latch.await();
You create a CountDownLatch, initialised to how many documents you are set()ing, then register a listener which is called on completion of each set (but note the sets are still asynchronous). At the end you then call await() on the latch to ensure that all set operations have completed before continuing.
This approach is described in more detail in the Understanding and Using Asynchronous Operations section of the Couchbase Java SDK Developer guide, along with a more compact syntax if you're using Java 8.

Related

How to handle Modbus exception 0x5

I'm writing a Modbus client program using Qt5 and the QModbusTcpClient class. Here the code I'm using for open a connection and read something:
QModbusClient *_modbus;
bool ModbusMaster::open(QString host, int port)
{
// Disconnect and delete any existing instance
if (_modbus)
{
_modbus->disconnectDevice();
delete _modbus;
}
// Create and open the new connection
_modbus = new QModbusTcpClient(this);
_modbus->setConnectionParameter(QModbusDevice::NetworkPortParameter, port);
_modbus->setConnectionParameter(QModbusDevice::NetworkAddressParameter, host);
_modbus->setTimeout(250);
_modbus->setNumberOfRetries(1);
return _modbus->connectDevice();
}
bool ModbusMaster::read(QModbusDataUnit::RegisterType type, int startAddress, quint16 count)
{
if (!_modbus) return false;
if (_modbus->state() != QModbusDevice::ConnectedState) return false;
QModbusDataUnit req(type, startAddress, count);
if (auto *reply = _modbus->sendReadRequest(req, _id))
{
if (!reply->isFinished()) connect(reply, &QModbusReply::finished, this, &ModbusMaster::readReady);
else delete reply;
return true;
}
return false;
}
void ModbusMaster::readReady()
{
auto reply = qobject_cast<QModbusReply *>(sender());
if (!reply) return;
reply->deleteLater();
if (reply->error() == QModbusDevice::NoError)
{
// do something
}
else if (reply->error() == QModbusDevice::ProtocolError)
{
qDebug() << QString("Read response error: %1 (Mobus exception: 0x%2)").
arg(reply->errorString()).
arg(reply->rawResult().exceptionCode(), -1, 16);
} else {
qDebug() << QString("Read response error: %1 (code: 0x%2)").
arg(reply->errorString()).
arg(reply->error(), -1, 16);
}
}
Sometimes when I read something from the remote device it happens the device returns the exception 0x5. Reading the official Modbus documentation, at page 48 I read:
Specialized use in conjunction with programming
commands.
The server has accepted the request and is
processing it, but a long duration of time will be
required to do so. This response is returned to
prevent a timeout error from occurring in the
client. The client can next issue a Poll Program
Complete message to determine if processing is
completed.
[bold is mine]
I cannot find a description of this "Poll Program Complete message" that seems I must use to handle the exception 0x5.
Did I search wrong? Is there another way to handle this exception?
It depends on type of an equipment, you are working with. You just have to follow the logic, described in equipment mans for this particular exception.
In general there is no special 'Program Complete' event. That means, as it is written for 0x5 - "Specialized use in conjunction with programming commands.". So you just have to poll (read) some flag from your device meaning the internal process in device, which caused this exception, is complete.
Just as an example, I've met with such exception in relay protection device, which issued it when it has been in a process of writing a disturbance record. I had just to check for that record readiness in some time.

Castle.Facilities.AutoTx doesn't start new commitable transaction when [Transaction(TransactionScopeOption.RequiresNew)] is used

When using Castle.Facilities.AutoTx facility with [Transaction(TransactionScopeOption.RequiresNew)] attribute, the expected new System.Transactions.CommittableTransaction is not created.
You can easily test it with the following unit test
using System.Transactions;
using Castle.Facilities.AutoTx.Testing;
using Castle.MicroKernel.Registration;
using Castle.Transactions;
using Castle.Windsor;
using NUnit.Framework;
namespace Castle.Facilities.AutoTx.Tests
{
public class TransService
{
private readonly NewTransService _s2;
public TransService(NewTransService s2)
{
_s2 = s2;
}
[Transaction]
public virtual string DoInTrans()
{
var currentTransaction = System.Transactions.Transaction.Current;
Assert.That(currentTransaction != null, "The current transaction mustn't be null.");
string transId = currentTransaction.TransactionInformation.LocalIdentifier;
_s2.DoInNewTrans(transId);
return transId;
}
}
public class NewTransService
{
[Transaction(TransactionScopeOption.RequiresNew)]
public virtual string DoInNewTrans(string parentTransId)
{
var currentTransaction = System.Transactions.Transaction.Current;
Assert.That(currentTransaction != null, "The current transaction mustn't be null.");
string transId = currentTransaction.TransactionInformation.LocalIdentifier;
Assert.AreNotEqual(parentTransId, transId, "Ambient transaction must differ from parent");
return transId;
}
}
public class SingleThread_NewAmbient
{
private WindsorContainer _Container;
[SetUp]
public void SetUp()
{
_Container = new WindsorContainer();
_Container.AddFacility<AutoTxFacility>();
_Container.Register(Component.For<TransService>());
_Container.Register(Component.For<NewTransService>());
}
[TearDown]
public void TearDown()
{
_Container.Dispose();
}
[Test]
public void Automatically_Starts_New_CommitableTransaction()
{
using (var scope = new ResolveScope<TransService>(_Container))
scope.Service.DoInTrans();
}
}
}
Am I misunderstanding the purpose of [Transaction(TransactionScopeOption.RequiresNew)] or is it a bug?
I have been digging into the Castle.Transactions source code and I was able to fix the behavior by changing following piece of code in Castle.Transactions.TransactionManager.ITransactionManager.CreateTransaction(ITransactionOptions transactionOptions):
if (activity.Count == 0)
tx = new Transaction(new CommittableTransaction(new TransactionOptions
...
to
if (activity.Count == 0 || transactionOptions.Mode == TransactionScopeOption.RequiresNew)
tx = new Transaction(new CommittableTransaction(new TransactionOptions
...
Can someone from Castle experts / owners check this?
Author here,
I think your code is great and would merge a PR with it if it doesn't break other tests. =)
The reason that RequiresNew isn't well supported is because it's in 99% of the cases an anti-pattern. You're after encapsulating your unit of work in a transaction; and your unit of work should correspond 1-1 with the business operation that should be consistent.
Now, if you have a transaction going on the current thread, like you'd have in your case if you need to use 'RequiresNew', then you're either after reading dirty data or spawning an unrelated transaction (from a business-operation perspective). Hence, you should be doing it in another thread.
Since transactions are 'ambient' and not explicit in the control flow in a programming language like C#, you're left with your 'call context slots' to save the transaction reference in; but from your code's point of view these don't exist; what you have are partial functions that only work in a transactional context. If you spawn a second transaction; how are you going to coordinate that with the current transaction? It's hard; and likely to lead to problems.
An aside
In other transactional systems, like geteventstore.com you get an explicit transaction identifier -- you have one in System.Transactions, too, but it's not explicit in the API/ABI/ADO.Net to the database, hence you can't use it the same way. With an explicit transaction identifier you can solve failures when it becomes 'in doubt', i.e. the 'generals problem'; you can't with System.Transactions. Instead you must mount the transaction MMC on the DTC in question and manually roll it back or forward.

How to count all HTTP requests sent, retries in?

Some use cases require being able to count the requests sent by the Apache API. For example, when massively requesting a web API, which API requires an authentication through an API key, and which TOS limits the requests count in time for each key.
Being more specific on the case, I'm requesting https://domain1/fooNeedNoKey, and depending on its response analyzed data, I request https://domain2/fooNeedKeyWithRequestsCountRestrictions. All sends of those 1-to-2-requests sequences, are performed through a single org.apache.http.impl.client.FutureRequestExecutionService.
As of now, depending on org.apache.httpcomponents:httpclient:4.3.3, I'm using those API elements:
org.apache.http.impl.client.FutureRequestExecutionService, to perform multi-threaded HTTP requests. It offers time metrics (how much time did an HTTP thread took until terminated), but no requests counter metrics
final CloseableHttpClient httpClient = HttpClients.custom()
// the auto-retry feature of the Apache API will retry up to 5
// times on failure, being also allowed to send again requests
// that were already sent if necessary (I don't really understand
// the purpose of the second parameter below)
.setRetryHandler(new StandardHttpRequestRetryHandler(5, true))
// for HTTP 503 'Service unavailable' errors, also retrying up to
// 5 times, waiting 500ms between each retry. Guessed is that those
// 5 retries are part of the previous "global" 5 retries setting.
// The below setting, when used alone, would allow to only enable
// retries for HTTP 503, or to get a greater count of retries for
// this specific error
.setServiceUnavailableRetryStrategy(new DefaultServiceUnavailableRetryStrategy(5, 500))
.build();, which customizes the Apache API retry behavior
Getting back to the topic :
A request counter could be created by extending the Apache API retry-related classes quoted before
Alternatively, an Apache API support unrelated ticket tends to indicate this requests-counter metrics could be available and forwarded out of the API, into Java NIO
Edit 1:
Looks like the Apache API won't permit this to be done.
Quote from the inside of the API, RetryExec not beeing extendable in the API code I/Os:
package org.apache.http.impl.execchain;
public class RetryExec implements ClientExecChain {
..
public CloseableHttpResponse execute(
final HttpRoute route,
final HttpRequestWrapper request,
final HttpClientContext context,
final HttpExecutionAware execAware) throws IOException, HttpException {
..
for (int execCount = 1;; execCount++) {
try {
return this.requestExecutor.execute(route, request, context, execAware);
} catch (final IOException ex) {
..
if (retryHandler.retryRequest(ex, execCount, context)) {
..
}
..
}
}
The 'execCount' variable is the needed info, and it can't be accessed since it's only locally used.
As well, one can extend 'retryHandler', and manually count requests in it, but 'retryHandler.retryRequest(ex, execCount, context)' is not provided with the 'request' variable, making it impossible to know on what we're incrementing a counter (one may only want to increment the counter for requests sent to a specific domain).
I'm out of Java ideas for it. A 3rd party alternative: having the Java process polling a file on disk, managed by a shell script counting the desired requests. Sure it will make a lot of disk read-accesses and will be a hardware killer option.
Ok, the work around was easy, the HttpContext class of the API is intended for this:
// optionnally, in case your HttpCLient is configured for retry
class URIAwareHttpRequestRetryHandler extends StandardHttpRequestRetryHandler {
public URIAwareHttpRequestRetryHandler(final int retryCount, final boolean requestSentRetryEnabled)
{
super(retryCount, requestSentRetryEnabled);
}
#Override
public boolean retryRequest(final IOException exception, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(exception, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// optionnally, in addition to the previous one, in case your HttpClient has specific settings for the 'Service unavailable' errors retries
class URIAwareServiceUnavailableRetryStrategy extends DefaultServiceUnavailableRetryStrategy {
public URIAwareServiceUnavailableRetryStrategy(final int maxRetries, final int retryInterval)
{
super(maxRetries, retryInterval);
}
#Override
public boolean retryRequest(final HttpResponse response, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(response, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// main HTTP querying code: retain the URI in the HttpContext to make it available in the custom retry-handlers code
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
try {
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
httpClient.execute(httpGET, getHTTPResponseHandlerLazy(), httpContext);
// if request got successful with no need of retries, of if it succeeded on the last send: in any cases, this is the last query sent to server and it got successful
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final ClientProtocolException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final IOException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} finally {
// restoring the context as it was initially
httpContext.removeAttribute("requestURI");
}
Solved.

Handling database connection exceptions with Linq to SQL and Rx

I am trying learn how to best use the Reactive Extensions library and have set up simple test WPF application to view a logging database table. In a ViewModel class I am populating an ObservableCollection with the first 100 log entries from a Linq to Sql DataContext and I'm trying to use Rx to keep the UI responsive.
The following snippet works unless the database is unavailable at which point the app throws an exception and crashes. Where would be the best place to handle database connection exceptions and why are they not handled by the OnError method of the Observer?
ObservableCollection<LogEntry> _logEntries = new ObservableCollection<LogEntry>();
DataContext dataContext = new DataContext( "connection string" );
(from e in dataContext.LogEntries
select e).Take( 100 ).ToObservable()
.SubscribeOn( Scheduler.ThreadPool )
.ObserveOnDispatcher()
.Subscribe( _logEntries.Add, ex => System.Diagnostics.Debug.WriteLine( ex.ToString() ) );
Try this instead of ToObservable:
public static IObservable<T> SafeToObservable(this IEnumerable<T> This)
{
return Observable.Create(subj => {
try {
foreach(var v in This) {
subj.OnNext(v);
}
subj.OnCompleted();
} catch (Exception ex) {
subj.OnError(ex);
}
return Disposable.Empty;
});
}
In general though, this isn't a great use of Rx since the data source isn't very easy to Rx'ify - in fact, the code will execute most of the work on the UI thread, send it out to random worker threads, then send it back (i.e. completely wasted work). Task + Dispatcher.BeginInvoke might suit you better here.

Exception when sending big soap request

There is a web-service deployed on tomcat 6 and exposed via apache-cxf 2.3.3. A generated sources stubs using wsdl2java to be able to call this service.
Things seemed fine until I sent big request(~1Mb). This request wasn't processed and failing with exception:
Interceptor for {http://localhost/}ResourceAllocationServiceSoapService has thrown
exception, unwinding now org.apache.cxf.binding.soap.SoapFault:
Error reading XMLStreamReader.
...
com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]
Is some kind of max request length here, I'm totally stuck with it.
Vladimir's suggestion worked. This code below will help others with understanding where to put the 1000000.
public void handleMessage(SoapMessage message) throws Fault {
// Get message content for dirty editing...
InputStream inputStream = message.getContent(InputStream.class);
if (inputStream != null)
{
String processedSoapEnv = "";
// Cache InputStream so it can be read independently
CachedOutputStream cachedInputStream = new CachedOutputStream(1000000);
try {
IOUtils.copy(inputStream,cachedInputStream);
inputStream.close();
cachedInputStream.close();
InputStream tmpInputStream = cachedInputStream.getInputStream();
try{
String inputBuffer = "";
int data;
while((data = tmpInputStream.read()) != -1){
byte x = (byte)data;
inputBuffer += (char)x;
}
/**
* At this point you can choose to reformat the SOAP
* envelope or simply view it just make sure you put
* an InputStream back when you done (see below)
* otherwise CXF will complain.
*/
processedSoapEnv = fixSoapEnvelope(inputBuffer);
}
catch(IOException e){
}
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
// Re-set the SOAP InputStream with the new envelope
message.setContent(InputStream.class,new ByteArrayInputStream( processedSoapEnv.getBytes()));
/**
* If you just want to read the InputStream and not
* modify it then you just need to put it back where
* it was using the CXF cached inputstream
*
* message.setContent(InputStream.class,cachedInputStream.getInputStream());
*/
}
}
I figured out what was wrong. Actually it was bug inside interceptor's code:
CachedOutputStream requestStream = new CachedOutputStream()
When I replaced this with
CachedOutputStream requestStream = new CachedOutputStream(1000000);
things start working fine.
So the request was just trunkated during copying of streams.
I run into same issue of geting "com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog" when using CachedOutputStream class.
Looking at sources of CachedOutputStream class the threshold is used to switch between storing stream's data from "in memory" to "a file".
Assuming stream operates on data that exceeds threshold it gets stored in a file thus following code is going to break
IOUtils.copy(inputStream,cachedInputStream);
inputStream.close();
cachedInputStream.close(); //closes the stream, the file on disk gets deleted
InputStream tmpInputStream = cachedInputStream.getInputStream(); //returned tmpInputStream is brand *empty* one
// ... reading tmpInputStream here will produce WstxEOFException
Increasing 'threshold' does help as all stream data is stored into memory and in such scenario calling cachedInputStream.close() does not really close the underlying stream implementation so one can still read from it later on.
Here is 'fixed' version of above code (at least it worked without exception for me)
IOUtils.copy(inputStream,cachedInputStream);
inputStream.close();
InputStream tmpInputStream = cachedInputStream.getInputStream();
cachedInputStream.close();
// reading from tmpInputStream here works fine
Temporary file gets deleted when close() is called on tmpInputStream and there are no more other references to it, see source code of CachedOutputStream.maybeDeleteTempFile()