I am building a service that continuously consumes messages from a IBM MQ Queue. This service runs in a private cloud with 80 replicas (containers or PODs) listening to the queue. Observing the application logs, I can see about 20 PODs that are not consuming any messages, even though I can check the queue PUT metric counter which shows about 200 req/s (rush hour). Most of the messages stay in the queue for a while (20 seconds) and get expired.
During the non rush hour, the application receives about 40 req/s and can process all the messages without any being expired.
I wonder if there is a limit of active listeners that the IBM MQ Server support? If so, is there any way to increase this limit?
I create a JMSContext using this method:
public JMSContext buildContext() {
if (this.context == null) {
try {
JmsConnectionFactory connection = JmsFactoryFactory.getInstance(JmsConstants.WMQ_PROVIDER).createConnectionFactory();
String hostname = System.getenv("HOSTNAME");
String appName = application.get();
if (hostname != null && !hostname.trim().isEmpty()) {
if (hostname.length() > 28) { // Tamanho máximo da propriedade WMQ_APPLICATIONNAME
appName = hostname.substring(hostname.length() - 28);
} else {
appName = hostname;
}
}
connection.setStringProperty(CommonConstants.WMQ_HOST_NAME, host.get());
connection.setIntProperty(CommonConstants.WMQ_PORT, port.get());
connection.setStringProperty(CommonConstants.WMQ_CHANNEL, channel.get());
connection.setStringProperty(CommonConstants.WMQ_QUEUE_MANAGER, queueManager);
connection.setStringProperty(JmsConstants.USERID, userid);
connection.setStringProperty(JmsConstants.PASSWORD, password);
connection.setBooleanProperty(JmsConstants.USER_AUTHENTICATION_MQCSP, isUserAuthenticationMqcsp);
connection.setIntProperty(CommonConstants.WMQ_CONNECTION_MODE, CommonConstants.WMQ_CM_CLIENT);
connection.setStringProperty(CommonConstants.WMQ_APPLICATIONNAME, appName);
connection.setIntProperty(CommonConstants.WMQ_CLIENT_RECONNECT_OPTIONS, CommonConstants.WMQ_CLIENT_RECONNECT);
LOGGER.info(String.format("buildContext - Parametros da fila a alta [%s],[%s],[%s],[%s],[%s]",
host.get(), channel.get(), application.get(), queueManager, userid));
this.context = connection.createContext();
this.context.start();
this.connectionFactory = connection;
} catch (JMSException e) {
LOGGER.error(String.format("buildContext - error - code: [%s], message: [%s].", e.getErrorCode(), e.getMessage()));
}
}
return this.context;
}
I also configure my message listener using a separate thread as following:
#Override
public void run() {
JMSContext context = factory.buildContext();
this.creditoQueueConsumer = context.createConsumer(context.createQueue(Constantes.PREFIXO_QUEUE.concat(creditoQueue)));
this.creditoQueueConsumer.setMessageListener(mqAltaPlataformaCreditoListener);
// mqAltaPlataformaCreditoListener is an implementation of MessageListener
context.start();
}
The application seems to work fine, however there is a problem when some listeners suddenly stop receiving messages.
I assume the MQ server works as a proxy and then distributes equally messages to all configured listeners, but for some reason, some of the listeners stop receiving messages. The MQ server team mentioned that some connection timeouts are occurring in the server side, as shown in the following picture.
Is there any configuration that can be done in the client to avoid those timeouts ou to avoid the service to stop receiving messages?
Related
I am trying to use MassTransit for Request/Response communication through Azure service bus queue. Sender is an Azure WebApp, Consumer is a windows service installed at on-premise machine.
Everything works fine when it is about small volumes of messages. However as soon as I start sending more than ~20 msg/sec i see severe(1-2 sec) delays in responses from consumer. My telemetry tells me that delay is happening at point when consumer needs to grab messages from queue.
One strange, but I think important part of behavior: I can see that with current load amount of unread messages in queue is on avg constant and its 25. If I send 2x more messages, than I see on avg 50messages in queue. With delays on consumption side i would expect queue to GROW, but it is constant, so it is definitely something inside code that throttles the connection.
Quick info:
There are no problems with hardware on the machine. CPU/Mem not high.
I tried playing with the UseConcurrencyLimit, MaxConcurrentCalls, PrefetchCount configs on consuner side. It did not help
My solution code of sender and consumer are next to classic examples.
Consumer: .Net framework 4.7.2 and MassTransit.Azure.ServiceBus.Core 5.5.2
Here's my listener class with all business logic removed:
public class QueueListener
{
private IBusControl Bus { get; set; }
public QueueListener()
{
Bus = MassTransit.Bus.Factory.CreateUsingAzureServiceBus(serviceBusFactoryConfigurator =>
{
var host = serviceBusFactoryConfigurator.Host(SettingsHelper.AzureServiceBusConnectionString,
(config) =>
{
config.OperationTimeout = TimeSpan.FromSeconds(60);
config.TransportType = TransportType.AmqpWebSockets;
});
serviceBusFactoryConfigurator.ReceiveEndpoint(host, SettingsHelper.CouponQueryQueueName, e =>
{
e.Handler<JToken>(HandleMessage);
e.UseConcurrencyLimit(16);
e.MaxConcurrentCalls = 16;
e.PrefetchCount = 32;
});
serviceBusFactoryConfigurator.EnableBatchedOperations = true;
serviceBusFactoryConfigurator.DefaultMessageTimeToLive = TimeSpan.FromSeconds(60);
});
}
private async Task HandleMessage(ConsumeContext context)
{
await Task.Delay(800);
if (context.ExpirationTime > SystemDateTime.Now)
{
await context.RespondAsync(new CouponUsedList { CouponsUsed = new List<CouponCurrentUsed>() });
}
}
public Task LaunchAsync()
{
return Bus.StartAsync();
}
public Task StopAsync()
{
return Bus.StopAsync();
}
}
Seems that here, once again, it was just missing one config. All code that you write inside ReceiveEndpoint configurates the consumer listener queue and configurations that you provide in CreateUsingAzureServiceBus are configurations for a consumer response queue.
All I needed was to add one line inside consumer configuration. Without this config all prefetched messages are handled gradually
e.EnableBatchedOperations = true;
I have a Flex app that connects to a JBoss/MS-SQL back-end. Some of our customers have a proxy server in front of their JBoss with a timeout of 90 seconds. In our application there are searches that can take up to 2-3 minutes for complex criteria. Since the proxy isn't smart enough to recognize AMF's keep alive pings for what they are the proxy sends a 503 to the client, which in Flex land becomes a "Channel Call Failed" event. In searching SO and other places, this seems to be a common problem. We can't do anything about the proxy or lengthen the timeout, the application needs to handle it.
Of course the back-end continues to process and eventually ships the results to the client. But the user gets an ugly error message and assumes the app is broke.
The solution I have settled on is to consume the CCF error and have the client continue to wait. I have managed the first part, but I can't figure out how to keep the client's handlers active to receive the data (and/or consume another timeout if necessary).
Current error handler:
private function handleSearchError(event : FaultEvent) : void {
if (event.fault.faultCode == "Channel.Call.Failed") {
event.stopImmediatePropagation(); // doesn't seem to help
return;
}
if (searchProgress != null) {
PopUpManager.removePopUp(searchProgress);
searchProgress = null;
}
etc...
}
This is the setup:
<mx:Button id="btnSearch" label="
{resourceManager.getString('recon_perspective',
'ReconPerspective.ReconView.search')}" icon="{iconSearch}"
click="handleSearch()" includeIn="search, default"/>
And:
<mx:method name="search" result="event.token.resultHandler(event);"
fault="handleSearchError(event);"/>
Kicking off the call:
var token : AsyncToken = null;
token = sMSrv.search(searchType.toString(), getSearchMode(), criteria,
smartMatchParent.isArchiveMode);
searchProgress = LoadProgress(PopUpManager.createPopUp
(FlexGlobals.topLevelApplication as DisplayObject, LoadProgress, true));
searchProgress.title = resourceManager.getString('matching', 'smartmatch.loading.trans');
searchProgress.token = token;
searchProgress.showCancelButton = true;
PopUpManager.centerPopUp(searchProgress);
token.resultHandler = handleSearchResults;
token.cancelSearch = false;
So my question is how do I keep handleSearch and handleSearchError alive to consume the events from the server?
I verified that the data comes back from the server using WebDeveloper in the browser to watch the network traffic and if you cause the app to refresh that screen, the data gets displayed.
I'm very in experienced but would this help?
private function handleSearchError(event : FaultEvent) : void {
if (event.fault.faultCode == "Channel.Call.Failed") {
event.stopImmediatePropagation(); // doesn't seem to help
if(event.isImmediatePropagationStopped(true)) {
//After stopped do something here?
}
return;
}
if (searchProgress != null) {
PopUpManager.removePopUp(searchProgress);
searchProgress = null;
}
etc...
}
I'm trying to use STOMP with Apache AMQ as I was hoping web sockets would give me a better performance than the typicalorg.activemq.Amq Ajax connection.
Anyway, my activemq config file has the proper entry
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
And I'm connecting to it via the following means:
function amqWebSocketConn() {
var url = "ws://my.ip.address:61614/stomp";
var client = Stomp.client(url);
var connect_callback = function() {
alert('connected to stomp');
client.subscribe("topic://MY.TOPIC",callback);
var callback = function(message) {
if (message.body) {
alert("got message with body " + message.body);
} else { alert("got empty message"); }
};
};
client.connect("", "", connect_callback);
}
When I first open up the web browser & navigate to http://localhost:8161/admin/connections.jsp It shows the following:
Name Remote Address Active Slow
ID:mymachine-58770-1406129136930-4:9 StompSocket_657224557 true false
Shortly there after - it removes itself. Is there something else I need such as a heart beat to keep the connection alive?
Using
var amq = org.activemq.Amq;
amq.init({
uri : '/myDomain/amq',
timeout : 50,
clientId : (new Date()).getTime().toString()
});
Kept the connection up for the TCP AJAX Connection
I have faced similar problem, solved it using this
client.heartbeat.incoming = 0;
client.heartbeat.outgoing = 0;
You have to add these two lines before connect.
Even after this I have seen disconnection after 5-10 minutes, if there are no incoming messages. To solve that you have to implement ondisconnect call back of connect method.
client.connect('','',connect_callback,function(frame){
//Connection Lost
console.log(frame);
//Reconnect and subscribe again from here
});
This is successfully working in my application.
I'm using HttpClient from WP8 and do a Post request. I know the call may take long time as I'm actually simulating slow network scenarios. Therefore I set the HttpClient.Timeout accordingly to 5 minutes.
However, I get a Timeout at around 60s. I believe the Timeout is not working.
I believe there is an issue with this for WP as stated in this question:
HttpClient Portable returns 404 notfound on WP8.
They use a workaround but that does not applies to my scenario. I do actually want to wait for long time.
My questions:
1) Is it a bug/issue of HttpClient for WP8 or I'm not setting it properly?
2) Do you think of a workaround still using HttpClient?
I've read that maybe HttpWebRequest is an option. However, I believe HttpClient should be ideal for this 'simple' scenario.
My code is simple:
private static async Task<HttpResponseMessage> PostAsync(Uri serverUri, HttpContent httpContent)
{
var client = new HttpClient();
client.Timeout = TimeSpan.FromMinutes(5);
return await client.PostAsync(serverUri, httpContent).ConfigureAwait(false);
}
The server receives the request and while is processing it, the client aborts.
UPDATE: The HttpResponseMessage returned by HttpClient.PostAsyn is this "{StatusCode: 404, ReasonPhrase: '', Version: 0.0, Content: System.Net.Http.StreamContent, Headers: { Content-Length: 0 }}". As I said, the server is found and is receiving the data and processing it.
After some search and some tests I've came to the conclusion that the problem is Windows Phone itself and that it has a 60 seconds timeout (irrespective of the HttpClient) and that cannot be changed to my knowledge. See http://social.msdn.microsoft.com/Forums/en-US/faf00a04-8a2e-4a64-b1c1-74c52cf685d3/httpwebrequest-60-seconds-timeout.
As I'm programming the server as well, I will try the advice by Darin Rousseau in the link provided above, specifically to send an OK and then do some more processing.
UPDATE: The problem seems to be the Windows Phone emulator as stated here:
http://social.msdn.microsoft.com/forums/wpapps/en-us/6c114ae9-4dc1-4e1f-afb2-a6b9004bf0c6/httpclient-doesnt-work-on-windows-phone?forum=wpdevelop. In my experience the tcp connection times-out if it doesn't hear anything for 60s.
Therefore my solution is to use the Http header characters as a way of keep alive. The first line Http header response always starts with HTTP/1.0. So I send the characters one by one with a delay <60s between them. Of course, if the response gets ready, everything that is left is sent right away. This buys some time, for instance if using a delay of 50s per 9 character we get about 450s.
This is a project for my degree so I wouldn't recommend it for production.
By the way, I also tried with other characters instead the sub string of the header, for instance space character, but that results in a http protocol violation.
This is the main part of the code:
private const string Header1 = #"HTTP/1.0 ";
private int _keepAliveCounter = 0;
private readonly object _sendingLock = new object();
private bool _keepAliveDone = true;
private void StartKeepAlive()
{
Task.Run(() => KeepAlive());
}
/// <summary>
/// Keeps the connection alive sending the first characters of the http response with an interval.
/// This is a hack for Windows Phone 8 that need reponses within 60s interval.
/// </summary>
private void KeepAlive()
{
try
{
_keepAliveDone = false;
_keepAliveCounter = 0;
while (!_keepAliveDone && _keepAliveCounter < Header1.Length)
{
Task.Delay(TimeSpan.FromSeconds(50)).Wait();
lock (_sendingLock)
{
if (!_keepAliveDone)
{
var sw = new StreamWriter(OutputStream);
sw.Write(Header1[_keepAliveCounter]);
Console.Out.WriteLine("Wrote keep alive char '{0}'", Header1[_keepAliveCounter]);
_keepAliveCounter++;
sw.Flush();
}
}
}
_keepAliveCounter = 0;
_keepAliveDone = true;
}
catch (Exception e)
{
// log the exception
Console.Out.WriteLine("Error while sending keepalive: " + e.Message);
}
}
Then, the actual processing happens in a different thread.
Comments and critics are appreciated.
It is possible that you are hitting the timeout of the network stream. You can change this by doing,
var handler = new WebRequestHandler();
handler.ReadWriteTimeout= 5 * 60 * 1000;
var client = new HttpClient(handler);
client.Timeout = TimeSpan.FromMinutes(5);
return await client.PostAsync(serverUri, httpContent).ConfigureAwait(false);
The default on the desktop OS is already 5mins. However, it is possible that on Windows Phone it has been reduced by default.
I'm developing an AIR project to do machine integration in Flex. In one machine, my app works really good that I'm able to receive data from the machine as I wanted. In the second one, I think it is like we have to send 'READY' kind off signal to the machine for which we will get 'ACK' in return (Handshake) and then only the communication will begin.
How could this 'READY' signal be sent from ActionAcript using Socket.
My ActionScript Class File will look like this..
protected var socket:Socket;
public function init():void
{
socket = new Socket;
socket.addEventListener(Event.CONNECT, onConnect);
socket.addEventListener(ProgressEvent.SOCKET_DATA, onSocketData);
socket.addEventListener(Event.CLOSE, onClose);
}
/*** connect to the socket */
public function connect():void
{
if(socket.connected)
{
Status = "Socket is already listening to Port " + socket.remotePort + " in " + socket.remoteAddress;
return;
}
socket.connect("localhost", 5331);
}
/*** handles socket connect event */
private function onConnect(event:Event):void
{
if(socket.connected)
Status = "Connection established successfully from " + Capabilities.os;
else
Status = "Connection failure!";
}
/*** manipulation of received data */
private function onSocketData(event:ProgressEvent):void
{
//Data Recieved
}
/*** handles socket close event */
protected function onClose(event:Event):void
{
Status = "Socket Connection is being closed..";
}
The machine(KR-8900)'s output is RS232.. 8 pin mini din (Male).. and to the system it is db9 pin (as usual).. the serial port communication is done by an External Tool 'SerProxy'. SerProxy will send and receive the data to/from machine-System. Using the app, I will have to connect to the port in the System using Socket and perform Read & Write operations.
My problem here is I don't receive any data in my onSocketData function. Before the communication begins, I need to send READY signal.. I'm stuck here as I don't know how to do this in Flex. Any idea or suggestion are eagerly welcomed.
Socket API allows you to write strings and bytes to a binary socket.
When you socket is connected, you can do something like :
socket.writeUTFBytes("READY");
socket.flush();
You'll have to adapt this code so it will send exactly what you'll need it to send. You can use writeByte to write a null ("\0") character or any control character, too.
To read data, you need to use read methods.